Configuring Brocade ICX-Series Ethernet Switches

From Alteeve Wiki
Jump to navigation Jump to search

 AN!Wiki :: How To :: Configuring Brocade ICX-Series Ethernet Switches

Brocade ICX6450-24 ethernet switch. Photo by brocade.

Purpose of This Tutorial

The Brocade-brand ICX-series switches, with a focus on the ICX6450 and ICX6610 switches, are great ethernet switches for use in an Anvil! system.

This mini-tutorial covers how to set them up to provide a stacked, hitless failover configuration with appropriate VLANs to isolate and protect the three networks.

Serial Access

Antaira UTS-1110A single-port USB to RS-232] serial adapter. Photo by Antaira.

By default, the APC AP7900 does not have an IP address assigned.

The easiest way to configure an IP address is to log into the PDU over a serial port. Most servers still come with a built-in 9-pin serial port, but most desktops and laptops do not. If you don't have one, many vendors sell USB to serial adapters. The one we've used quite successfully is the Antaira UTS-1110A, but almost any adapter you can find at your local computer store will work.

Note: The m2 Anvil! installs screen on both nodes and Striker dashboards.

We're going to use a program called screen to talk to the PDU over the serial port.

To do this, you need to know what device the PDU's serial cable is plugged into. Some common ones:

Type Device path
Serial port 1 /dev/ttyS0
Serial port 2 /dev/ttyS1
USB-Connected port 1 /dev/ttyUSB0
USB-Connected port 2 /dev/ttyUSB1

... and so on.

In my case, I'm using the port on a USB to Serial adapter, so I will use '/dev/ttyUSB0'.

Knowing this, we're ready to go!

Open a terminal window and run:

screen /dev/ttyUSB0
Note: When you are done, to exit screen, press '<ctrl>' + 'a', release, then press 'k'. You will be prompted to kill the session, type 'y'.

Connecting

Note: The prompt will reflect the specific switch's model number. In the examples below, we will be working on an ICX6610.

Using the serial port, connect to the switch.

ICX6610-24 Switch>

Resetting To Factory Defaults

Warning: This will erase any loaded licenses. If you have loaded licenses, be sure you have them backed up before proceeding!

This tutorial assumes the switches are starting with a fresh configuration. To assist with that, lets start with the process of resetting the switches to their factory defaults.

ICX6610-24 Switch> erase startup-config
Erase startup-config Done.
dhcp server lease database is also removed
Note: Do not write memory! This will reload the running configuration back onto the system.

Now reboot.

ICX6610-24 Switch> reload
Are you sure? (enter 'y' or 'n'): 
Sent SIGKILL to all processes
Requesting system reboot
Restarting syste

Once it comes back up, your switch will be as-new.

Assign IP to Switch

Note: In order to update the firmware or load licenses, you will need to be able to access the switch via an IP address.

Log into the switch over serial and switch to the configuration terminal:

ICX6610-48 Switch#configure terminal

Assign the IP address and subnet mask to the switch:

ICX6610-48 Switch(config)#ip address 10.20.1.1 255.255.0.0

Exit the configuration terminal:

ICX6610-48 Switch(config)#exit
Note: Our router is at 10.20.255.254, which is what we will ping here. You can choose any target device on the same subnet instead.

Test by pinging an external device on the same subnet:

ICX6610-48 Switch#ping 10.20.255.254
Sending 1, 16-byte ICMP Echo to 10.20.255.254, timeout 5000 msec, TTL 64
Type Control-c to abort
Reply from 10.20.255.254   : bytes=16 time<1ms TTL=64
Success rate is 100 percent (1/1), round-trip min/avg/max=0/0/0 ms.

Good, save the changes:

ICX6610-48 Switch#write memory
Write startup-config done.

Automatic copy to member units:  2
Flash Memory Write (8192 bytes per dot) .
Flash to Flash Done.

Done!

Update the firmware

Warning: Be sure to have your switches on UPSes before flashing them! As with all devices, a power loss during the flash process may cause the switch to become unusable.

Selecting the Firmware Version

TODO: This is an old firmware example, update with newer versions.

Before we start, we need to decide what version of the firmware we want to install. To do this, start by logging in to your switch(es) and check their current firmware version.

ICX6610-48 Switch#show flash
Stack unit 1:
  Compressed Pri Code size = 5370497, Version:07.3.00fT7f1 (/foundry/FGS/os/FCXS07300f.bin)
  Compressed Sec Code size = 6905002, Version:07.3.00fT7f3 (/foundry/FGS/os/FCXR07300f.bin)
  Compressed Boot-Monitor Image size = 370555, Version:07.3.02T7f5
  Code Flash Free Space = 52822016
Stack unit 2:
  Compressed Pri Code size = 5370497, Version 07.3.00fT7f1 (/foundry/FGS/os/FCXS07300f.bin)
  Compressed Sec Code size = 6905002, Version 07.3.00fT7f3 (/foundry/FGS/os/FCXR07300f.bin)
  Compressed Boot-Monitor Image size = 370555, Version 07.3.02T7f5
  Code Flash Free Space = 52822016

Note that both switches are running the major version 07.3.00f. Being that we're interested in High Availability, we are going to be conservative and upgrade to the latest subversion available. Both switches are 7.3, so that is what we will use. If one had been 7.4, for example, then we would have upgraded both to the latest 7.4 version.

Layer 2 vs Layer 3

These switches have two firmwares;

  • Layer 2 (S); This is seen in the above output in the current version's flash file "/foundry/FGS/os/FCXS07300f.bin".
  • Layer 3 (R); This is seen in the above output in the current version's flash file "/foundry/FGS/os/FCXR07300f.bin".

We're only going to update the layer 2 "S" firmware. We don't use the layer 3 features for HA so we won't update them. If you want to updated them, that is fine. The upgrade process below is easily ported to the layer 3 firmware update process.

Downloading the Firmware

To get the firmware:

  • Create or log into your My Brocade account.
  • Click on 'Downloads'.
  • In the Download by select box, select Ethernet Switches.
  • Scroll down and click to expand the arrow to the left of "ICX6610" (or your switch model).
  • Click to expand the version of firmware you want to use. For this tutorial, we're going to be conservative and stay in the same "7.3" release, so we will expand 07300.
  • Click to expand the most recent subversion for your selected firmware major number. In my case, that is "07300j".
  • This takes you to a note about export restrictions. Ensure you meet the criteria and then click to acknowledge compliance, the click on "Submit".
  • You will see the EULA. Read it, understand it and then agree to it by clicking on "I Accept".
  • Save the file on your computer.

Extracting the Image

Navigate to the downloaded file. It will be a .zip file. Expand it. This will create several directories.

To upload the firmware, we need to copy the image to the root of our TFTP server (/var/lib/tftpboot/ on EL6).

Browse into the extracted 07300j/ICX/Images/ directory (or the matching directory for your firmware). You will see two file:

cd 07300j/ICX/Images/
ls -lah
total 12M
drwxr-xr-x 2 root root 4.0K May  8  2014 .
drwxr-xr-x 8 root root 4.0K May  9  2014 ..
-rwxr-xr-x 1 root root 6.7M May  8  2014 FCXR07300j.bin
-rwxr-xr-x 1 root root 5.2M May  8  2014 FCXS07300j.bin

As discussed above, we will be upgrading the layer 2 "S" image. So this is the file we will copy to the TFTP root directory.

cp FCXS07300j.bin /var/lib/tftpboot/
ls -lah /var/lib/tftpboot/FCXS07300j.bin

Performing the Flash

Note: We will be flashing both switches at once as they are stacked.

We're ready to flash! So it's time to log into the switch.

ICX6610-48 Switch#copy tftp flash 10.20.255.254 FCXS07300j.bin primary
Flash Memory Write (8192 bytes per dot) 
Automatic copy to member units:  2
....<lots of dots>....
TFTP to Flash Done.
ICX6610-48 Switch#show flash
Stack unit 1:
  Compressed Pri Code size = 5430491, Version:07.3.00jT7f1 (FCXS07300j.bin)
  Compressed Sec Code size = 6905002, Version:07.3.00fT7f3 (/foundry/FGS/os/FCXR07300f.bin)
  Compressed Boot-Monitor Image size = 370555, Version:07.3.02T7f5
  Code Flash Free Space = 52690944
Stack unit 2:
  Compressed Pri Code size = 5430491, Version 07.3.00jT7f1 (FCXS07300j.bin)
  Compressed Sec Code size = 6905002, Version 07.3.00fT7f3 (/foundry/FGS/os/FCXR07300f.bin)
  Compressed Boot-Monitor Image size = 370555, Version 07.3.02T7f5
  Code Flash Free Space = 52690944

Reboot to load the new firmware.

ICX6610-48 Switch#write memory
ICX6610-48 Switch#reload
Are you sure? (enter 'y' or 'n'): y
Note: It takes a while to reboot because it will update the firmware of the second node. Please be patient.
ICX6610-48 Switch#show flash
Stack unit 1:
  Compressed Pri Code size = 5430491, Version:07.3.00jT7f1 (FCXS07300j.bin)
  Compressed Sec Code size = 6905002, Version:07.3.00fT7f3 (/foundry/FGS/os/FCXR07300f.bin)
  Compressed Boot-Monitor Image size = 370555, Version:07.3.02T7f5
  Code Flash Free Space = 52690944
Stack unit 2:
  Compressed Pri Code size = 5430491, Version 07.3.00jT7f1 (FCXS07300j.bin)
  Compressed Sec Code size = 6905002, Version 07.3.00fT7f3 (/foundry/FGS/os/FCXR07300f.bin)
  Compressed Boot-Monitor Image size = 370555, Version 07.3.02T7f5
  Code Flash Free Space = 52690944

Done!

Form the Stack

A "stack" is a way of linking the fabric of two switches, allowing seamless routing of data between the switches and to provide a common configuration and status interface.

The primary benefit, from the Anvil! perspective, is the common configuration interface. It makes it less likely to have differing configurations and thus provides a more reliable configuration. At the same time, the cables used to link the switches route traffic without regard for VLANs, meaning the one set can route traffic for all three networks.

Warning: The stacking cables must be cables up properly! See the connection maps above or consult the switch documentation. Incorrect cabling will prevent stacking from succeeding!
Note: The method of stacking the ICX-series switches is the same, save for the smaller 12-port switches not covered here. As such, this section will use the 6610 switch, but you can follow the steps exactly with the 6450.
Note: All commands can be run from one switch. Communication will be established with the other switch as appropriate.

First, connect to the serial interface and then log in.

ICX6610-24 Switch>enable

Switch to the configuration section:

ICX6610-24 Switch#configure terminal

Enable the stacking feature:

ICX6610-24 Switch(config)#stack enable
Enable stacking. This unit actively participates in stacking
stacking is enable. optical monitoring for stacking ports 1/2/1, 1/2/6 is not available.

Exit the configuration section:

ICX6610-24 Switch(config)#exit

Now start the stack configuration process:

ICX6610-24 Switch#stack secure-setup
Discovering the stack topology...

Current Discovered Topology - RING

Available UPSTREAM units
Hop(s)  Id      Type          Mac Address
1       new  ICX6610-24     xxxx.xxxx.xxxx

Available DOWNSTREAM units
Hop(s)  Id      Type          Mac Address
1       new  ICX6610-24     xxxx.xxxx.xxxx

Accept the stack topology:

Do you accept the topology (RING) (y/n)?: y
Selected Topology:
Active  Id      Type          Mac Address
        1    ICX6610-24     yyyy.yyyy.yyyy

Selected UPSTREAM units
Hop(s)  Id      Type        Mac Address
1       2    ICX6610-24     xxxx.xxxx.xxxx

Selected DOWNSTREAM units
Hop(s)  Id      Type        Mac Address
1       2    ICX6610-24     xxxx.xxxx.xxxx
Note: The process of switching the stack IDs is a little complicated and outside the scope of this tutorial. If you want to force the IDs, please refer to the switch's manual.

Accept the stack unit IDs:

Do you accept the unit id's (y/n)?: y
T=1d20h24m2.8: Election, was alone --> active, ID=1, pri=128, 2U(1-2), A=u1, nbr#=1 1, 
T=1d20h24m3.5: Election, was active, no change, ID=1, pri=128, 2U(1-2), A=u1, nbr#=1 1, 
reset unit 2: u2 diff bootup id=1
Unit 1 loses all neighbors.
Active unit 1 deletes u2 and its config because it is learned.

Config changed due to add/del units. Do write mem if you want to keep it

The switch will now reboot. Please just wait and watch the output.

T=1d20h26m43.2: Election, was active, no change, ID=1, pri=128, 2U(1-2), A=u1, nbr#=1 1, 
Done hot swap: active controller unit 1 sets unit 2 to Ready.

Config changed due to add/del units. Do write mem if you want to keep it

Make the changes permanent:

ICX6610-24 Switch#write memory
Write startup-config done.

Automatic copy to member units:  2
Flash Memory Write (8192 bytes per dot) .
Flash to Flash Done.

A moment later, more boot messages. In this case, the 6610 has internal redundant power supplies, which will come online.

Stack unit 2 Power supply 1 is up
Stack unit 2 Power supply 2 is up
Assigned unit 2 to be standby
start running config sync
Running config sync to standby is complete

Now you can view the new stack topology:

ICX6610-24 Switch#show stack
alone: standalone, D: dynamic config, S: static config
ID   Type         Role    Mac Address    Pri State   Comment                   
1  S ICX6610-24   active  xxxx.xxxx.xxxx 128 local   Ready
2  S ICX6610-24   standby yyyy.yyyy.yyyy   0 remote  Ready

    active       standby                                                       
     +---+        +---+                                                        
 =2/1| 1 |2/6==2/1| 2 |2/6=                                                    
 |   +---+        +---+   |                                                    
 |                        |                                                    
 |------------------------|                                                    
                                                                              
Standby u2 - No hitless failover. Reason: hitless-failover not configured
Current stack management MAC is xxxx.xxxx.xxxx
Note: no "stack mac" config. My MAC will change after failover.
Warning: These are critical step for surviving switch recovery!

Configure the stack to handle switch "hitless" failure and recovery. This allows a switch to rejoin the stack without causing the existing switch to block traffic.

Switch to the configuration terminal again:

ICX6610-24 Switch#config terminal

Enable hitless-failover:

ICX6610-24 Switch(config)#hitless-failover enable
Note: Brocade's representation of MAC addresses is in 'aabb.ccdd.eeff' format, as opposed to the usual 'aa:bb:cc:dd:ee:ff' format.

We don't want the MAC address to change depending on which switch is master, so we need to specify a MAC address to use for the stack. In this case, I always copy stack unit 1's real MAC address.

ICX6610-24 Switch(config)#stack mac cc4e.24b9.5624

Last, switch to unit 2:

ICX6610-24 Switch(config)#stack unit 2

Change it's priority to match the priority of unit 1.

ICX6610-24 Switch(config-unit-2)#priority 128
Priority change needs 120 seconds to take effect.

Note that the changes will take two minutes. We don't need to wait, however. Switch back to switch 1.

ICX6610-24 Switch(config-unit-2)#exit

Make the changes permanent:

ICX6610-24 Switch(config)#write memory
Write startup-config done.

Automatic copy to member units:  2
Flash Memory Write (8192 bytes per dot) .
Flash to Flash Done.

Two minutes after you change the stack priority, the switch run an election.

Will do election in 30 sec due to priority change
Will do election in 10 sec due to priority change
T=1d20h35m20.2: Election, was active, no change, ID=1, pri=128, 2U(1-2), A=u1, nbr#=1 1,

Verify the stack is configured the way we want.

ICX6610-24 Switch(config)#show stack
alone: standalone, D: dynamic config, S: static config
ID   Type         Role    Mac Address    Pri State   Comment                   
1  S ICX6610-24   active  xxxx.xxxx.xxxx 128 local   Ready
2  S ICX6610-24   standby yyyy.yyyy.yyyy 128 remote  Ready

    active       standby                                                       
     +---+        +---+                                                        
 =2/1| 1 |2/6==2/1| 2 |2/6=                                                    
 |   +---+        +---+   |                                                    
 |                        |                                                    
 |------------------------|                                                    
                                                                              
Standby u2 - protocols ready, can failover or manually switch over
Current stack management MAC is xxxx.xxxx.xxxx

Excellent, done!

License Key Management

The ICX6610-series switches ship by default with ports X1 through X8 at 1 Gbps. To upgrade them to 10 Gbps ports, you need to buy a license. Each license allows four ports to be upgraded. In our case, we want to run all eight ports at 10 Gbps, so we will have two licenses per switch.

To get a license, you must purchase the license from a Brocade reseller. With it, you will get a spreadsheet containing a TRANSACTION KEY.

To convert this into a loadable license file, you will need to get the switch's 'LID', then create an account on "My Brocade". You can then combine the LID and the TRANSACTION KEY and Brocade will email you the license key.

Get the License ID (LID)

To get the LID, log into the switch over serial and run:

ICX6610-48 Switch#show version
  Copyright (c) 1996-2013 Brocade Communications Systems, Inc.
    UNIT 1: compiled on May 10 2013 at 16:48:26 labeled as FCXS07300f
                (5370497 bytes) from Primary /foundry/FGS/os/FCXS07300f.bin
        SW: Version 07.3.00fT7f1 
    UNIT 2: compiled on May 10 2013 at 16:48:26 labeled as FCXS07300f
                (5370497 bytes) from Primary /foundry/FGS/os/FCXS07300f.bin
        SW: Version 07.3.00fT7f1 
  Boot-Monitor Image size = 370555, Version:07.3.02T7f5 (grz07302)
  HW: Stackable ICX6610-48
==========================================================================
UNIT 1: SL 1: ICX6610-48 48-port Management Module
         Serial  #: xxxxxxxxxxx
         License: BASE_SOFT_PACKAGE   (LID: yyyyyyyyyyy)
         P-ENGINE  0: type E02B, rev 01
         P-ENGINE  1: type E02B, rev 01
==========================================================================
UNIT 1: SL 2: ICX6610-QSFP 10-port 160G Module
==========================================================================
UNIT 1: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module
==========================================================================
UNIT 2: SL 1: ICX6610-48 48-port Management Module
         Serial  #: aaaaaaaaaaa
         License: BASE_SOFT_PACKAGE   (LID: bbbbbbbbbbb)
         P-ENGINE  0: type E02B, rev 01                           
         P-ENGINE  1: type E02B, rev 01
==========================================================================
UNIT 2: SL 2: ICX6610-QSFP 10-port 160G Module
==========================================================================
UNIT 2: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module
==========================================================================
  800 MHz Power PC processor 8544E (version 0021/0023) 400 MHz bus
65536 KB flash memory
  512 MB DRAM
STACKID 1  system uptime is 1 days 17 hours 8 minutes 17 seconds 
STACKID 2  system uptime is 22 hours 36 minutes 24 seconds 
The system : started=warm start  reloaded=by "reload"
My stack unit ID = 1, bootup role = active

Note the LID for each switch.

Navigating Brocade's Website

TODO: Make sure this is still accurate now that their website has been updated.

With the TRANSACTION KEY and LID, and having logged into Brocade Software Portal;

  • Hover over the top-left 'License Management' button. A drop-down menu will appear. Hover over 'Brocade IP/ADP'. Another pop-out menu will appear, click on 'License Generation with Transaction Key'.
  • A form will appear. Fill it our as instructed. Check to accept the EULA and then click 'Add'.
    • IF YOU ARE ADDING TWO LICENSES TO ONE SWITCH:
    • Fill out the LID and Transaction key for the next license, again accept the EULA and click 'Add' again.
  • Once all licenses for the specified switch are added, click on 'Generate'.
  • If all was well, you will be shown a summary and an email with the license will be emailed to you.

Repeat as needed for each switch.

Installing Keys

Once you get the license file from Brocade, copy the key to the root of your TFTP server (/var/lib/tftpboot/ on EL6).

Note the LID number and which stack ID. In this case, I renamed the license files so that an-switch01_XXXXXXXXXXX.xml, where XXXXXXXXXXX is the switch's LID. In the example below, I will load two licenses, one for either switch in the stack.

First license:

ICX6610-24 Switch#copy tftp license 10.255.255.254 an-switch01_XXXXXXXXXXX.xml unit 1
Flash Memory Write (8192 bytes per dot) .
Copy Software License from TFTP to Flash Done.
T=1d20h6m27.7: Election, was active, no change, ID=1, pri=128, 2U(1-2), A=u1, nbr#=1 1,

Second license:

ICX6610-24 Switch#copy tftp license 10.255.255.254 an-switch02_XXXXXXXXXXX.xml unit 2
Flash Memory Write (8192 bytes per dot) .
Copy Software License from TFTP to Flash Done.

Copy software license to stack unit 2 success
Flash Memory Write (8192 bytes per dot) .
T=1d20h7m50.1: Election, was active, no change, ID=1, pri=128, 2U(1-2), A=u1, nbr#=1 1,

Show the licenses:

ICX6610-24 Switch#show license
Index      License Name              Lid          License Type    Status     License Period  License Capacity  
Stack unit 1:
1          ICX6610-10G-LIC-POD       xxxxxxxxxxx  Normal          Active     Unlimited                  8 
Stack unit 2:
1          ICX6610-10G-LIC-POD       aaaaaaaaaaa  Normal          Active     Unlimited                  8

Save the changes:

ICX6610-24 Switch#write memory
TODO: Does the user need to switch to switch 2 in the stack first?

Repeat for the second switch.

Enable 10 Gbps Ports

After installing license files to upgrade the eight 1 Gbps SFP+ ports to run as 10 Gbps, we need to actually enable the ports. This may seem silly, but consider that the upgrades are sold in lots of 4-ports at a time. This step would allow you to specify which four ports were upgraded. In our case, we upgraded them all.

Switch to the configuration terminal:

ICX6610-48 Switch#config terminal

Select the interfaces you want to upgrade. You can do one at a time, or multiple at once. We're upgrading all eight so we'll do them all at once:

ICX6610-48 Switch(config)#interface ethernet 1/3/1 to 1/3/8

Change their speed:

ICX6610-48 Switch(config-mif-1/3/1-1/3/8)#speed-duplex 10g-full

This will take a minute, please wait.

Now upgrade the ports on the second switch:

ICX6610-48 Switch(config-mif-1/3/1-1/3/8)#interface ethernet 2/3/1 to 2/3/8
ICX6610-48 Switch(config-mif-2/3/1-2/3/8)#speed-duplex 10g-full

And exit the port configuration:

ICX6610-48 Switch(config-mif-2/3/1-2/3/8)#exit

Note: If you (accidentally) use a 1 Gbps SFP+ cable, the upgrade will likely throw an error like the one below:

U2-MSG: INFO: 2/3/4: optics <-> speed mismatch. Replace with SFP+ to enable link.
U2-MSG: INFO: 2/3/4: optics <-> speed mismatch. Replace with SFP+ to enable link.

You can reduce the port speed to make it work (albeit at 1 Gbps):

ICX6610-48 Switch(config-mif-1/3/1-1/3/8)#interface ethernet 2/3/4
ICX6610-48 Switch(config-if-e10000-2/3/4)#speed-duplex 1000-full-master

When done, save your changes.

ICX6610-48 Switch(config)#write memory
Write startup-config done.

Automatic copy to member units:  2
ICX6610-48 Switch(config)#Flash Memory Write (8192 bytes per dot) .
Flash to Flash Done.

Deleting Keys

If you need to remove a key (ie: replacing a demo license), you can do so thusly:

ICX6610-24 Switch#license delete unit 1 index 1
ICX6610-24 Switch#show license
License record empty

Enable Jumbo Frames

Note: The ICX6610 support packets up to 10,240 bytes (10 KiB) and the ICX6450 supports 9,216 bytes (9 KiB). That is to say, if jumbo frames are enabled, the MTU of the switch is 9 or 10 KiB, depending on the model.

Jumbo frames are a way to increase performance on a network by allowing more data in a packet's payload, increasing transmission efficiency and reducing the number of packets needed for a given transmission. To work, however, you must be aware of the maximum transmission size of your network interfaces, switches and all other devices between your machine and the receiver. If you enable large MTUs (called "jumbo frames"), be certain to test that you really can use the requested MTU!

The easiest way to do this is to ping the remote machine(s) using 'ping -M do -s X', where 'X' is the desired MTU, minus 28 bytes for packet overhead. That is to say, if you set your MTU to '9216', test by pinging the target with 'ping -M do -s 9188'. See the manual page for 'ping' for more information.

Connect to the switch, log in and enter the configuration terminal:

ICX6610-48 Switch#configure terminal

Enable jumbo frames:

ICX6610-48 Switch(config)#jumbo
Jumbo mode setting requires a reload to take effect!

Save the changes:

ICX6610-48 Switch(config)#write memory
Write startup-config done.

Automatic copy to member units:  2
Flash Memory Write (8192 bytes per dot) .
Flash to Flash Done.

Exit the configuration terminal:

ICX6610-48 Switch(config)#exit

Finally, reboot the switch.

ICX6610-48 Switch#reload
Are you sure? (enter 'y' or 'n'): y
Reload request sent to attached stack member(s)...

Rebooting(0)...
Warning: Test fragmentless pings between all node and dashboard interfaces before starting the Anvil! system! If anything is misconfigured and the requested MTU is not working, the cluster can break!

Configure VLAN

VLANs provide a way of dividing up a switch (or stack of switches) into multiple, isolated switches (this is actually somewhat simplistic, but it gets the idea across). Traffic from one VLAN segment can not reach ports allocated to another VLAN, providing a useful layer of security.

Planning

Note: You could use a 48-port switch to host five Anvil! pairs, but it is NOT recommended. An important part of Intelligent Availability is to distribute risk. Multiple two-pair foundation packs serves that purpose best.

We're creating three separate VLANs;

By default, all ports default to VID 1. If you want to improve security, you can leave all unused ports on VID 1. In this tutorial, we need to allocate all ports for use by other equipment.

Before you start, you need to decide how you want to assign physical ports on your switches to VLAN IDs. I like to do this by creating a table showing all the ports, label their external port numbers and logical numbers and then write in what will be plugged into each port. With this, it's fairly easy to mark what ports will be assigned to what VLANs.

We will cover the port assignments for two scenarios;

  1. . Both ICX6540 and ICX6610 switches with 24x 1 Gbps ports configured for two Anvil! pairs.
  2. . Using the 10 Gbps ports on the ICX6610 to support one Anvil! node pair operating at 10 Gbps.

Cabling a pair of ICX6450-24

In this example, we're showing how to cable up a pair of ICX6450-24 switches in a very typical configuration. It can drive two Anvil! systems along with the usual foundation pack.

The table below represents a pair of these stacked together.
an-switch01  
Stack   Back-Channel Network]] Storage Network Internet-Facing Network
X1 (1/2/1) X3 (1/2/3)   1 (1/1/1) 3 (1/1/3) 5 (1/1/5) 7 (1/1/7) 9 (1/1/9) 11 (1/1/11)   13 (1/1/13) 15 (1/1/15) 17 (1/1/17) 19 (1/1/19) 21 (1/1/21) 23 (1/1/23)
To an-switch02
X3
To an-switch02
X1
an-a01n01
BCN - Link 1
an-a01n01
IPMI / Fencing
an-a02n01
BCN - Link 1
an-a02n01
IPMI / Fencing
an-striker01
BCN - Link 2
an-a01n01
SN - Link 1
  an-a02n01
SN - Link 1
an-a01n01
IFN - Link 1
an-a02n01
IFN - Link 1
an-striker01
IFN - Link 1
-- --
-- -- an-a01n02
BCN - Link 1
an-a01n02
IPMI / Fencing
an-a02n02
BCN - Link 1
an-a02n02
BCN - Link 1
-- an-a01n02
SN - Link 1
  an-a02n02
SN - Link 1
an-a01n02
IFN - Link 1
an-a02n02
IFN - Link 1
-- -- Uplink 1
X2 (1/2/2) X4 (1/2/4) 2 (1/1/2) 4 (1/1/4) 6 (1/1/6) 8 (1/1/8) 10 (1/1/10) 12 (1/1/12)   14 (1/1/14) 16 (1/1/16) 18 (1/1/18) 20 (1/1/20) 22 (1/1/22) 24 (1/1/24)
Unused   VID 100 VID 200 VID 300
 
an-switch02  
Stack   Back-Channel Network]] Storage Network Internet-Facing Network
X1 (2/2/1) X3 (2/2/3)   1 (2/1/1) 3 (2/1/3) 5 (2/1/5) 7 (2/1/7) 9 (2/1/9) 11 (2/1/11)   13 (2/1/13) 15 (2/1/15) 17 (2/1/17) 19 (2/1/19) 21 (2/1/21) 23 (2/1/23)
To an-switch01
X3
To an-switch01
X1
an-a01n01
BCN - Link 2
an-pdu01 an-a02n01
BCN - Link 2
an-ups01 an-striker02
BCN - Link 1
an-a01n01
SN - Link 2
  an-a02n01
SN - Link 2
an-a01n01
IFN - Link 2
an-a02n01
IFN - Link 2
an-striker01
IFN - Link 2
-- --
-- -- an-a01n02
BCN - Link 2
an-pdu02 an-a02n02
BCN - Link 2
an-ups02 -- an-a01n02
SN - Link 2
  an-a02n02
SN - Link 2
an-a01n02
IFN - Link 2
an-a02n02
IFN - Link 2
-- -- Uplink 2
X2 (2/2/2) X4 (2/2/4) 2 (2/1/2) 4 (2/1/4) 6 (2/1/6) 8 (2/1/8) 10 (2/1/10) 12 (2/1/12)   14 (2/1/14) 16 (2/1/16) 18 (2/1/18) 20 (2/1/20) 22 (2/1/22) 24 (2/1/24)
Unused   VID 100 VID 200 VID 300

Cabling a pair of ICX6610-24

In this example, we're showing how to cable up a pair of ICX6610-48 switches in a rather unusual configuration. It is driving three Anvil! pairs. However, only the first is running at 10 Gbps while the second and third pair are running at 1 Gbps.

Note: This is the only case where a foundation pack will support three Anvil! pairs. In all other cases, only two pairs should be used for any given foundation pack in order to maximise isolation in large installations.
The Brocade ICX 6610-24. Picture by Brocade.
an-switch01    
Stack   BCN SN IFN   BCN SN IFN   Back-Channel Network   Storage Network Internet-Facing Network
XL1 (1/3/1) XL2-5 (1/3/2~5)                 1 (1/1/1) 3 (1/1/3) 5 (1/1/5) 7 (1/1/7) 9 (1/1/9) 11 (1/1/11) 13 (1/1/13) 15 (1/1/15) 17 (1/1/17) 19 (1/1/19) 21 (1/1/21) 23 (1/1/23)
To an-switch02
XL6
To an-switch02
XL7~10
an-a01n01
IPMI
an-a02n01
BCN - Link 1
an-a02n01
IPMI
an-a03n01
BCN - Link 1
an-a03n01
IPMI
an-striker01
BCN - Link 1
an-a02n01
SN - Link 1
an-a03n01
SN - Link 1
an-a02n01
IFN - Link 1
an-a03n01
IFN - Link 1
an-striker01
IFN - Link 1
--
XL6 (1/3/6) XL7~10 (1/3/7~10) X1 (1/2/1) X2 (1/2/2) X3 (1/2/3) X4 (1/2/4) X5 (1/2/5) X6 (1/2/6) X7 (1/2/7) X8 (1/2/8) 2 (1/1/2) 4 (1/1/4) 6 (1/1/6) 8 (1/1/8) 10 (1/1/10) 12 (1/1/12) 14 (1/1/14) 16 (1/1/16) 18 (1/1/18) 20 (1/1/20) 22 (1/1/22) 24 (1/1/24)
To an-switch02
XL1
To an-switch02
XL2~5
an-a01n01
BCN - Link 1
an-a01n01
SN - Link 1
an-a01n01
IFN - Link 1
-- an-a01n02
BCN - Link 1
an-a01n02
SN - Link 1
an-a01n02
IFN - Link 1
-- an-a01n02
IPMI
an-a02n02
BCN - Link 1
an-a02n02
IPMI
an-a03n01
BCN - Link 1
an-a03n02
IPMI
an-striker01
BCN - Link 1
an-a02n02
SN - Link 1
an-a03n02
SN - Link 1
an-a02n02
IFN - Link 1
an-a03n02
IFN - Link 1
an-striker02
IFN - Link 1
Uplink 1
Stack VID 100 VID 200 VID 300 VID 100 VID 200 VID 300 VID 100 VID 200 VID 300
 
an-switch02  
Stack   BCN SN IFN   BCN SN IFN   Back-Channel Network   Storage Network Internet-Facing Network
XL1 (2/3/1) XL2-5 (2/3/2~5)                 1 (2/1/1) 3 (2/1/3) 5 (2/1/5) 7 (2/1/7) 9 (2/1/9) 11 (2/1/11) 13 (2/1/13) 15 (2/1/15) 17 (2/1/17) 19 (2/1/19) 21 (2/1/21) 23 (2/1/23)
To an-switch01
XL6
To an-switch01
XL7~10
an-pdu01 an-a02n01
BCN - Link 2
an-ups01 an-a03n01
BCN - Link 2
Maintenance an-striker01
BCN - Link 2
an-a02n01
SN - Link 2
an-a03n01
SN - Link 2
an-a02n01
IFN - Link 2
an-a03n01
IFN - Link 2
an-striker01
IFN - Link 2
Maintenance
XL6 (2/3/6) XL7~10 (2/3/7~10) X1 (2/2/1) X2 (2/2/2) X3 (2/2/3) X4 (2/2/4) X5 (2/2/5) X6 (2/2/6) X7 (2/2/7) X8 (2/2/8) 2 (2/1/2) 4 (2/1/4) 6 (2/1/6) 8 (2/1/8) 10 (2/1/10) 12 (2/1/12) 14 (2/1/14) 16 (2/1/16) 18 (2/1/18) 20 (2/1/20) 22 (2/1/22) 24 (2/1/24)
To an-switch01
XL1
To an-switch01
XL2~5
an-a01n01
BCN - Link 2
an-a01n01
SN - Link 2
an-a01n01
IFN - Link 2
-- an-a01n02
BCN - Link 2
an-a01n02
SN - Link 2
an-a01n02
IFN - Link 2
-- an-pdu02 an-a02n02
BCN - Link 2
an-ups02 an-a03n01
BCN - Link 2
-- an-striker01
BCN - Link 2
an-a02n02
SN - Link 2
an-a03n02
SN - Link 2
an-a02n02
IFN - Link 2
an-a03n02
IFN - Link 2
an-striker02
IFN - Link 2
Uplink 1
Stack VID 100 VID 200 VID 300 VID 100 VID 200 VID 300 VID 100 VID 200 VID 300
 

Understanding Brocade Logical Port Numbers

Brocade uses the follow scheme for naming their ports:

  • stack ID/module/port

The Stack ID is simply the integer representing the ID number shown when you run 'show stack'.

The module number is static across all switches. They are:

  • 1: Standard RJ-45 1 Gbps copper ports.
  • 2: Ports used for stacking. Typically these are only used for stacking.
  • 3: The SFP+ fiber or copper ports that normally run at 1 Gbps but can be licensed to run at 10 Gbps.

The port number is the simple integer representing the port. This is the number found above or below the physical port on the switch.

You see these x/y/z numbers in the table above, and they will be the method used below to identify ports and port ranges.

Here is an example of how ports are represented:

ICX6610-24 Switch(config)#show interfaces brief

Port    Link    State   Dupl Speed Trunk Tag Pvid Pri MAC            Name      
1/1/1   Down    None    None None  None  No  1    0   cc4e.24b9.4b74          
1/1/2   Down    None    None None  None  No  1    0   cc4e.24b9.4b75          
1/1/3   Down    None    None None  None  No  1    0   cc4e.24b9.4b76          
1/1/4   Down    None    None None  None  No  1    0   cc4e.24b9.4b77          
1/1/5   Down    None    None None  None  No  1    0   cc4e.24b9.4b78          
1/1/6   Down    None    None None  None  No  1    0   cc4e.24b9.4b79          
1/1/7   Down    None    None None  None  No  1    0   cc4e.24b9.4b7a          
1/1/8   Down    None    None None  None  No  1    0   cc4e.24b9.4b7b          
1/1/9   Down    None    None None  None  No  1    0   cc4e.24b9.4b7c          
1/1/10  Down    None    None None  None  No  1    0   cc4e.24b9.4b7d          
1/1/11  Down    None    None None  None  No  1    0   cc4e.24b9.4b7e          
1/1/12  Down    None    None None  None  No  1    0   cc4e.24b9.4b7f          
1/1/13  Down    None    None None  None  No  1    0   cc4e.24b9.4b80          
1/1/14  Down    None    None None  None  No  1    0   cc4e.24b9.4b81          
1/1/15  Down    None    None None  None  No  1    0   cc4e.24b9.4b82          
1/1/16  Down    None    None None  None  No  1    0   cc4e.24b9.4b83          
1/1/17  Down    None    None None  None  No  1    0   cc4e.24b9.4b84          
1/1/18  Down    None    None None  None  No  1    0   cc4e.24b9.4b85          
1/1/19  Down    None    None None  None  No  1    0   cc4e.24b9.4b86          
1/1/20  Down    None    None None  None  No  1    0   cc4e.24b9.4b87          
1/1/21  Down    None    None None  None  No  1    0   cc4e.24b9.4b88          
1/1/22  Down    None    None None  None  No  1    0   cc4e.24b9.4b89          
1/1/23  Down    None    None None  None  No  1    0   cc4e.24b9.4b8a          
1/1/24  Down    None    None None  None  No  1    0   cc4e.24b9.4b8b          
1/2/1   Up      Forward Full 40G   None  No  N/A  0   cc4e.24b9.4ba5          
1/2/2   Up      Forward Full 10G   None  No  N/A  0   cc4e.24b9.4ba6          
1/2/6   Up      Forward Full 40G   None  No  N/A  0   cc4e.24b9.4ba7          
1/2/7   Up      Forward Full 10G   None  No  N/A  0   cc4e.24b9.4ba8          
1/3/1   Down    None    None None  None  No  1    0   cc4e.24b9.4ba9          
1/3/2   Up      Forward Full 10G   None  No  1    0   cc4e.24b9.4baa          
1/3/3   Down    None    None None  None  No  1    0   cc4e.24b9.4bab          
1/3/4   Down    None    None None  None  No  1    0   cc4e.24b9.4bac          
1/3/5   Down    None    None None  None  No  1    0   cc4e.24b9.4bad          
1/3/6   Up      Forward Full 10G   None  No  1    0   cc4e.24b9.4bae          
1/3/7   Down    None    None None  None  No  1    0   cc4e.24b9.4baf          
1/3/8   Down    None    None None  None  No  1    0   cc4e.24b9.4bb0          
mgmt1   Down    None    None None  None  No  None 0   cc4e.24b9.4b74

Forming the VLANs

TODO: These are adaptations of output from an ICX6610-48. Redo to capture direct output.

To start, lets take a look at the current VLAN configuration.

ICX6610-24 Switch#configure terminal
ICX6610-24 Switch(config)#show vlan
Total PORT-VLAN entries: 1
Maximum PORT-VLAN entries: 64

Legend: [Stk=Stack-Id, S=Slot]

Not much!

So lets get started.

VLAN Configuration on the ICX6450-24

Note: This will apply to any 24-port switch, which is the majority of all installs.

First, we will build the BCN VLAN, which will use VID 100.

ICX6450-24 Switch(config)#vlan 100 name bcn

The BCN spans ports 1 through 10 on both switches, which are expressed as 1/1/1 to 1/1/10 and 2/1/1 to 2/1/10 for switch 1 and 2, respectively. We can assign these two ranges to this VLAN as one command.

ICX6450-24 Switch(config-vlan-100)#untag ethernet 1/1/1 to 1/1/10 ethernet 2/1/1 to 2/1/10
Added untagged port(s) ethe 1/1/1 to 1/1/10 ethe 2/1/1 to 2/1/10 to port-vlan 100.

The SN spans ports 11 to 14 on each switch, which are expressed as 1/1/11 to 1/1/14 and 2/1/11 to 2/1/14 for switch 1 and 2, respectively. We can assign these two ranges to this VLAN as one command as well. First though, we need to define the SN VLAN and switch to it.

ICX6450-24 Switch(config-vlan-100)#vlan 200 name sn
ICX6450-24 Switch(config-vlan-200)#untag ethernet 1/1/11 to 1/1/14 ethernet 2/1/11 to 2/1/14
Added untagged port(s) ethe 1/1/11 to 1/1/14 ethe 2/1/11 to 2/2/14 to port-vlan 200.

Last, the IFN spans ports 15 to 24 on each switch, which are expressed as 1/1/15 to 1/1/24 and 2/1/15 to 2/1/24 for switch 1 and 2, respectively. We can assign these two ranges to this VLAN as one command as well. Again, we need to define the IFN VLAN and switch to it.

ICX6450-24 Switch(config-vlan-100)#vlan 300 name ifn
ICX6450-24 Switch(config-vlan-300)#untag ethernet 1/1/15 to 1/1/24 ethernet 2/1/15 to 2/1/24
Added untagged port(s) ethe 1/1/15 to 1/1/24 ethe 2/1/15 to 2/1/24 to port-vlan 300.

Done! Exit out of the VLAN configuration:

ICX6450-24 Switch(config-vlan-300)#exit

Let's see what it looks like now:

ICX6450-24 Switch(config)#show vlan
Total PORT-VLAN entries: 4
Maximum PORT-VLAN entries: 64

Legend: [Stk=Stack-Id, S=Slot]

PORT-VLAN 1, Name DEFAULT-VLAN, Priority level0, Spanning tree On
 Untagged Ports: None
   Tagged Ports: None
   Uplink Ports: None
 DualMode Ports: None
 Mac-Vlan Ports: None
     Monitoring: Disabled
PORT-VLAN 100, Name bcn, Priority level0, Spanning tree On
 Untagged Ports: (U1/M1)   1   2   3   4   5   6   7   8   9  10 
 Untagged Ports: (U2/M1)   1   2   3   4   5   6   7   8   9  10 
   Tagged Ports: None
   Uplink Ports: None
 DualMode Ports: None
 Mac-Vlan Ports: None
     Monitoring: Disabled                                         
PORT-VLAN 200, Name sn, Priority level0, Spanning tree On
 Untagged Ports: (U1/M3)  11  12  13  14 
 Untagged Ports: (U2/M3)  11  12  13  14 
   Tagged Ports: None
   Uplink Ports: None
 DualMode Ports: None
 Mac-Vlan Ports: None
     Monitoring: Disabled
PORT-VLAN 300, Name ifn, Priority level0, Spanning tree On
 Untagged Ports: (U1/M1)  15  16  17  18  19  20  21  22  23  24 
 Untagged Ports: (U2/M1)  15  16  17  18  19  20  21  22  23  24 
   Tagged Ports: None
   Uplink Ports: None
 DualMode Ports: None
 Mac-Vlan Ports: None
     Monitoring: Disabled

Perfectly perfect. Save the changes and then we're done.

ICX6450-24 Switch(config)#write memory
Write startup-config done.

Automatic copy to member units:  2
Flash Memory Write (8192 bytes per dot) .
Flash to Flash Done.

Done!

VLAN Configuration on the ICX6610-24

Note: This will apply to any 8+24-port switch, only applies to the ICX6610.

First, we will build the BCN VLAN, which will use VID 100.

ICX6450-24 Switch(config)#vlan 100 name bcn

The BCN spans SFP+ ports ports 1 and 5, plus RJ45 ports 1 through 12 on both switches. These are expressed as:

  • SPF+, Switch 1: 1/3/1 and 1/3/5
  • SPF+, Switch 2: 2/3/1 and 2/3/5
  • RJ45, Switch 1: 1/1/1 to 1/1/12
  • RJ45, Switch 2: 2/1/1 to 2/1/12

We can assign these two ports plus the two ranges to this VLAN as one command.

ICX6610-24 Switch(config-vlan-100)#untag ethernet 1/3/1 ethernet 1/3/5 ethernet 2/3/1 ethernet 2/3/5 ethernet 1/1/1 to 1/1/12 ethernet 2/1/1 to 2/1/12
Added untagged port(s) ethe 1/3/1 ethe 1/3/5 ethe 2/3/1 ethe 2/3/5 ethe 1/1/1 to 1/1/12 ethe 2/1/1 to 2/1/12 to port-vlan 100.

The SN spans SFP+ ports ports 2 and 6, plus RJ45 ports 13 through 16 on both switches. These are expressed as:

  • SPF+, Switch 1: 1/3/2 and 1/3/6
  • SPF+, Switch 2: 2/3/2 and 2/3/6
  • RJ45, Switch 1: 1/1/13 to 1/1/16
  • RJ45, Switch 2: 2/1/13 to 2/1/16

We can again assign these two ports plus the two ranges to this VLAN as one command. First though, we need to define the SN VLAN and switch to it.

ICX6610-24 Switch(config-vlan-100)#vlan 200 name sn
ICX6610-24 Switch(config-vlan-200)#untag ethernet 1/3/2 ethernet 1/3/6 ethernet 2/3/2 ethernet 2/3/6 ethernet 1/1/13 to 1/1/16 ethernet 2/1/13 to 2/1/16
Added untagged port(s) ethe 1/3/2 ethe 1/3/6 ethe 2/3/2 ethe 2/3/6 ethe 1/1/13 to 1/1/16 ethe 2/1/13 to 2/1/16 to port-vlan 200.

Finally, the biggest VLAN.

The IFN spans SFP+ ports ports 3 to 4 and 7 to 8, plus RJ45 ports 17 through 24 on both switches. These are expressed as:

  • SPF+, Switch 1: 1/3/3 to 1/3/4 and 1/3/7 to 1/3/8
  • SPF+, Switch 2: 2/3/3 to 2/3/4 and 2/3/7 to 2/3/8
  • RJ45, Switch 1: 1/1/17 to 1/1/24
  • RJ45, Switch 2: 2/1/17 to 2/1/24

This will be a long one, but we can still express this in a single command. First, create and switch to the new VLAN.

ICX6610-24 Switch(config-vlan-200)#vlan 300 name ifn
ICX6610-24 Switch(config-vlan-300)#untag ethernet 1/3/3 to 1/3/4 ethernet 1/3/7 to 1/3/8 ethernet 2/3/3 to 2/3/4 ethernet 2/3/7 to 2/3/8 ethernet 1/1/17 to 1/1/24 ethernet 2/1/17 to 2/1/24
Added untagged port(s) ethe 1/3/3 to 1/3/4 ethe 1/3/7 to 1/3/8 ethe 2/3/3 to 2/3/4 ethe 2/3/7 to 2/3/8 ethe 1/1/17 to 1/1/24 ethe 2/1/17 to 2/1/24 to port-vlan 300.

Done! Exit out of the VLAN configuration:

ICX6610-24 Switch(config-vlan-300)#exit

Let's see what it looks like now:

ICX6610-24 Switch(config)#show vlan
Total PORT-VLAN entries: 4
Maximum PORT-VLAN entries: 64

Legend: [Stk=Stack-Id, S=Slot]

PORT-VLAN 1, Name DEFAULT-VLAN, Priority level0, Spanning tree On
 Untagged Ports: None
   Tagged Ports: None
   Uplink Ports: None
 DualMode Ports: None
 Mac-Vlan Ports: None
     Monitoring: Disabled
PORT-VLAN 100, Name bcn, Priority level0, Spanning tree On
 Untagged Ports: (U1/M1)   1   2   3   4   5   6   7   8   9  10  11  12 
 Untagged Ports: (U1/M3)   1   5 
 Untagged Ports: (U2/M1)   1   2   3   4   5   6   7   8   9  10  11  12 
 Untagged Ports: (U2/M3)   1   5 
   Tagged Ports: None
   Uplink Ports: None
 DualMode Ports: None
 Mac-Vlan Ports: None
     Monitoring: Disabled                                         
PORT-VLAN 200, Name sn, Priority level0, Spanning tree On
 Untagged Ports: (U1/M1)  13  14  15  16 
 Untagged Ports: (U1/M3)   2   6 
 Untagged Ports: (U2/M1)  13  14  15  16 
 Untagged Ports: (U2/M3)   2   6 
   Tagged Ports: None
   Uplink Ports: None
 DualMode Ports: None
 Mac-Vlan Ports: None
     Monitoring: Disabled
PORT-VLAN 300, Name ifn, Priority level0, Spanning tree On
 Untagged Ports: (U1/M1)  17  18  19  20  21  22  23  24 
 Untagged Ports: (U1/M3)   3   4   7   8 
 Untagged Ports: (U2/M1)  17  18  19  20  21  22  23  24 
 Untagged Ports: (U2/M3)   3   4   7   8 
   Tagged Ports: None
   Uplink Ports: None
 DualMode Ports: None
 Mac-Vlan Ports: None
     Monitoring: Disabled

Perfectly perfect. Save the changes and then we're done.

ICX6610-24 Switch(config)#write memory
Write startup-config done.

Automatic copy to member units:  2
Flash Memory Write (8192 bytes per dot) .
Flash to Flash Done.

Changing a Port's VLAN Assignment

In order to change a port from one VLAN to another, it must first to untagged as VLAN ID 1, then it can be re-assigned to the new VLAN ID.

### (config-vlan-200) no untag ethernet 2/1/15 to 2/1/24
### (config-vlan-200) show vlan
Note: A port in VID 1 will simply change to the newly assigned VID. Ports in other VIDs need to be reset with no untag ... before reconfiguring for the new VLAN.



 

Any questions, feedback, advice, complaints or meanderings are welcome.
Alteeve's Niche! Enterprise Support:
Alteeve Support
Community Support
© Alteeve's Niche! Inc. 1997-2024   Anvil! "Intelligent Availability®" Platform
legal stuff: All info is provided "As-Is". Do not use anything here unless you are willing and able to take responsibility for your own actions.