Cisco Port Configuration Best Practices

I run into inconsistent network configurations wherever I go, from customers who just lets everything live in the native VLAN to ones who horrendously over-complicate things.  For the majority of Cisco deployments I have a simple set of configuration standards I adhere to.  You will always get corner-case requiring special configurations, but I find that the below works very well for most use cases.

Standard Access Port Configuration

This configuration is applied to ports connecting to standard end-user equipment, like PC’s, printers etc.
switchport mode access
switchport access vlan 5
spanning-tree portfast
spanning-tree bpduguard enable

The first two lines are self-explanatory, it’s an access port living in vlan 5.  Portfast is something we configure on an access port, which tells STP not to bother and just put the port in a forwarding state, as opposed to taking time to go through the listening and learning states. 
BPDU’s are basically STP messages exchanged between switches, therefore a BPDU is not something we expect to see on a normal access-port.  BPDUguard tells the switch that if it receives a BPDU (for example because someone connected an unauthorized switch), it should shut the port down.  Finally BPDUFilter tells the switch to not send or receive BPDU’s on ports configured as portfast.

ESXi Host Port Configuration

ESXi uses internal vSwitches and PortGroups which allows for VM’s running on one host to reside in different VLANs
switchport mode trunk
switchport trunk allowed vlan 5-10,200
spanning-tree port type edge trunk
spanning-tree bpduguard enable
Fairly straightforward, we configure our link as a trunk which carries vlan 5 to 10 and 200.  We then tell the switch that even though it’s a trunk we’re no connecting to another switch so no need to worry about STP on the port.  Lastly bpduguard protects us against incorrect cabling – if the port receives a BPDU (which will happen if you inadvertently hook it up to another switch) it will shut down.
 

Standard Trunk Port Configuration

In the Cisco world a trunked link is a link that carries multiple VLANs.  Not to be confused with link aggregation, which in Cisco parlance is called an Port-Channel.
switchport mode trunk
switchport trunk encapsulation dot1q
switchport trunk native vlan 10
switchport trunk allowed vlan 10-15
Both sides of the trunk needs to have the same default VLAN, by default the native VLAN is VLAN 1, but in all but the simplest deployments you will have to change this.  We can also do VLAN pruning on our trunks, that is only carry certain VLANs accross our trunk.

Port-Channel Configuration

It is possible to aggregate multiple links between two switches and treat them as a single link.  This gives us link redundancy and bandwidth increases.  As a rule we do not use LACP to ensure compatibility with, for example, vSphere vSwitches.
interface Port-channel10
switchport trunk encapsulation dot1q
switchport mode trunk
Once again simplicity is the name of the game.  We create the interface, set the encapsulation (not necessary, strictly speaking since Cisco defaults to dot1q) and set it as a trunk.  Of course it does not need to be a trunk link, this is optional.

Directly Connecting a Brocade 815 HBA to a EMC VNX5300

I’m busy with a project which involves getting two ESXi hosts hooked up to a VNX5300 configured in block mode.  The order we placed with Dell specified Emulex 12000 HBA’s, but Dell got creative and shipped Brocade 815’s instead.  Only problem was that they didn’t work when directly connected to the front-end ports on the VNX.  I’m documenting the symptoms here as well, so that the next person does not have to battle for two days.

The Symptoms

When directly connecting the HBA’s to the VNX fiber ports the following events pop up in the SP event logs

• EV_VirtualArrayFeature::_mergeInternalObjects() - No parent for HBA,
• EV_TargetMapEntry::GetHostInitiatorPort() - NULL HBAPort pointer

Running NaviSECCli.exe -Address 172.20.10.27 port -list –sfpstate outputs the following:

SP Name:             SP A
SP Port ID:          1
SP UID:              50:06:01:60:BE:A0:72:F9:50:06:01:61:3E:A0:72:F9
Link Status:         Up
Port Status:         Online
Switch Present:      NO
SFP State:           Online

This tells us that things are fine on a physical layer, but not much else is happening higher up the stack.

The Fix

First we need to upgrade the HBA firmware to version 3.1.  There are various OS specific ways to do it, easiest is probably to download the livecd from Brocade.  Since this HBA is not on the ESXi 5.1 HCL we need to install the driver.  You need to install at least the v3.1   I include the steps for the sake of completeness

1. Enable SSH on your ESXi host
2. Use scp for Windows or the following command from a linux / max host:  scp brocade_driver_esx50_v3-1-0-0.tar root@<ip address>:/tmp
3. SSH into your ESXi host and navigate to the /tmp folder with cd /tmp
4. Execute tar xf brocadedriveresx50_v3-1-0-0.tar
5. Execute ./brocade_install_esxi.sh
6. Wait for the installation to finish (takes about 1 – 2 mins) and reboot host once done

Now we need to configure the HBA for direct connection, or more technically,  FC-AL mode

1. SSH into your ESXi host and navigate to /opt/brocade/bin/ by entering cd /opt/brocade/bin/
2. ./bcu port --topology 1/0 loop
3. ./bcu port —disable 1/0
4. ./bcu port —enable 1/0
5. ./bcu port --topology 2/0 loop
6. ./bcu port —disable 1/0
7. ./bcu port —enable 1/0

Your ESXi host should now show up as a host on the VNX where you can add it to a storage group and assign LUNs.

Partial/No Redundancy on iSCSI Datastores

Expensive fiber SANs are not price-compatible with a lot of my clients, therefore a lot of my time is spent in iSCSI environments.  I’ve noticed in all instances that the Multipathing Status for all my iSCSI datastores are Partial/No Redundancy when viewed on the Storage Views tabs in vCenter.  This bothers me because I always go to great lengths to ensure that I set up my iSCSI multipathing correctly.

I therefore breathed a big sigh of relief when I discovered that this behaviour is a bug as confirmed by VMware Technical Support. The rule for displaying the “Multipathing Status” is as follows:

Full Redundancy – If you have 2 separate adapters and 2 separate paths to the datastore
Partial/No Redundancy – If there is one path which is Up
Unknown – If there is at least one path with an “Unknown” status
All Paths Down – No way to reach the datastore

You will always only have one adapter when using a software iSCSI Initiator, this implies a single point of failure which gives us the dreaded “Partial/No Redundancy” status.  So as things stand now software iSCSI will always be displayed with a degraded status.  Methinks VMWare should develop separate rules / algorithms for fiber and iSCSI SANs…