Partial/No Redundancy on iSCSI Datastores

Expensive fiber SANs are not price-compatible with a lot of my clients, therefore a lot of my time is spent in iSCSI environments.  I’ve noticed in all instances that the Multipathing Status for all my iSCSI datastores are Partial/No Redundancy when viewed on the Storage Views tabs in vCenter.  This bothers me because I always go to great lengths to ensure that I set up my iSCSI multipathing correctly.

I therefore breathed a big sigh of relief when I discovered that this behaviour is a bug as confirmed by VMware Technical Support. The rule for displaying the “Multipathing Status” is as follows:

Full Redundancy – If you have 2 separate adapters and 2 separate paths to the datastore
Partial/No Redundancy – If there is one path which is Up
Unknown – If there is at least one path with an “Unknown” status
All Paths Down – No way to reach the datastore

You will always only have one adapter when using a software iSCSI Initiator, this implies a single point of failure which gives us the dreaded “Partial/No Redundancy” status.  So as things stand now software iSCSI will always be displayed with a degraded status.  Methinks VMWare should develop separate rules / algorithms for fiber and iSCSI SANs…

ESX Hosts Disconnecting After Upgrade to vSphere 4.1

When you upgrade to vSphere 4.1 your hosts might start disconnecting from your vCenter Server with the following error message: A general system error occurred: internal error: vmodl.fault.HostCommunication.  Restarting the management agents does not resolve the error, nor does rebooting the host.  This VMware KB points to name resolution issues, but that is not at fault here.  The issue is vCenter Server cannot manage an ESX 4.1 host.

Workaround / Solution

Currently there are two solutions available:

1. Upgrade your vCenter Server to version 4.1.  (Once you've upgraded you'll have to remove the hosts from your inventory and re-add it - simply reconnecting didn't work in my case)
2. Downgrade your ESX hosts to version 4

Strangely enough I could not find this documented anywhere on the VMware Knowledge Base, even though it seems to be a pretty widely reported problem.

Upgrading to vSphere 4.1 via a SSH CLI Session

I was tasked with upgrading a clients vSphere installation from vSphere 4.0 to 4.1.  Due to various external factors the client couldn't make use of the vSphere Update Manager, so I had to do it old-school style from the command line.  Here's how to do it:


Download the required updates

1.  Navigate to http://downloads.vmware.com/d/info/datacenter_downloads/vmware_vsphere_4/4
2. Download pre-upgrade-from-ESX4.0-to-4.1.0-0.0.260247-release.zip
3. Download  upgrade-from-ESX4.0-to-4.1.0-0.0.260247-release.zip

Install the updates

1. Put your ESX host in maintenance mode with the following command: vimsh -n -e /hostsvc/maintenance_mode_enter
2. Install the pre-upgrade patch:  esxupdate update --bundle=pre-upgrade-from-ESX4.0-to-4.1.0-0.0.260247-release.zip
3. Install the actual upgrade patch: esxupdate update --bundle=upgrade-from-ESX4.0-to-4.1.0-0.0.260247-release.zip
4. Reboot the ESX host via the reboot command
5. Last step is to exit maintenance mode: vimsh -n -e /hostsvc/maintenance_mode_exit

 All of the above can of course be automated using the Update Manager, but for those occasions where it's not possible to use it the above will come in handy.

Enabling Jumbo Frames in vSphere

Hi Kids, todays post is brought to you by the letter J. I previously mentioned a vSphere deployment I had to do, connecting to a EMC AX4-5i iSCSI SAN. Once I got the storage hooked up to my ESX hosts I of course wanted to enable jumbo frames.

Jumbo Frames, for the uninitiated heathens out there, is basically a Ethernet frame with a payload of more than 1500 bytes, up to a typical maximum of 9000 bytes per frame. Why would I want to do this? Performance, in a nutshell. The only requirement would be that your switch supports Jumbo Frames, which I believe most, if not all, mid to high-end kit does. So let's get down to business, shall we?

Switch Configuration

I'm fairly proficient with Cisco and HP networking gear, so I'll give a quick rundown of the commands needed to enable jumbo frames on their kit:

• Cisco: Go into conf t mode and enter the following command "system mtu jumbo 9000". Once that is done issue a "reload" command to reboot your switch
• HP Procurve: On these babies Jumbo frames need to be enabled per VLAN. Execute the following command: "vlan # jumbo"

That takes care of the networking side of things, let's move on to the ESX...

Configure Jumbo frames in vSphere

1. Log on to your ESX host using your favorite SSH client
2. Change your chosen vSwitch MTU with the following command "esxcfg-vswitch -m 9000 vSwitch#" Replace vSwitch# with the name of the vSwitch you want to modify
3. Seeing as one cannot change an existing VMKernel port MTU, you will either have to remove and recreate your existing VMKernel port, or create a new one. To delete an existing VMKernel port, enter the following command: "esxcfg-vmknic -d -p VMKernelport"
4. Now let's add a VMKernel port called "iSCSI01" to our vSwitch, like so: "esxcfg-vswitch –A iSCSI01 vSwitch#"
5. Now we enable jumbo frames on our VMKernel port: "esxcfg-vmknic –a –i 192.168.x.x –n 255.255.255.0 –m 9000 iSCSI01". You do know you need to change the IP address and mask to reflect your environment, right?
   
6. Let's make sure we didn't screw up somewhere by running "esxcfg-vmknic –l". Verify that the MTU is set to 9000
7. All done! You can refer to my previous iSCSI Multipathing post to ensure Jumbo frame enabled iSCSI Multipathing goodness!

I have thought of merging this post with the iSCSI Multipathing post, but seeing as not all environments can/will support end to end Jumbo frames, I have decided to keep them seperate for the time being.

ESX 4 iSCSI Multipathing

There I was, setting up a small DR site for a client, hardware was a blinged up Proliant DL380, connecting to a sweet, sweet little EMC AX4-5i SAN. This being my first non-FC SAN, I obviously investigated the multipathing options. I was kind of (no, really!) taken aback when I discovered that it's not so straightforward to set up. What's a boy to do? I took a deep breath, bowed in the direction of the (not-so-sweet-anymore) AX4-5i SAN and unleashed the fury of my Google-Fu! Hee-Ya! About 3 hours later the fiendish ESX 4 and a AX4-5i had to yield to the fury of my touch typing - below I chronicle it for all eternity.

Setting up Multipathing
The DL380's I worked on here had an additional 4 port NIC installed, for a total of 8 1Gb ports. The 4 on-board ports I dedicated to the Service console, vmkernel et al, standard stuff. The 4 ports on the add-on NIC I used for iSCSI, perfect considering the AX4-5i also has 4 iSCSI ports. What I did then was set up a new vSwitch, added my 4 Nics to the vSwitch and then created 4 seperate VmKernel ports, so that we have a one-to-one mapping between the VMkernels and the NICs. Here's how to do that:

1. Create a new vSwitch that we're going to dedicate to iSCSI
2. Connect your NICs to this vSwitch by going vSwitch - Properties. Click the Network Adapters tab and click Add. Select your NIC's and click Next and Finish.
3. Now we create VMkernel ports for all our newly added adapters, like so: Go vSwitch - Properties and click the Ports tab and click Add. Select VMkernel, give it a nice label and a suitable IP address and click Finish. Create a VMkernel port for every adapter assigned to your iSCSI vSwitch.
4. Now the important bit, we're going to do the one-to-one VMkernel to NIC mapping: On the ports tab, select one of your new VMkernel port and click Edit.
5. Select the Nic Teaming Tab and check the Override vSwitch failover order
6. Make sure only one adapter is active, move all the other NIC's down to the Unused Adapters section
7. Rinse and repeat for every VMkernel port on your vSwitch

Alas, we're not done yet - now I'm going to introduce you to my little friend esxcli. This fun little guy will enable us to connect our newly created VMkernel ports to the ESX iSCSI initiator

Enable Software iSCSI Multipathing

1. From the CLI (console, ssh, pick your poison) run the following command esxcli swiscsi nic add -n vmk# -d vmhba33
2. Rinse and repeat for all your ports, verify your work by executing the following command: esxcli swiscsi nic list -d vmhba33
3. You can then rescan your iSCSI initiator via the vSphere GUI. You can also verify multpathing via the paths view for the vmhba33 adapter.

Happy Multipathing! It should be mentioned that iSCSI and Jumbo frames go hand-in-hand. I will do a post detailing that can of worms in the near future.