The Overheating EVA

I had a "interesting" experience recently, namely had a EVA 4400 overheating due to environmental issues (fancy-talk for Aircon failure).  The client phoned me, complaining that half of their Hyper-V VM's are not running.  Further investigation revealed that the CSV's were offline.  Hmmm, this was getting serious.  I logged into Command View and saw that most of my VDisks were faulted, this was due in no small matter to the fact that all the drives in one of my shelves were faulted.

Event Logs

I had a look at the relevant EVA logs and discovered the following relevant entries:

•  Temperature within an HSV300 controller becoming too hot.
  View corrective actions.  Corrective action code: 2e
•  A drive enclosure temperature sensor out of range condition has been reported by one of the drive enclosure link modules.
• A physical disk drive has disappeared.
  View corrective actions.  Corrective action code: 42
• A Volume has transitioned to the MISSING state.
  View corrective actions.  Corrective action code: bf

What Happened

In retrospect it was a fairly simple sequence of events, as evidenced by the entries above.  The Air Conditioner failed, which caused the temperature within the Drive Shelf to rise (this is the HSV300 controller referred to in the event log).  To prevent damage to itself, the drive then switched itself off, which prompted the log entry about the physical drive disappearing.

We then started seeing volumes transitioning to the missing state, i.e. our VDisks went missing.  Hardly surprising considering that the drives containing them switched themselves off.

Resolution

1. Restored Air Conditioning (goes without saying I guess)
2. Powered off the EVA and all attached disk shelves
3. Powered on disk shelves and waited for the Numeric ID LED's at the back to display the proper IDs.
4. Powered up the Controller
5. Lo and behold!  All the previously failed physical disks came on-line, meaning that my missing VDisks also made a most welcome return
6. Unfortunately my Hyper-V Hosts still couldn't access the Vdisks, so I had to unpresent and re-present them via Command View.  I assume the EVA assigned new WWN's to the LUNs.
7. I re-scanned for storage from the Disk Management MMC on the Hyper-V Hosts
8. Brought the Disks and CSV's online via cluster manager
9. Started up the VM's

Conclusion

This was quite a harrowing experience, obviously.  What struck me as ridiculous is that HP does not have *ANY* thermal shutdown logic / capabilities on the EVA controller itself.  It keeps on trucking till the drives themselves fail, causing a very ungraceful failure of the VDisks.  There is also no guarantee that your drives and VDisks will come back online.  In essence - if your EVA overheats there is a distinct possibility that you lose your Data.  Caveat Emptor...

Move WSUS server to a new server

Namibia is a third world....errr....developing country.  So apart from rampant (55%) unemployment it also means that we are bandwidth starved.  This also means that some businesses are running on capped accounts, so attempts need to be made to conserve bandwidth.  I've had to move WSUS servers to new hardware or VM's a couple of times, and needless to say it's a huge time-sink and waste of bandwidth to re-download all updates every time you move your WSUS server.  Thus I've come up with a way to move WSUS without downloading tons of patches - steps are outlined below for your enjoyment.

1. Install WSUS on your new server, making sure to select the option to use the existing Windows Internal Database
2. During the Choose Upstream Server of the configuration wizard, be sure to select Synchronise from another Windows Server Update Services Server
3. Ensure that the This is a replica of the upstream server check box is selected.  This ensures that existing approvals, settings, computers and groups are maintained
4. Complete this Wizard
5. Your new replica server will synchronise with your upstream server.  This is what we're talking about - no re-downloading many GB's worth of patches!  N.B.  Wait for this process to complete before carrying on with step 6.
6. Now change your Update Source and Proxy Server settings to Synchronise from Microsoft Update
7. Now for the magic bit.  Download and install the WSUS 3 API Samples and Tools on both your old and new WSUS servers
8. Open up a CMD Prompt and navigate to "C:\Program Files\Update Services 3.0 API Samples and Tools\WsusMigrate\WsusMigrationExport" folder on your old WSUS server
9. Run "wsusmigrationexport.exe WSUS_Settings.xml" to export the settings. This will backup your approvals and target groups to an XML file
10. Transfer the WSUS_Settings.xml created above to your new WSUS server
11. Again navigate to "C:\Program Files\Update Services 3.0 API Samples and Tools\WsusMigrate\WsusMigrationImport" folder (on the new WSUS  server). Run "wsusmigrationimport.exe WSUS_Settings.xml All None"
12. Review and compare settings on your two WSUS servers, ensuring that they match each other
13. Update the relevant GPO's to ensure clients are pointing to the new WSUS server

One last point, I've noticed that if you are not using GPO's to assign your computers to Computer Groups all your clients will get stuck into the Unassigned Computers group.  You'll have to manually sort them into the appropriate groups again.  Bummer, but you really should be using Computer Groups....

Back to Basics - Create a LUN and present it as a Cluster Shared Volume

You know when you do something and you think to yourself "Oi, I should really document this"?  And then you go "Nahhh, this is easy peasy lemon squeezy - I won't forget" and a month later you have to repeat the task and wonder of wonders, you did forget.  Well Goddammit - never will I have to wonder again how to create a LUN on a HP EVA 4400 SAN and use said LUN (or Vdisk in vHP's vVirtual vSpeak vWhere vEverything vStarts vWith a vStupid vV). vWhere...er I mean where was I.  Ahh yes, lets create a vDisk...

Create a vDisk on the EVA SAN

1. Log into Command View (what a cool name, sounds like you can bomb some backwater country into the Stone Age with a mere misplaced mouse-click)
2. Expand your Virtual Disks Folder
3. Click Create vDisk
4. Name your disk, then give it a size and RAID level
5. Click Create vDisk

Present the vDisk to your Hosts

1. Navigate to your newly created vDisk and click the Presentation tab
2. Click Present and select all hosts to present the vDisk to.  For some obscure reason every 4400 I've worked on would only allow me to present to four hosts / HBA's at a time, anything more and it errors out

Initialise and format the vDisk

1. Open the Disk Management MMC on one of your SAN-connected Hyper-V hosts
2. Right-click the Disk Management node and click Rescan on the context menu
3. Right-click on the newly appeared unallocated disk and select Bring Online
4. Right-click the disk and click Initialize
5. Right-click (how's that middle finger doing?) on the unallocated storage and click Create Simple Volume
6. Complete the wizard, but do not assign a drive letter

Create a Cluster Shared Volume using our vDisk

1. Open Failover Cluster Manager
2. Go to the Storage node and click Add a Disk in the Actions pane
3. Select the disk you wish to add and click OK
4. Right-click the Disk and click Properties - now give it a descriptive name.  Because I say so
5. Go to Cluster Shared Volumes and click Add Storage in the Actions pane
6. Select the disks you wish to add and click OK
7. The CSV will now magically appear in your ClusterStorage namespace.  Navigate there and give it a descriptive name
8. You've just greated a CSV full of win, which can now host a highly-available VM! 

The low-down on VLAN's

I find myself having to explain the concepts behind VLAN's a lot, so I've decided to do a blog post about it.

The Wikipedia article defines a VLAN as: "a group of hosts with a common set of requirements that communicate as if they were attached to the same broadcast domain, regardless of their physical location. A VLAN has the same attributes as a physical LAN, but it allows for end stations to be grouped together even if they are not located on the same network switch. Network reconfiguration can be done through software instead of physically relocating devices".

PROTOCOL CONCEPTS

We have regular "untagged" Ethernet frames. ("Frame" is the data-link (layer 2) term for what some protocols calls a "packet" or "datagram".) These are the same Ethernet frames from before VLANs were invented.

Then we have "tagged" frames. These take a regular Ethernet frame and add a marker -- the "tag". The tag itself contains the VLAN number (along with a priority code). Thus, a tagged frame is associated with a particular VLAN number.

Inside the frame, the tag is basically just a reserved Ethernet frame type, followed by the tag information, followed by the rest of the original frame (complete with its original frame type). So to network nodes which are *not* VLAN aware, a tagged frame looks just like a regular Ethernet frame, but with an "unknown" frame type.

For computers, a VLAN-aware network stack will usually turn each VLAN into a separate network interface, as if each VLAN was a different network card with a different cable. The "stock" network stack included with Win 2000/XP does *not* include VLAN capability, so they can only handle untagged frames. Tagged frames will generally be silently ignored. Third-party software (often included with high-end network cards) can add VLAN capability. I'm not sure about 2003/Vista/2008.

Switches which are not VLAN-aware, along with any repeaters, will treat tagged frames exactly like they do any other frame.

SWITCH CONCEPTS

Switches which are VLAN-aware generally let you control which ports are associated with which VLAN(s). The terminology, commands, etc., will vary with manufacturer. I'm most familiar with the HP ProCurve managed witches, so I'll speak mostly to that. The concepts should carry over to other manufacturers.

With HP, you can tell the switch that each VLAN is tagged, untagged, or not available for each port. Internally, frames are *always* associated a VLAN number. The configuration just tells the switch which VLANs go with which ports, and which VLAN number to use for untagged frames.

When the switch receives an untagged frame, it needs to know what VLAN to associate it with. If the port has an untagged VLAN associated it, the fame will go to that VLAN. If there is no untagged VLAN associated with the port, the frame is dropped. This is why a port can have (at most) one untagged VLAN associated with it -- how could the switch know which one to use otherwise?

When the switch receives a tagged frame, it simply checks to see if that VLAN is associated with the port. If so, it accepts it; if not, it drops it. This is basically just an access control mechanism

When the switch has a frame to *transmit*, it only considers ports which are associated with the frame's VLAN. If a port is configured as tagged for the frame's VLAN, it transmits the frame with the VLAN tag. If a port is configured as untagged for the VLAN, the frame is transmitted on that port without a VLAN tag.

Real-World Implementations

Okay, enough conceptual stuff, how do we use it? Below I'll describe a implementation I had to do for a client recently.

VLAN's were numbered and split up as follows -

1 – Reserved

2 – Main company private network

3 – Guest network

4 – ISP uplink

5 – IP telephony

I used separate VLANs for security, manageability, and robustness. I don't want guests to be able to probe/sniff/screw-up the corporate network. I didn't want the cable modem link (which is full of garbage traffic from other subscribers) on the main LAN. I want the phones to have higher priority over regular data, and to have their own DHCP server, so if the "data side" of the house goes down, they still have phones.

I don't, as a rule, use VLAN 1 because that's the default on most things. This also avoids confusion.

VLAN 2 is where most traffic is in this scenario. Most ports are configured as untagged for VLAN 2. The computers, printers, etc., plugged into those ports aren't even aware they are on a VLAN. All the employee workstations are like this, as are most servers. The syntax to configure this is easy. For example, if I want port 1 to be untagged for VLAN 2 like this:

vlan 2 untagged 1

VLAN 3 is for guests (visitors). For obvious reasons, I don't let them plug in to the main network. They get a separate IP subnet, a separate DHCP server, and bypass much of the network filtering. Again, all untagged, and connected nodes are not aware of the VLAN.

VLAN 4 was created because the client has a ADSL Internet feed they use for employee web browsing. Circumstances meant the modem had to go in a different building from the main server/network room. So the cable modem plugs into a particular switch port, which uses VLAN 4 to carry it to the firewall in a different building. Again, untagged, unaware.

VLAN 5 is for premises voice-over-IP. The phones include a built-in switch for "daisy chain", and are VLAN-aware. Switch ports connected to phones are tagged for VLAN 5 and untagged for VLAN 2, so you can plug a PC (not VLAN aware) into the phone. Traffic is kept separate, and voice traffic is given priority. The central phone equipment has an Ethernet port which we connect to one of the core switches, with that switch port untagged for VLAN 5. So the central equipment just thinks it is talking to a regular LAN.

For example, suppose port 20 was the central phone equipment, and port 7 had a VoIP phone. We would do:

vlan 2 untagged 7

vlan 5 tagged 7

vlan 5 untagged 20

For links between switches, each VLAN is configured as *tagged* for those ports, with *no* untagged VLAN. Those ports are only used to carry traffic between VLAN-aware switches. (Cisco would call this a "trunk port", IIRC.) The HP syntax for (e.g.) port 24 as a switch link would be:

vlan 2 tagged 24

vlan 3 tagged 24

vlan 4 tagged 24

vlan 5 tagged 24

Finally, our firewall, which *is* VLAN-aware, is configured as tagged for all VLANs on its port, with no untagged VLAN. Virtual network interfaces are configured in the firewall OS, and all the IP configuration and filtering rules are done with those virtual interfaces, not the physical interface. This also means our firewall only has one network cable plugged into it. Configures the same as a switch link, really.

Hope this helps someone!

How to add a Node to a Windows 2008 R2 Hyper-V Failover Cluster

Dammit - me and my lack of documentation. Worked with Hyper-V for the first time six months ago (I'm predominantly a VMware dude). I Had to add a node to a failover cluster this morning, and found myself wishing I had documented it better the first time around. Got it sorted after a bit of scratching around; as you'll see below I've also finally gotten round to documenting the process. The below assumes that you've hooked it presented appropriate the appropriate LUNs on the SAN etc. etc....

1. Install the OS (no sh!t Sherlock). Make sure you run the same edition on all nodes. Also installed any required MPIO software that may be required by your SAN.
2. Join the server to the same domain that your other Cluster nodes are members of
   
3. Enable and configure Windows Updates
4. Make sure that your network adapter naming convention is the same as on the other nodes. This will make management easier
5. Install the Hyper-V role, reboot when prompted
6. Make sure to have naming consistency in your Virtual network. In other words, exactly the same name and capitalization
7. Install the Failover Clustering feature, reboot when prompted
8. Physically connect the server to your SAN
9. Join the node to the cluster. This can be done via the Failover Cluster Management MMC: Right click your cluster name, select Add Node, enter the server name and click OK. The command line alternative is "cluster.exe /cluster:Cluster_Name /add /node:Node_Name"

It might be necessary to change your quorum settings afterwards - this Microsoft webpage explains the options rather nicely. The quorum settings are changed via the Fialover Cluster Manager MMC, in the Actions pane there's a Configure Cluster Quorum Settings link.

How to Expand a VHD on Hyper-V 2008 R2

This is a fairly basic task that needs to be done fairly frequently, but I've never documented it anywhere so it always takes me 10 minutes longer than what it should. The client where I had to do this today happens to run a HP EVA 4400 Fiber SAN, so I will include the instructions for the EVA side of things as well. Without further ado...

Expanding a existing LUN on an HP EVA 4400 SAN

1. Log into Command View (HP's delightfully named SAN configuration solution)
2. Navigate to the the Virtual Disks folder, more specifically to the Vdisk you want to expand
3. Click the Vdisk, and wait 10 seconds for the incredibly laggy Command View interface to catch up
4. Type your new, preferred size in the box cunningly labelled "Capacity"
5. Click "Save"

Expanding a VHD

1. Use Cluster Resource Manager to determine who owns the Disk Resource you wish to expand
2. Once identified, go to "Disk Management" on the owner
3. Right-click the Disk Management node, and select "Rescan Disks". The un-allocated space should show up now
4. Right-click the volume you wish to expand and select "Extend Volume". Click Next 2 times and finally Finish
5. Fire up SCVMM and shut down the VM whose VHD you want to expand
6. Click the "Hardware Configuration" tab, and select the Disk you want to expand
7. Check the "Expand this disk" checkbox and enter the desired size
8. Click OK

This should take care expanding the VHD, but this does not mean the underlying OS will automatically make use of the added capacity, no sirrreeee. In my case the VM was running Windows 2008 Server R2, and this is what I had to do....

Using Diskpart to Extend a Partition

1. Click Start, type CMD and press Enter
2. In the resulting Command Prompt, type "Diskpart"
3. Select the correct drive and partition to perform the extend operation on. In my case it was Disk 1, Partition 1. The correct commands in my case were "Select disk 1" and "Select Partition 1". The "List disk" and "List partition" commands, surprisingly enough, will list your existing disks and partitions
4. Now that we've got the correct disks and partitions selected, we simply give the "Extend" command. This will extend the disk using all the contiguous free space available

And with that done, you should be the proud Admin of a newly extended VHD!

Enabling Jumbo Frames in vSphere

Hi Kids, todays post is brought to you by the letter J. I previously mentioned a vSphere deployment I had to do, connecting to a EMC AX4-5i iSCSI SAN. Once I got the storage hooked up to my ESX hosts I of course wanted to enable jumbo frames.

Jumbo Frames, for the uninitiated heathens out there, is basically a Ethernet frame with a payload of more than 1500 bytes, up to a typical maximum of 9000 bytes per frame. Why would I want to do this? Performance, in a nutshell. The only requirement would be that your switch supports Jumbo Frames, which I believe most, if not all, mid to high-end kit does. So let's get down to business, shall we?

Switch Configuration

I'm fairly proficient with Cisco and HP networking gear, so I'll give a quick rundown of the commands needed to enable jumbo frames on their kit:

• Cisco: Go into conf t mode and enter the following command "system mtu jumbo 9000". Once that is done issue a "reload" command to reboot your switch
• HP Procurve: On these babies Jumbo frames need to be enabled per VLAN. Execute the following command: "vlan # jumbo"

That takes care of the networking side of things, let's move on to the ESX...

Configure Jumbo frames in vSphere

1. Log on to your ESX host using your favorite SSH client
2. Change your chosen vSwitch MTU with the following command "esxcfg-vswitch -m 9000 vSwitch#" Replace vSwitch# with the name of the vSwitch you want to modify
3. Seeing as one cannot change an existing VMKernel port MTU, you will either have to remove and recreate your existing VMKernel port, or create a new one. To delete an existing VMKernel port, enter the following command: "esxcfg-vmknic -d -p VMKernelport"
4. Now let's add a VMKernel port called "iSCSI01" to our vSwitch, like so: "esxcfg-vswitch –A iSCSI01 vSwitch#"
5. Now we enable jumbo frames on our VMKernel port: "esxcfg-vmknic –a –i 192.168.x.x –n 255.255.255.0 –m 9000 iSCSI01". You do know you need to change the IP address and mask to reflect your environment, right?
   
6. Let's make sure we didn't screw up somewhere by running "esxcfg-vmknic –l". Verify that the MTU is set to 9000
7. All done! You can refer to my previous iSCSI Multipathing post to ensure Jumbo frame enabled iSCSI Multipathing goodness!

I have thought of merging this post with the iSCSI Multipathing post, but seeing as not all environments can/will support end to end Jumbo frames, I have decided to keep them seperate for the time being.