Problem accessing CSV from passive Cluster Node

I recently had a perplexing problem where Cluster Shared Volumes in a Hyper-V cluster were not working correctly.  The volumes were only accessible from the node currently owning the volume.  Attempts to access the volumes from any of the other nodes resulted Windows Explorer hanging indefinitely.  Enabling maintenance or redirected mode made no difference.

Event ID 5120 was logged:  Cluster Shared Volume 'Volume1' ('Cluster Disk 1') is no longer available on this node because of 'STATUS_BAD_NETWORK_PATH(c00000be)'. All I/O will temporarily be queued until a path to the volume is re-established.
Event ID 5142 also occurred:  Cluster Shared Volume 'Volume1' ('Cluster Disk 1') is no longer accessible from this cluster node because of error 'ERROR_TIMEOUT(1460)'. Please troubleshoot this node's connectivity to the storage device and network connectivity.

I could ping all nodes over both the Production and Heartbeat network links, and I could access file shares from any node on any node.

The Problem
After much troubleshooting I realised I disabled both File and Print Sharing and Client for Microsoft Networks on the Heartbeat NIC on all nodes.  This is a best practice drummed into me since working on Microsoft Clustering when it was still code-named Wolfpack.

The Resolution
I enabled File and Print Sharing and Client for Microsoft Networks and immediately afterwards all my Cluster Shared Volumes started functioning as expected.

The Explanation
It’s documented in MS KB Article 2008795.  When accessing a CSV volume from a passive (non-coordinator) node, the disk I/O to the owning (coordinator) node is routed through a 'preferred' network adapter and requires SMB be enabled on that network adapter. For SMB connections to work on these network adapters, the aforementioned protocols must be enabled.  Ugh.

Setting up a KMS server on Server 2008 R2

Today we’ll deal with setting up a Microsoft Key Management Server (KMS).  a KMS is used to activate Microsoft Volume Licensed products such as Windows 7, Office 2010 and Windows server 2008 R2, amongst others.

a KMS server activates a client for a period of 180 days.  The activated machine will communicate with the KMS every 7 days to renew it’s activation information.  It then resets the license counter back to 180 days if successful.  If not it attempts to background connect to the KMS every 2 hours.

If, after 180 days, the machine has not been able to contact the KMS it will go into the 30 day grace period and notify the user.  After that the machine will enter a reduced functionality mode until it can again connect to a KMS.

That was quite a mouthful – so let’s get down to setting up a KMS on a Windows 2008 R2 host.  In addition we’ll also set it up so that it can activate Office 2010 clients.

Setting up a KMS

1. Activate Windows with a KMS key.  This will automatically configure the server as a KMS
2. Download the Office 2010 KMS Host License Pack
3. Enter your KMS host key when prompted (you will get this key from your Microsoft Volume Licensing website)
4. Make sure to allow the Key Management Service through the Windows Firewall

Verify that KMS is published in DNS

nslookup -type=srv _vlmcs._tcp.<your DNS domain>

Checking the KMS status on your KMS

From an elevated command prompt, type SLMGR.vbs /dlv

Checking the license and activation status on a client

slmgr.vbs –dli

a KMS goes a significant way to easing administrative burden, so go ahead and set it up, it’s as easy-peasy!

Start-up Sequence for VM’s in a Hyper-V Environment

I’ve had customers phone me several times with regards to powering up their servers after an outage.  If not done properly you can run into a situation whereby you cannot log onto a host to start a VM, because the virtualised DC is still down.  Not good.

So, if me or my clients are recovering from a shutdown I usually do the following:

1. Power on the Hyper-V server hosting the VM that holds the PDC emulator FSMO role.  The VM should preferably also be a GC and DNS server and refer to itself for either primary or secondary DNS
2. Sign on to the Hyper-V host using a Local Administrative Account.
3. Start the VM.
4. Log out of the Hyper-V host once the VM has booted successfully.
5. Log into the Hyper-V host using a “proper” Domain Account.
6. Start any other virtualised DC’s / GC’s you might have.
7. Ensure you have at least one DC / GC running in each AD site.
8. Boot up the rest of your environment

Another option (pointed out to me in the comments section), is to set startup delays on your VM’s so that your DC’s starts up before your other servers.  This seems very basic, but it’s amazing how quickly common sense goes out of the window when the pressure is on to get an environment up and running. 

Another way of dealing with this (my preferred way) is to have the PDC FSMO role on a physical machine.  If you go this route also ensure that your physical machine is a GC.

Publishing Remote Desktop Gateway (RDG) with TMG 2010

I recently had the pleasure of creating an Remote Desktop Services (RDS), Remote Desktop Gateway (RDG) and RemoteApp environment for a client.  This was a bit more technical and involved than I originally envisioned, no thanks to the scant documentation that exists.  I will detail all that in a later blog post, for now I will focus on publishing your RDG, RDS and RA environment through a Microsoft TMG 2010 Firewall.

First we have to create an SSL Listener

1. Specify an IP address for the Listener
2. Enable both HTTP and SSL connections
3. For HTTP to HTTPS redirection select redirect all traffic from HTTP to HTTPS
4. On the Certificates tab select “Use a single certificate for this web listener” and select an appropriate certificate
5. Authentication should be set to “No Authentication”

Now we create the actual publishing rule:

1. Allow
2. From Anywhere
3. To – Your RDSG IP or Host – Forward original host header – request appears to come from TMG
4. Traffic HTTPS
5. Listener – Select the one we created earlier
6. Public name – This is the Public DNS name
7. Paths should be /rdsweb/* and /rpc/*
8. Authentication delegation – “No Delegation, client may authenticate directly”

These were the steps I had to take to successfully and securely publish the client's RDSG to the internet.  Once again I found the existing documentation to be lacking in the extreme.  Hope this helps someone out there.

Installing Windows 7 / 2008 R2 from a USB Stick

Too much of my life is spent staring at install screens - no more I say!  That time is better spent looking at the blinkenlights, drinking coffee or browsing slashdot.  One way of significantly speeding up install time is using a USB device instead of CD/DVD media.  Here is how to create bootable USB Installation media for Windows 7 and Server 2008 R2.

• Launch the DiskPart utility by typing diskpart at the Start Menu.  If your system does not have diskpart you can download it from here
• Run the list disk command to, surprisingly, list the disks in your system
• Now run select disk % where the "%" is actually the number of your USB drive (obtained above).
• Run clean.
• Now run create partition primary.
• Let's set the partition to active by entering active
• Now we format the drive with the FAT32 filesystem via the format fs=fat32 quick command
• Run the assign command to assign a drive letter to our USB device
• Copy the entire contents of the installation DVD to your USB device (a simple drag and drop will do)
• Plug the USB drive into the target system and proceed with the (hopefully faster) installation.

Configuring the relevant BIOS options is left as an exercise to the reader. 

Creating a Hyper-V Cluster after the fact, or, how to preserve and add existing VMs to a Cluster.

I was faced with an interesting challenge recently. a Client was running two standalone Hyper-V hosts with about 4 VMs each running on a Storage Area Network(SAN).  I installed the SAN previously to provide increased IO performance for their SCADA (Citect, for those taking notes) system. 

This was essentially a very effective proof of concept as far as the client was concerned and they wished to take advantage of the more advanced features offered by Clustered Hyper-V (stuff like live migration etc.).

This posed a challenge, because we needed to convert the LUNs occupied by the VM’s to highly available Cluster Shared Volumes (CSV’s).  In Hyper-V, a VM needs to be hosted on a CSV in order to be made highly-available.  So off I went trying to figure out a non disruptive way to convert all my LUNs to CSV, without losing any data.  This is what I came up with.

1. Shut down your VM(s)
2. Open Disk Management on your Hyper-V host and remove drive letter from LUN hosting the VM
3. Open Failover Cluster Manager (FCM) –> Storage –> Add Disk –> Select Disk from Step 2 –> Click OK
4. Still in FCM - Go to Cluster Shared Volume - Add Storage – select the disk you added in Step 3
5. Open up Hyper-V Manager on the same host (notice the VM status is critical because you removed the drive letter).  Remove the VM
6. Create a new VM, opt to store it under the %systemdrive%\ClusterStorage folder which was created automatically when you performed Step 4.  VERY IMPORTANT – Do not add any disks to the VM!
7. Right click the VM you created in Step 6 and choose Edit Settings.  Add the original VM’s disks (boot drive to be added to IDE controller 0).  The existing VHD’s will be found in %systemdrive%\ClusterStorage
8. Open FCM – Go to Services and Applications – in the Action pane select Configure a Service or Application –> select Virtual Machine –> Check the VM created in Step 6 –> Complete Wizard
9. Ensure that the VM is connected to the correct network in Hyper-V Manager
10. Because we are connecting a new NIC to the VM you will have to re-specify the IP address inside the VM once the VM has started up

Rinse and repeat for all existing VMs you want to make highly available.  The Microsoft way would be to export the VM’s and import it again.  Nothing wrong with that, apart from the fact that it takes a lot of time and storage to do, depending on the size of the VM.  My way is quick and easy and it works!

Back to Basics - Create a LUN and present it as a Cluster Shared Volume

You know when you do something and you think to yourself "Oi, I should really document this"?  And then you go "Nahhh, this is easy peasy lemon squeezy - I won't forget" and a month later you have to repeat the task and wonder of wonders, you did forget.  Well Goddammit - never will I have to wonder again how to create a LUN on a HP EVA 4400 SAN and use said LUN (or Vdisk in vHP's vVirtual vSpeak vWhere vEverything vStarts vWith a vStupid vV). vWhere...er I mean where was I.  Ahh yes, lets create a vDisk...

Create a vDisk on the EVA SAN

1. Log into Command View (what a cool name, sounds like you can bomb some backwater country into the Stone Age with a mere misplaced mouse-click)
2. Expand your Virtual Disks Folder
3. Click Create vDisk
4. Name your disk, then give it a size and RAID level
5. Click Create vDisk

Present the vDisk to your Hosts

1. Navigate to your newly created vDisk and click the Presentation tab
2. Click Present and select all hosts to present the vDisk to.  For some obscure reason every 4400 I've worked on would only allow me to present to four hosts / HBA's at a time, anything more and it errors out

Initialise and format the vDisk

1. Open the Disk Management MMC on one of your SAN-connected Hyper-V hosts
2. Right-click the Disk Management node and click Rescan on the context menu
3. Right-click on the newly appeared unallocated disk and select Bring Online
4. Right-click the disk and click Initialize
5. Right-click (how's that middle finger doing?) on the unallocated storage and click Create Simple Volume
6. Complete the wizard, but do not assign a drive letter

Create a Cluster Shared Volume using our vDisk

1. Open Failover Cluster Manager
2. Go to the Storage node and click Add a Disk in the Actions pane
3. Select the disk you wish to add and click OK
4. Right-click the Disk and click Properties - now give it a descriptive name.  Because I say so
5. Go to Cluster Shared Volumes and click Add Storage in the Actions pane
6. Select the disks you wish to add and click OK
7. The CSV will now magically appear in your ClusterStorage namespace.  Navigate there and give it a descriptive name
8. You've just greated a CSV full of win, which can now host a highly-available VM! 

How to add a Node to a Windows 2008 R2 Hyper-V Failover Cluster

Dammit - me and my lack of documentation. Worked with Hyper-V for the first time six months ago (I'm predominantly a VMware dude). I Had to add a node to a failover cluster this morning, and found myself wishing I had documented it better the first time around. Got it sorted after a bit of scratching around; as you'll see below I've also finally gotten round to documenting the process. The below assumes that you've hooked it presented appropriate the appropriate LUNs on the SAN etc. etc....

1. Install the OS (no sh!t Sherlock). Make sure you run the same edition on all nodes. Also installed any required MPIO software that may be required by your SAN.
2. Join the server to the same domain that your other Cluster nodes are members of
   
3. Enable and configure Windows Updates
4. Make sure that your network adapter naming convention is the same as on the other nodes. This will make management easier
5. Install the Hyper-V role, reboot when prompted
6. Make sure to have naming consistency in your Virtual network. In other words, exactly the same name and capitalization
7. Install the Failover Clustering feature, reboot when prompted
8. Physically connect the server to your SAN
9. Join the node to the cluster. This can be done via the Failover Cluster Management MMC: Right click your cluster name, select Add Node, enter the server name and click OK. The command line alternative is "cluster.exe /cluster:Cluster_Name /add /node:Node_Name"

It might be necessary to change your quorum settings afterwards - this Microsoft webpage explains the options rather nicely. The quorum settings are changed via the Fialover Cluster Manager MMC, in the Actions pane there's a Configure Cluster Quorum Settings link.