Backing up Hyper-V VMs using DPM with software VSS Providers

DPM is a terrific backup solution for the all-round Microsoft shop.  In my humble opinion it’s also one of the best ways to back up Hyper-V Virtual Machines.  It integrates beautifully with your SAN vendors hardware VSS providers, but if you don’t have the budget to splash out on the requisite SAN licenses, there is a way to do it in software.

The Challenge

When backing up VM’s residing on a CSV, you might run into the following DPM error message:

Failed to prepare a Cluster Shared Volume (CSV) for backup as another backup using the same CSV is in progress. (ID 32612 Details: Unknown error (0x8007173d) (0x8007173D))

The Solution

1. Run the DSConfig.ps1 script (included below) on any node in your CSV containing cluster
2. This will create a file called DataSourceGroups.xml in the same folder that you executed the DSConfig.ps1 script
3. Copy DataSourceGroups.xml to the %programfiles%\Microsoft DPM\DPM\Config folder on your DPM Server
4. This file needs to be updated every time you create and protect a VM using DPM
5. If a protection group has already been created for the virtual machines, execute the Modify Protection Group Wizard. If a protection group has not been created, create a new protection group and the job serialization described above will take effect.

DSConfig.ps1 Script

The updated script can be downloaded from here

This will allow you to get rid of the pesky alerts and streamline your backup process.  The optimum solution would be to use a hardware VSS provider, but this is not always financially feasible, or if you have an iSCSI SAN.

Securing HP Procurve Switches (or how to placate Auditors)

I was recently approached by an organization who suffered through an IT Systems and Security Audit.  Needless to say the audited highlighted a couple of holes in their system, and I was tasked with implementing the audit recommendations.  Part of my scope of work included securing their HP Procurve switches, this post will outline a couple of best-practices when securing your network equipment.  It is by no means exhaustive, but it should provide you with a good baseline.  All instructions assume you're logged into the switch and at the (config)# prompt

Defaults Configurations and Secure Alternatives

Telnet vs SSH
Telnet is an antiquated and insecure protocol; it sends all traffic across the wire in clear text. This includes user
names and passwords. Any attacker may eavesdrop on a Telnet session and obtain credentials.  Below are the commands detailing how to disable Telnet and and implement SSH.

1. crypto key generate ssh
2. ip ssh
3. no telnet-server

HTTP vs HTTPS
ProCurve switches and routers can, by default, be configured through the web interface. The HTTP
interface has the same limitations as Telnet. It is therefore recommended that the HTTPS interface be configured and the HTTP interface be disabled, like so:

1. crypto key generate cert 1024
2. web-management ssl
3. no web-management plaintext

TFTP vs SFTP & SCP
TFTP client and server should be disabled as they do not require any authentication. Secure File
Transfer Protocol (SFTP) and Secure Copy Protocol (SCP) are part of the SSH protocol suite. As such provide an encrypted session using public/private keys between client and server just like
SSH.  You configure it like so:

1. crypto key generate ssh
2. ip ssh filetransfer 
3. no tftp server
4. no tftp client

SNMPv1/2c vs. SNMPv3
SNMP version 2 is enabled by default.  SNMPv2 community names are sent across the network as clear text (similiar to Telnet).  SNMP version 3 overcomes these weaknesses via encryption.  Here's how to configure it:

1. snmpv3 enable
2. snmpv3 only

If, for whatever reason, you cannot run SNMP v3, it is possible to run SNMP v2 in restricted mode, i.e. provide Read-Only access to your switch:

1. snmp-server community restricted 

It is also possible to completely disable SNMP altogether:

1. no snmp-server enable

Authorised IP Managers
It’s possible to restrict IP access to the switch to certain management IP's only.  This is configured via the following command:

1. ip authorized-manager <IP address> mask  

Whilst not the be-all and end-all of security, the above should go a long way to ensure a reasonably secure base line for your HP Procurve network equipment.

Exchange 2010 ActiveSync Issue (or the dreaded HTTP Error 500)

Exchange ActiveSync (EAS for us smart people) is a pretty rocking way to get Smart phones hooked up to your Exchange Server.  9 times out of ten it "just works" if you follow the docs, but sometimes it doesn't.  Otherwise I wouldn't writing about it now, would I?

I had a rather perplexing case where EAS only worked for some users.  On the non-working users the only error message displayed was the rather unhelpful HTTP Error 500.  The only suggestion the usually excellent www.testexchangeconnectivity.com could offer was an old article applicable to Exchange 2003.

Diagnosing

The next step was to scour the Application log on the Exchange Server, and lo and behold, the following error was logged:

Source: MSExchange ActiveSync
Event ID: 1053
Level: Error


Exchange ActiveSync doesn't have sufficient permissions to create the "CN=User Name,OU=Admins,OU=Employees,OU=People,DC=domain,DC=local" container under Active Directory user "Active Directory operation failed on server01.domain.local. This error is not retriable. Additional information: Access is denied.
Active directory response: 00000005: SecErr: DSID-031521D0, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0
".
Make sure the user has inherited permission granted to domain\Exchange Servers to allow List, Create child, Delete child of object type "msExchangeActiveSyncDevices" and doesn't have any deny permissions that block such operations.

Righto.  So it would appear that to actually use EAS, information has to be written to the user account and to do this the Exchange Servers group needs to have permissions to the user object.  Further investigation revealed that the only accounts giving the error was Admin accounts.

AdminSDHolder and SDPROP are the culprits

Because the failing accounts has domain admins rights their User Object permissions were being reset by
AdminSDHolder, via a process called SDPROP.  I quote "Each Active Directory domain has an object called AdminSDHolder, which resides in the System container of the domain. The AdminSDHolder object has a unique Access Control List (ACL), which is used to control the permissions of security principals that are members of built-in privileged Active Directory groups (what I like to call "protected" groups). Every hour, a background process runs on the domain controller that holds the PDC Emulator operations master role. It compares the ACL on all security principals (users, groups and computer accounts) that belong to protected groups against the ACL on the AdminSDHolder object. If the size or the binary string is different, the security descriptor on the object is overwritten by the security descriptor from the AdminSDHolder object.."

Resolution

Because SDPROP runs every hour, the fix needs to be applied and a EAS sync done before an hour has gone past.  Here is how:

1. Launch the EMC from your Exchange CAS Server
2. Browse to Server Configuration and select your server
3. Go to the System Settings tab to see which Domain Controller is in use
4. Launch ADUC from the DC located in step 3 and make sure Advanced View is enabled
5. Right-click the user account and go to Properties
6. Click the Security tab, then click Advanced
7. Check the Include inheritable permission.... checkbox
8. OK out of all the dialog boxes
9. Sync your device

The sync should be successful, providing not more than an hour has passed between the sync and step 1.  Whilst this has been a very difficult issue to nail down, there is a lesson in there somewhere...DON'T USE PRIVILEGED ACCOUNTS FOR DAY-TO-DAY USE!!!

ESX Hosts Disconnecting After Upgrade to vSphere 4.1

When you upgrade to vSphere 4.1 your hosts might start disconnecting from your vCenter Server with the following error message: A general system error occurred: internal error: vmodl.fault.HostCommunication.  Restarting the management agents does not resolve the error, nor does rebooting the host.  This VMware KB points to name resolution issues, but that is not at fault here.  The issue is vCenter Server cannot manage an ESX 4.1 host.

Workaround / Solution

Currently there are two solutions available:

1. Upgrade your vCenter Server to version 4.1.  (Once you've upgraded you'll have to remove the hosts from your inventory and re-add it - simply reconnecting didn't work in my case)
2. Downgrade your ESX hosts to version 4

Strangely enough I could not find this documented anywhere on the VMware Knowledge Base, even though it seems to be a pretty widely reported problem.

Upgrading to vSphere 4.1 via a SSH CLI Session

I was tasked with upgrading a clients vSphere installation from vSphere 4.0 to 4.1.  Due to various external factors the client couldn't make use of the vSphere Update Manager, so I had to do it old-school style from the command line.  Here's how to do it:


Download the required updates

1.  Navigate to http://downloads.vmware.com/d/info/datacenter_downloads/vmware_vsphere_4/4
2. Download pre-upgrade-from-ESX4.0-to-4.1.0-0.0.260247-release.zip
3. Download  upgrade-from-ESX4.0-to-4.1.0-0.0.260247-release.zip

Install the updates

1. Put your ESX host in maintenance mode with the following command: vimsh -n -e /hostsvc/maintenance_mode_enter
2. Install the pre-upgrade patch:  esxupdate update --bundle=pre-upgrade-from-ESX4.0-to-4.1.0-0.0.260247-release.zip
3. Install the actual upgrade patch: esxupdate update --bundle=upgrade-from-ESX4.0-to-4.1.0-0.0.260247-release.zip
4. Reboot the ESX host via the reboot command
5. Last step is to exit maintenance mode: vimsh -n -e /hostsvc/maintenance_mode_exit

 All of the above can of course be automated using the Update Manager, but for those occasions where it's not possible to use it the above will come in handy.

Truncating SQL 2005 Log Files

Had a fun scenario recently where a customer (a Casino!) ran out of disk space on their SQL 2005 Server.  This in turn caused all their slot / video poker / whatever gambling machines to stop working.  Turns out old Bill Shakespeare had it all wrong, because it turns out that Hell Hath No Fury Like A Gambler Scorned!

Long story short - their SQL log file grew to gargantuan proportions, and yours truly had to whip them back into shape.  Here's how it went down!

1. Run the following stored procedure: "use your_db_name"
2. Followed by: "exec sp_helpfile".  This will return the physical names and attributes of files associated with your DB.  Record the DB and log filenames, without the path and extension
3.  Enter the folowwing commands
1. USE your_db_name
2. GO
3. BACKUP LOG your_db_name WITH TRUNCATE_ONLY
4. GO
5. DBCC SHRINKFILE (your_dblog_filename, 1) 
6. GO
7. DBCC SHRINKFILE (your_db_filename, 1)
8. GO
9. exec sp_helpfile

Step 9 should output the same info has Step 2, you can now compare the filesizes to see if the process was succesfull.

You might get an error "Cannot shrink log file because all logical log files are in use".  In that case you can follow the instruction here to resolve.  I've detailed the steps below, if you're too lazy to follow the link.

1. Open SQL Enterprise Manager
2. Right-click on the database you want to shrink and click Properties
3. from the Data Properties go to Options.
4. Set the Recovery Model to Simple and click OK and try to shrink the database

Your Database and Database log files should now shrink succesfully!

Using Veeam backup to relocate VM's

Nope, I didn't fudge up the title.  Veeam Backup and Replication is a wonderful product, allowing you to replicate vSphere VMs to a offsite Disaster Recovery location.  When disaster strikes, it's a pretty straight-forward process to fail over to your DR site.  It's what I call a forehead procedure - you only have to hit the spacebar with your forehead.    Thanks - I'll be here all night!  

What's not so intuitive and well documented is using Veeam Backup to move VM's to a different location, for example a Server Room / Data Center relocation *and then commiting those changes*, i.e. not failing back to Production.  The below steps assume we've replicated and failed over our VM's to our DR location already.

Ruan's Step By Step Guide on using Veeam Backup to relocate VM's

1. Delete all Veeam VM snapshots using the vSphere Snapshot Manager
2. By default your DR replica will be named "VMname_replica", rename it back to its original name, i.e. VMname
3. Remove the VM replica from the list of replicas in the Veeam Backup and Replication Console
4. Delete the Production -> DR replication job responsible for replication the VM in question.  Recreate it to reflect the new Source and Target locations
5. Delete the .vrb file from the VM datastore, as we will no longer be using these restore points
6. Delete the replica.vrb and running.rbk files
7. Pat yourself on the back - you've just done the easiest VM move you'll ever do!