Microsoft Network Load Balancing, and how to make it work with a Cisco Nexus

Microsoft Network Load Balancing has one or two “interesting” issues due to the way it does networking.  More specifically, it utilises a Multicast MAC address with a Unicast IP address.  Someone much smarter than me explains the ins and outs of the issue here. 

This issue manifests itself in that clients outside of the cluster’s LAN cannot connect to the IP.

Now that we understand the issue,  how do we resolve it?  The fixes listed in the article are all valid, however we can get this working by tweaking the config on our Nexus.

interface Vlan10
  no ip redirects
  ip address 192.168.10.1/24
  ip arp 192.168.10.100 03BF.0A03.00CF
  description Server Farm
  no shutdown

Your clients should now be able connect to the cluster IP address without issue.

Updating the firmware (DDOS) on a DataDomain

EMC has done a great job with DDOS 5.2, as such I’ve had to do a couple of upgrades recently.  The upgrade manual is quite a tome, so this is my notes, distilled.  I’ve added a couple of extra steps just to make sure there are no error conditions that can hamper the upgrade or the operation of the DataDomain.

Pre-Upgrade checks

Review the output of the below commands for any errors and warnings.

1. Reboot the DataDomain (I do it just because)
2. ssh into the DataDomain and issue the commands listed below
3. enclosure show powersupply 1
4. alert show current
5. reg show config.net (confirm the following settings)
1. noauto.enabled
2. noauto.speed
3. noauto.full_duplex
6. ifgroup
7. disk show state
8. disk show reliability-data
9. enclosure show all
10. enclosure show topology
11. system show hardware
12. system show ports
13. filesys status
14. log view debug/platform/kern.info

Upgrade Steps

Well after all that the steps to actually upgrade the DDOS is a bit of a let-down.

1. Copy your DDOS .rpm to /ddr/var/releases
1. You can do this via a NFS mount or you can upload via the GUI
2. ssh into the DataDomain and issue the commands listed below
3. system upgrade 5.2.2.4-370754.rpm
4. Monitor progress with “system upgrade status” or “log watch debug/platform/infra.log”

Post-Upgrade Steps

I guess strictly speaking these steps are not necessary, as with the pre-upgrade steps.  I do it just to ensure I have a nice neat base to work from

1. ssh into the DataDomain and issue the commands listed below
2. ost disable
3. ost enable
4. nfs disable
5. nfs enable

And that is all there is to it, mostly.

Cisco Port Configuration Best Practices

I run into inconsistent network configurations wherever I go, from customers who just lets everything live in the native VLAN to ones who horrendously over-complicate things.  For the majority of Cisco deployments I have a simple set of configuration standards I adhere to.  You will always get corner-case requiring special configurations, but I find that the below works very well for most use cases.

Standard Access Port Configuration

This configuration is applied to ports connecting to standard end-user equipment, like PC’s, printers etc.
switchport mode access
switchport access vlan 5
spanning-tree portfast
spanning-tree bpduguard enable

The first two lines are self-explanatory, it’s an access port living in vlan 5.  Portfast is something we configure on an access port, which tells STP not to bother and just put the port in a forwarding state, as opposed to taking time to go through the listening and learning states. 
BPDU’s are basically STP messages exchanged between switches, therefore a BPDU is not something we expect to see on a normal access-port.  BPDUguard tells the switch that if it receives a BPDU (for example because someone connected an unauthorized switch), it should shut the port down.  Finally BPDUFilter tells the switch to not send or receive BPDU’s on ports configured as portfast.

ESXi Host Port Configuration

ESXi uses internal vSwitches and PortGroups which allows for VM’s running on one host to reside in different VLANs
switchport mode trunk
switchport trunk allowed vlan 5-10,200
spanning-tree port type edge trunk
spanning-tree bpduguard enable
Fairly straightforward, we configure our link as a trunk which carries vlan 5 to 10 and 200.  We then tell the switch that even though it’s a trunk we’re no connecting to another switch so no need to worry about STP on the port.  Lastly bpduguard protects us against incorrect cabling – if the port receives a BPDU (which will happen if you inadvertently hook it up to another switch) it will shut down.
 

Standard Trunk Port Configuration

In the Cisco world a trunked link is a link that carries multiple VLANs.  Not to be confused with link aggregation, which in Cisco parlance is called an Port-Channel.
switchport mode trunk
switchport trunk encapsulation dot1q
switchport trunk native vlan 10
switchport trunk allowed vlan 10-15
Both sides of the trunk needs to have the same default VLAN, by default the native VLAN is VLAN 1, but in all but the simplest deployments you will have to change this.  We can also do VLAN pruning on our trunks, that is only carry certain VLANs accross our trunk.

Port-Channel Configuration

It is possible to aggregate multiple links between two switches and treat them as a single link.  This gives us link redundancy and bandwidth increases.  As a rule we do not use LACP to ensure compatibility with, for example, vSphere vSwitches.
interface Port-channel10
switchport trunk encapsulation dot1q
switchport mode trunk
Once again simplicity is the name of the game.  We create the interface, set the encapsulation (not necessary, strictly speaking since Cisco defaults to dot1q) and set it as a trunk.  Of course it does not need to be a trunk link, this is optional.

Issues and Workarounds – vSphere Site Recovery Manager with the NetApp Storage Replication Adapter 2.01

I have just completed a project where I had to Install and configure VMware vSphere Site Recovery Manager.  Storage was provided by NetApp FAS and V-Series filers, thus I had to use the NetApp provided Storage Replication Adapter.  As of the time of this writing the latest version was 2.01.  True to form I ran into a couple of bugs, which took a bit of figuring out.

Unable to add a controller: “Error: SRA command ‘discoverArrays’ failed”

Execute the following commands on your filers

• options httpd.admin.enable on
• options httpd.enable on
• options httpd.admin.ssl.enable off

Error when adding an Array Pair: “Internal error: std::exception 'class Dr::Xml::XmlValidateException' "Element 'SourceDevices' is not valid for content model: '(SourceDevice,)”

There are two solutions to this issue

• Downgrade back to NetApp SRA version 2.0.0
• Manually include the lists of volumes you want discovered by the SRA.  You’ll need to do this on both controllers in the pair.

This is a documented bug

Reprotect Job fails after recovering to Disaster Recovery Site

The SRM / SRA timeouts seems a bit aggressive to me.  This is highlighted when you do a reprotect on a failed over Protection Group.  Part of the task sequences is to reverse the direction of replication, but this fails consistently due to the SRM not waiting long enough for this reversal to take place.

You can kludge it by:

• Re-running the reprotect until it works
• Manually refresh the Array Manager while the Re-Protect job is running

Recovered DataStores have snap-xxx prefixes

More of a cosmetic irritant than a true bug, I wanted this fixed nonetheless.

1. Within SRM, right-click your site and select Advanced Settings
2. Click StorageProvider
3. Select the storageProvider.fixRecoveredDatastoresNames check box

Tip

I would suggest increasing your SRM SAN provider timeout settings to something a bit more sane, like double.  Instructions can be found here.

Also make sure that the ALUA settings on your iGroups in both the protected and recovery sites are the same.

Configuring Backup Exec 2012 to backup to a DataDomain 670 using the OST plugin

DataDomain Configuration

First things first, let’s verify that we have the DD Boost is correctly configured on the DataDomain

1. Log into your DataDomain Appliance and navigate to System Settings – Licenses and verify the correct licenses, it should look like this:
   
   
2. Navigate to Data Management – DD Boost – Settings and verify that DD Boost is enabled, like so:
   
3. Now click on the + sign to add your BackupExec host.  Enter the hostname and click OK
   
4. Next we need to create a DD Boost Storage Unit.  Navigate to Data Management – DD Boost – Storage Units and click Create.  Enter a descriptive name and configure any quotas, if desired.  Click OK
   
5. Now we need to enable an interface for DD Boost.  Navigate to Data Management – DD Boost – IP Network.  Highlight and edit an interface group and tick the “Enabled” check box in the resultant dialogue box and click OK.
   

That takes care of the configuration on the DataDomain side of things.  Let’s move over to Backup Exec

Backup Exec 2012 Configuration

1. Download and install the latest version of the EMC DataDomain Boost for Symantec OpenStorage (Version 2.5.0.3-314845) at the time of writing.  This file is available from the DataDomain support site (Powerlink Login required)
2. Open up the Backup Exec console, select the Storage tab and click Configure Storage
3. The type of storage is OpenStorage
   
4. Enter a name and description for the DataDomain
   
5. Select the DataDomain provider (the DataDomain only shows up once you completed the OST plugin installation as per Step 1)
   
6. Enter the connection details for the your DataDomain device
   
7. Enter the Storage Location configured on the DataDomain (BackupExec in our case)
   
8. Select the amount of concurrent operations allowed to the DataDomain
   
9. Click Finish on the Summary screen and confirm that you would like to restart the Backup Exec services
   

With that completed you’ll be able to select the DataDomain as a deduplication target (as opposed to B2D device).

Rebuilding the Exchange 2010 Offline Address Book (OAB) from scratch

IT peeps supporting Microsoft Exchange can be divided into two groups – those that have experienced problems with the OAB and those that will.  If you’re in the first camp – I feel your pain.  Those of you lucky enough to be in the second camp – file this away, it might come in handy…no I’m not being facetious.
The process below will blow away your OAB and create a new one, so be mindful.  FWIW I’ve never had any issues with this process.  This is especially effective if you’ve screwed up on the public folder replication during Exchange Migrations (don’t ask).
We rebuild the Exchange 2010 OAB like so:

Create a new Offline Address Book object

1. Open the Exchange Management Console (EMC) and navigate to Organisation Configuration – Mailbox
2. Click the Offline Address Book tab.  Right click in the blank area and click New Offline Address Book
3. Give your OAB a different name than the existing one
4. Select your Exchange 2010 MBX server as the OAB generation server
5. Check Include the default Global Address Lists option
6. Check Enable Web-based Distribution as well as the Enable public folder distribution option
7. Finish the wizard

Restart Exchange Services

1. Restart the Microsoft Exchange System Attendant service
2. Restart the Microsoft Exchange File Distribution service

Update and set the OAB as default Offline Address Book

1. Right-click your newly created OAB and click Update.  This can take a couple of minutes, confirm successful completion via your Application log
2. Right-click the OAB from step 1 and and click Set as default

Assign the OAB to the affected users databases

1. Open the Exchange Management Shell (EMS)
2. Execute the following:
   Get-MailboxDatabase |  set-MailboxDatabase  -OfflineAddressBook "%your new OAB%"
3. Wait for Outlook to complete it’s OAB download cycle (can be as much as 24 hours)

You can safely delete the old OAB once you’ve verified that your clients are successfully downloading the newly created one.

Performing a Non-Disruptive Disk and Shelf Firmware upgrade on a NetApp FAS2040

I received a mail from NetApp this morning, pointing my attention to KB ID 7010014.  In a nutshell, there is a drive firmware upgrade available which the lowers the drive failure rates.  AutoSupport has also been nagging me about out of date DS4243 shelf firmware, so I thought this would be a perfect opportunity to upgrade it all in in one go.  It goes without saying that the upgrades must to have zero impact on client access.  The process below was run on Data Ontap Release 8.1 7-Mode.

Update the Disk Qualification Package

1. Download the latest DQP from the NetApp support site
2. Extract the files and copy it to the /etc folder on your filer, overwriting the existing files

Update the Disk Shelf Software

1. Download the appropriate disk shelf software upgrade from the NetApp support site
2. Extract and copy it to the /etc/shelf_fw folder on your filer
3. Run the options shelf.fw.ndu.enable command and verify it is set to on
   • If not, enable it with the options shelf.fw.ndu.enable on command
4. Execute the storage download shelf command to update the shelf firmware and enter yes when prompted

Update the Disk Firmware

1. Download the latest disk firmware from the NetApp support site
2. Verify the following, otherwise you will not be able to do a non-disruptive upgrade
   • Aggregates need to be RAID-DP or mirrored RAID4
   • You need to have functioning spares
3. Run the options raid.background_disk_fw_update.enable command and verify it is set to on
   • If not, enable it with the options raid.background_disk_fw_update.enable on command
4. Extract and copy the disk firmware to the /etc/disk_fw folder on your filer
5. The upgrade should start automatically in a couple of minutes
6. Repeat for both controllers

Verifying the upgrade

Execute the sysconfig –v command to verify successful installation
And there we go, we have non-disruptively upgraded the firmware and disk drives in our filer!