Configuring Backup Exec 2012 to backup to a DataDomain 670 using the OST plugin

DataDomain Configuration

First things first, let’s verify that we have the DD Boost is correctly configured on the DataDomain

1. Log into your DataDomain Appliance and navigate to System Settings – Licenses and verify the correct licenses, it should look like this:
   
   
2. Navigate to Data Management – DD Boost – Settings and verify that DD Boost is enabled, like so:
   
3. Now click on the + sign to add your BackupExec host.  Enter the hostname and click OK
   
4. Next we need to create a DD Boost Storage Unit.  Navigate to Data Management – DD Boost – Storage Units and click Create.  Enter a descriptive name and configure any quotas, if desired.  Click OK
   
5. Now we need to enable an interface for DD Boost.  Navigate to Data Management – DD Boost – IP Network.  Highlight and edit an interface group and tick the “Enabled” check box in the resultant dialogue box and click OK.
   

That takes care of the configuration on the DataDomain side of things.  Let’s move over to Backup Exec

Backup Exec 2012 Configuration

1. Download and install the latest version of the EMC DataDomain Boost for Symantec OpenStorage (Version 2.5.0.3-314845) at the time of writing.  This file is available from the DataDomain support site (Powerlink Login required)
2. Open up the Backup Exec console, select the Storage tab and click Configure Storage
3. The type of storage is OpenStorage
   
4. Enter a name and description for the DataDomain
   
5. Select the DataDomain provider (the DataDomain only shows up once you completed the OST plugin installation as per Step 1)
   
6. Enter the connection details for the your DataDomain device
   
7. Enter the Storage Location configured on the DataDomain (BackupExec in our case)
   
8. Select the amount of concurrent operations allowed to the DataDomain
   
9. Click Finish on the Summary screen and confirm that you would like to restart the Backup Exec services
   

With that completed you’ll be able to select the DataDomain as a deduplication target (as opposed to B2D device).

Rebuilding the Exchange 2010 Offline Address Book (OAB) from scratch

IT peeps supporting Microsoft Exchange can be divided into two groups – those that have experienced problems with the OAB and those that will.  If you’re in the first camp – I feel your pain.  Those of you lucky enough to be in the second camp – file this away, it might come in handy…no I’m not being facetious.
The process below will blow away your OAB and create a new one, so be mindful.  FWIW I’ve never had any issues with this process.  This is especially effective if you’ve screwed up on the public folder replication during Exchange Migrations (don’t ask).
We rebuild the Exchange 2010 OAB like so:

Create a new Offline Address Book object

1. Open the Exchange Management Console (EMC) and navigate to Organisation Configuration – Mailbox
2. Click the Offline Address Book tab.  Right click in the blank area and click New Offline Address Book
3. Give your OAB a different name than the existing one
4. Select your Exchange 2010 MBX server as the OAB generation server
5. Check Include the default Global Address Lists option
6. Check Enable Web-based Distribution as well as the Enable public folder distribution option
7. Finish the wizard

Restart Exchange Services

1. Restart the Microsoft Exchange System Attendant service
2. Restart the Microsoft Exchange File Distribution service

Update and set the OAB as default Offline Address Book

1. Right-click your newly created OAB and click Update.  This can take a couple of minutes, confirm successful completion via your Application log
2. Right-click the OAB from step 1 and and click Set as default

Assign the OAB to the affected users databases

1. Open the Exchange Management Shell (EMS)
2. Execute the following:
   Get-MailboxDatabase |  set-MailboxDatabase  -OfflineAddressBook "%your new OAB%"
3. Wait for Outlook to complete it’s OAB download cycle (can be as much as 24 hours)

You can safely delete the old OAB once you’ve verified that your clients are successfully downloading the newly created one.

Performing a Non-Disruptive Disk and Shelf Firmware upgrade on a NetApp FAS2040

I received a mail from NetApp this morning, pointing my attention to KB ID 7010014.  In a nutshell, there is a drive firmware upgrade available which the lowers the drive failure rates.  AutoSupport has also been nagging me about out of date DS4243 shelf firmware, so I thought this would be a perfect opportunity to upgrade it all in in one go.  It goes without saying that the upgrades must to have zero impact on client access.  The process below was run on Data Ontap Release 8.1 7-Mode.

Update the Disk Qualification Package

1. Download the latest DQP from the NetApp support site
2. Extract the files and copy it to the /etc folder on your filer, overwriting the existing files

Update the Disk Shelf Software

1. Download the appropriate disk shelf software upgrade from the NetApp support site
2. Extract and copy it to the /etc/shelf_fw folder on your filer
3. Run the options shelf.fw.ndu.enable command and verify it is set to on
   • If not, enable it with the options shelf.fw.ndu.enable on command
4. Execute the storage download shelf command to update the shelf firmware and enter yes when prompted

Update the Disk Firmware

1. Download the latest disk firmware from the NetApp support site
2. Verify the following, otherwise you will not be able to do a non-disruptive upgrade
   • Aggregates need to be RAID-DP or mirrored RAID4
   • You need to have functioning spares
3. Run the options raid.background_disk_fw_update.enable command and verify it is set to on
   • If not, enable it with the options raid.background_disk_fw_update.enable on command
4. Extract and copy the disk firmware to the /etc/disk_fw folder on your filer
5. The upgrade should start automatically in a couple of minutes
6. Repeat for both controllers

Verifying the upgrade

Execute the sysconfig –v command to verify successful installation
And there we go, we have non-disruptively upgraded the firmware and disk drives in our filer!

Migrating CIFS Shares to a new NetApp Filer

What better way to kick off the festive season than a with a storage migration (only being slightly ironic!).  A customer uses their existing NetApp kit to provide block storage to vSphere hosts and CIFS shares to Windows clients and they wanted me to do a swap out upgrade.  Migrating the vSphere data is a cinch nowadays, what with Storage vMotion and all, so I’ll just document the CIFS stuff.

1. First you’ll need to setup a SnapMirror relationship of the CIFS volume between the source and destination filers (no faffing around with robocopy and the like)
2. Make a backup copy of the /etc/cifsconfig_shares.cfg file
3. Execute cifs terminate on the source filer (downtime starts here)
4. Update (quiesce if necessary) and break the SnapMirror relationship
5. Take the source filer offline
6. Assign the source filer’s IP to the new filer
7. Reset the source filer’s account in Active Directory (if applicable)
8. Execute cifs setup on the new filer
1. It goes without saying that you will assign the source filer’s hostname to the destination filer, as well as join it to the AD (assuming the source filer was joined)
9. Execute cifs terminate on the destination filer and replace the cifsconfig_shares.cfg with the backup copy you made in step 2
10. Execute cifs restart on the destination filer
11. Test client access

Creating a WinPE boot disk for SCCM with integrated drivers

I’m busy with desktop deployment project, utilising SCCM’s OSD functionality.  This required creating a WinPE-based boot image which contains the necessary network and mass-storage drivers to allow the bare-metal deployment process to connect to my SCCM server.

Here is how I did it:

Download and Install the Windows Automated Installation Kit (WAIK)

1. Download the WAIK from here
2. Strictly speaking not necessary, but you can download the SP1 supplement from here
3. Install it – installation is straight-forward, next – next – done kind of thing

Creating the WinPE boot disk

I used c:\dellwinpe as my wim staging folder, you can change it to suit your environment.

1. Launch an administrative WAIK command prompt (normal command prompt won’t work)
2. copype x86 c:\dellwinpe
3. copy the Dell WinPE driver cab to the c:\dellwinpe folder and extract so that the extracted WinPE folder is under the root of your c:\dellwinpe folder
4. dism /Mount-Wim /WimFile:winpe.wim /index:1 /MountDir:Mount
5. dism /add-driver /driver:"winpe" /image:"mount" /recurse
6. dism /unmount-wim /mountdir:"mount" /commit
7. dism /mount-wim /wimfile:"winpe.wim" /index:1 /mountdir:"mount"

Now we add a couple of packages necessary for SCCM integration and deployment

1. dism /image:"mount" /add-package /packagepath:"C:\Program Files\Windows AIK\Tools\PETools\amd64\WinPE_FPs\WinPE-Scripting.cab"
2. dism /image:"mount" /add-package /packagepath:"C:\Program Files\Windows AIK\Tools\PETools\amd64\WinPE_FPs\WinPE-WMI.cab"
3. dism /unmount-wim /mountdir:"mount" /commit

There will now be a file called winpe.wim in your c:\dellwinpe folder, ready to be imported into SCCM as a boot image.

Directly Connecting a Brocade 815 HBA to a EMC VNX5300

I’m busy with a project which involves getting two ESXi hosts hooked up to a VNX5300 configured in block mode.  The order we placed with Dell specified Emulex 12000 HBA’s, but Dell got creative and shipped Brocade 815’s instead.  Only problem was that they didn’t work when directly connected to the front-end ports on the VNX.  I’m documenting the symptoms here as well, so that the next person does not have to battle for two days.

The Symptoms

When directly connecting the HBA’s to the VNX fiber ports the following events pop up in the SP event logs

• EV_VirtualArrayFeature::_mergeInternalObjects() - No parent for HBA,
• EV_TargetMapEntry::GetHostInitiatorPort() - NULL HBAPort pointer

Running NaviSECCli.exe -Address 172.20.10.27 port -list –sfpstate outputs the following:

SP Name:             SP A
SP Port ID:          1
SP UID:              50:06:01:60:BE:A0:72:F9:50:06:01:61:3E:A0:72:F9
Link Status:         Up
Port Status:         Online
Switch Present:      NO
SFP State:           Online

This tells us that things are fine on a physical layer, but not much else is happening higher up the stack.

The Fix

First we need to upgrade the HBA firmware to version 3.1.  There are various OS specific ways to do it, easiest is probably to download the livecd from Brocade.  Since this HBA is not on the ESXi 5.1 HCL we need to install the driver.  You need to install at least the v3.1   I include the steps for the sake of completeness

1. Enable SSH on your ESXi host
2. Use scp for Windows or the following command from a linux / max host:  scp brocade_driver_esx50_v3-1-0-0.tar root@<ip address>:/tmp
3. SSH into your ESXi host and navigate to the /tmp folder with cd /tmp
4. Execute tar xf brocadedriveresx50_v3-1-0-0.tar
5. Execute ./brocade_install_esxi.sh
6. Wait for the installation to finish (takes about 1 – 2 mins) and reboot host once done

Now we need to configure the HBA for direct connection, or more technically,  FC-AL mode

1. SSH into your ESXi host and navigate to /opt/brocade/bin/ by entering cd /opt/brocade/bin/
2. ./bcu port --topology 1/0 loop
3. ./bcu port —disable 1/0
4. ./bcu port —enable 1/0
5. ./bcu port --topology 2/0 loop
6. ./bcu port —disable 1/0
7. ./bcu port —enable 1/0

Your ESXi host should now show up as a host on the VNX where you can add it to a storage group and assign LUNs.

Setting up vSphere Active / Active iSCSI connections to a NetApp FAS2040

I recently had the opportunity to architect a solution consisting of 3 vSphere 5 boxes connecting to a NetApp FAS2040.  Storage connectivity would be via iSCSI.  The storage network would be running off of 2 Cisco 2960G switches, soon to be replaced by stacked Cisco 3750’s. 

The requirements were stock standard, as high a throughput as possible, with as much redundancy as possible.  This meant going active active on the iSCSI links.  Here is how I did it.

NetApp FAS2040 Configuration

This little SAN has 8 1GB Ethernet ports.  Due to the fact that the Cisco 2960G switches does not support multi-link switch aggregation (this is where the 3750’s will come in) I had to come up with a simpler design – what NetApp terms a Single-Mode design.  My design allows for:

• Two active connections to each controller, thus a total of four active sessions
• Storage path HA
• Load balancing across links
• Uses vSphere storage MPIO as opposed to switch-side configuration

Virtual Interface (VIF) Configuration:

All Vif's are single-mode / active passive
Cont1_Vif01 - e0a/e0b (e0a will be active, connected to switch 1 / e0b passive connected to switch 2) IP – 192.168.1.1
Cont1_Vif02 - e0c/e0d (e0c will be active, connected to switch 2 / e0d passive connected to switch 1) IP – 192.168.2.1
Cont2_Vif01 - e0a/e0b (e0a will be passive, connected to switch 1 / e0b active connected to switch 2) IP – 192.168.1.2
Cont2_Vif02 - e0c/e0d (e0c will be passive, connected to switch 2 / e0d active connected to switch 1) IP – 192.168.2.2

This image, courtesy of NetApp, explains it infinitely better than my wall of text:-)

I also configured partner takeover for all VIF.  In case of controller failure it allows the remaining controller to take over the VIFs.

Ethernet Storage Network Configuration

On the storage network I had to configure 2 critical settings:

• Spanning Tree Portfast
• Jumbo Frames

When connecting ESX and NetApp storage arrays to Ethernet storage networks, NetApp highly recommends configuring the Ethernet ports to which these systems connect as RSTP edge ports.  This is done like so:

Switch2960(config)# interface gigabitethernet2/0/2
Switch2960(config-if)# spanning-tree portfast

Next up, Jumbo Frames:

Switch2960(config)# system mtu jumbo 9000
Switch2960(config)# exit
Switch2960# reload

vSphere Configuration

I am in love with vSphere 5, and one of the biggest reasons for that is the fact that a lot of the configuration parameters that used to be command-line only has been moved into the GUI.  Another reason is Multiple TCP Session Support for iSCSI.  This feature enables round robin load balancing using VMware native multipathing and requires a VMkernel port to be
defined for each physical adapter port assigned to iSCSI traffic.  That said, let’s get configuring:

1. Open your vCenter Serve
2. Select an ESXi host
3. In the right pane, click the Configuration tab
4. In the Hardware box, select Networking
5. In the upper-right corner, click Add Networking to open the Add Network wizard
6. Select the VMkernel radio button and click Next
7. Configure the VMkernel by providing the required network information.  NetApp requires separate subnets for active/active iSCSI connections, therefore we will create two VMkernels, on the 192.168.1.x and 192.168.2.x subnets respectively.
8. Configure each VMkernel to use a single active adapter that is not used by any other iSCSI VMkernel. Also, each VMkernel must not have any standby adapters. If using a single vSwitch, it is necessary to override the switch failover order for each VMkernel port used for iSCSI. There must be only one active vmnic, and all others should be assigned to unused
9. The VMkernels created in the previous steps must be bound to the software iSCSI storage adapter. In the Hardware box for the selected ESXi server, select Storage Adapters.
10. Right-click the iSCSI Software Adapter and select properties. The iSCSI Initiator Properties dialog box appears
11. Click the Network Configuration tab
12. In the top window, the VMkernel ports that are currently bound to the iSCSI software interface are listed
13. To bind a new VMkernel port, click the Add button. A list of eligible VMkernel ports is displayed. If no eligible ports are displayed, make sure that the VMkernel ports have a 1:1 mapping to active vmnics as described earlier
14. Select the desired VMkernel port and click OK.
15. Click Close to close the dialog box
16. At this point, the vSphere Client will recommend rescanning the iSCSI adapters. After doing this, go back into the Network Configuration tab to verify that the new VMkernel ports are shown as active, as per the image below.

Congratulations, you now have active / active, redundant iSCSI sessions into your NetApp SAN!