AD and Exchange Forest Migration (Part I)

I’m currently in the middle of a big and relatively complex forest migration.  I’ve found that while there’s a ton of documentation on the subject, a lot of it is way too complex for 90% of engagements and the rest is very spotty.  Thus I’ve set out to document my processes in a simple and to the point way, keeping in mind that this is what works for me, in this specific client’s environment.  Caveat Emptor.

Current Environment

Source:

The source domain is a standalone forest, with a two-way forest trust to the target domain.

Source Domain Name:  olddomain.local
Domain Functional Level: Windows Server 2008 R2 domain level
Mode: Native
Forest Level: Windows Server 2008 R2 domain level
SMTP Address Space: company.com

Target:

The target domain is a child domain contained in a existing forest.

Target Domain Name: newdomain.local
Domain Functional Level: Windows Server 2008 R2 domain level
Mode: Native
Forest Level: Windows Server 2008 R2 domain level
SMTP Address Space: company.com

High-Level Overview

1. Clean up source domain by deleting unused accounts, mailboxes etc.
2. Setting up Name Resolution (DNS) to allow us to create a trust
3. Create a Two-Way Forest Trust between the source and target domains
4. Enable SID History and disable SID Filtering
5. Install the Active Directory Migration Tool (ADMT)
6. Install the ADMT Password Export Server (PES)
7. Use Prepare-MoveRequest.ps1 to create Mail Enabled Users (MEU’s) in the target domain
8. Configure Exchange servers in the source and target domains to operate within a shared address space
9. Use ADMT to migrate user accounts to the target domain
10. Use ADMT to re-ACL resources
11. Use ADMT to migrate computer accounts to the target domain
12. Move mailboxes to the Exchange server in the target domain
13. Decommission source Exchange server
14. Use ADMT to remove old ACL’s from resources
15. Use ADMT to migrate servers to the target domain
16. Decommission old servers, domain and forest

I will use the next series of blog posts to document all the above steps in detail.  As I said, I have been unable to find a single authoritative source for the process, so I aim to make my life easier the next time I’m faced with this challenge.  Hopefully I also save someone else some time and effort.

I want to conclude by saying that even though my documentation might suit your environment to a T, it is imperative that you lab the living daylights out of your processes.  Also, make sure you understand what each step does, and have a rollback procedure in place.

Synching a Windows AD domain with an external time source

Having your time set up correctly is of crucial importance to the well-being of your Active Directory infrastructure.  In every setup I do I ensure that my PDC Emulator FSMO holder synchs with an NTP server on the internet.  Unless there are specific customer requirements I use pool.ntp.org as my external source.  Here is how to configure your PDC Emulator to sync (all commands entered from an Administrative Command Prompt):

1. Ensure that your firewall allows outbound NTP traffic from your PDC emulator
2. Run w32tm /config /manualpeerlist:pool.ntp.org /syncfromflags:MANUAL /update
3. Restart the time service: net stop w32time && net start w32time
4. Force a synchronization with the newly added source: w32tm /resync
5. Verify the source: w32tm /query /source

The rest of your domain clients should automatically synchronise with your PDC emulator.  If this has been changed for whatever reason you can re-instate it with the following commands:

1. w32tm /config /update /syncfromflags:DOMHIER
2. net stop w32time && net start w32time
3. w32tm /query /status

The above will go a long way to giving you a proper time-setup, and should be sufficient and accurate enough for most organisations

Allowing NetApp SnapMirror through a firewall

During a recent NetApp Snapmirror implementation we had a tremendous time getting SnapMirror to work.  After much troubleshooting we discovered that it was due to ACL’s on the customer switches.  After a bit of digging I discovered NetApp Technical Report 3326, which details the firewall configuration required.  It’s a bit TL;DR, so I’ve condensed it for you.
TCP Ports used by NetApp SnapMirror

• TCP 10566 (Source System binds on this port)
• TCP 10569 (Source system listens on this port)
• TCP 10565 (If using multipath, this is what the destination System listens on)
• TCP 10565, 10567, 10568 (Destination System listens on these ports)
• Just open TCP 10565 – 10569 bi-directional and be done with it (if you can get away with it)

So, dear Network Mechanics, ensure that the above is allowed through your switch ACL’s and firewalls and make a storage admin happy!

Exchange 2010 Pre-Install Requirements

Lately I have found myself doing quite a few Exchange 2010 installations, and every single time the pre-install requirements trips me up.  Not really a show-stopper but it gets old quickly having to exit out of the installer, do something, start the installer and then have it error out on the very next step.  So without further ado – here is a list of things that needs doing to ensure Exchange 2010 installs smoothly

1. Target server needs to be running either
   • Windows Server 2008 R2 64-bit with SP2
   • Windows Server 2008 64-bit R2
2. Install the Microsoft Filter Pack (only if you’re going to host the Hub Transport or Mailbox Server roles).  The filter pack is available here
3. Run the following from an elevated PowerShell console: Import-Module ServerManager
4. Run the following from an elevated PowerShell console: Add-WindowsFeature NET-Framework,RSAT-ADDS,Web-Server,Web-Basic-Auth,Web-Windows-Auth,Web-Metabase,Web-Net-Ext,Web-Lgcy-Mgmt-Console,WAS-Process-Model,RSAT-Web-Server,Web-ISAPI-Ext,Web-Digest-Auth,Web-Dyn-Compression,NET-HTTP-Activation,RPC-Over-HTTP-Proxy –Restart
5. Run the following from an elevated PowerShell console: Set-Service NetTcpPortSharing -StartupType Automatic
6. Install latest patches and service packs via Windows Update
7. Proceed with the Exchange 2010 installation

If you want to go straight to the source, here is the Microsoft Technet article

NetApp Command Line Cheat-Sheet

I recently had the opportunity to work on a NetApp storage implementation project.  As always I really wanted to get my hands dirty, so I tried to learn as much about their CLI as possible.  It also helps when the NetApp System Manager GUI has got crazy bugs like Bug ID 548923, which prevents you from doing any FC related configs.  Anyhow, here is a list of commands which should get you up and running in no time.  I compiled this from a couple of web sources.

The Basics

setup (Re-Run initial setup)
halt (Reboots controller into bootrom)
reboot (Reboots the connected controller)
sysconfig -a (Dumps the system configuration)
storage show disk (shows physical information about disks)
passwd (Changes the password for the current user)
sasadmin shelf (shows a graphical layout of your shelves with occupied disk slots)
options trusted.hosts x.x.x.x or x.x.x.x/nn (hosts that are allowed telnet, http, https and ssh admin access. x.x.x.x = ip address, /nn is network bits)
options trusted.hosts * (Allows all hosts to the above command)

Diagnostics
Press DEL at boot up during memory test followed by boot_diags and select all
priv set diags (Enter diagnostics CLI mode from the Ontap CLI)
priv set (Return to normal CLI mode from diagnostics mode)

Software
software list (Lists software in the /etc/software directory)
software delete (Deletes software in the /etc/software directory)
software update 8.1RC2_e_image.zip -r (Install software. The -r prevents it rebooting afterwards)

Aggregates
aggr create aggregate_name (Creates an Aggregate)
aggr destroy aggregate_name (deletes an Aggregate)
aggr offline aggregate_name (takes an Aggregate offline)
aggr online aggregate_name (brings an Aggregate online)
aggr status (shows status of all aggregates)
aggr status aggregate_name (show status of a specific Aggregate)
aggr show_space aggregate_name (shows specific aggregate space information)

Volumes
vol create volume_name (Creates a volume)
vol status (gives the status of all volumes)

Snapshots
snap create volume_name snapshot_name (create a snapshot)
snap list volume_name (List snapshots for a volume)
snap delete volume_name snapshot_name (delete a snapshot on a volume)
snap delete -a volume_name (Deletes all snapshots for a volume)
snap restore -s snapshot_name volume_name (Restores a snapshot on the specified volume name)
options cifs.show_snapshot on (Sets snapshot directory to be browse-able via CIFS)
options nfs.hide_snapshot off (Sets snapshot directory to be visible via NFS)

SnapMirror
options snapmirror.enable on (turns on SnapMirror. Replace on with off to toggle)
vol restrict volume_name (Performed on the Destination. Makes the destination volume read only which must be done for volume based replication)
snapmirror initialize -S srcfiler:source_volume dstfiler:destination_volume (Performed on the destination. This is for full volume mirror. For example snapmirror initialize -S filer1:vol1 filer2:vol2)
snapmirror status (Shows the status of snapmirror and replicated volumes or qtree’s)
snapmirror status -l (Shows much more detail that the command above, i.e. snapshot name, bytes transferred, progress, etc)
snapmirror quiesce volume_name (Performed on Destination. Pauses the SnapMirror Replication. If you are removing the snapmirror relationship this is the first step.)
snapmirror break volume_name (Performed on Destination. Breaks or disengages the SnapMirror Replication. If you are removing the snapmirror relationship this is the second step followed by deleting the snapshot)
snapmirror resync volume_name (Performed on Destination. When data is out of date, for example working off DR site and wanting to resync back to primary, only performed when SnapMirror relationship is broken)
snapmirror update -S srcfiler:volume_name dstfiler:volume_name (Performed on Destination. Forces a new snapshot on the source and performs a replication, only if an initial replication baseline has been already done)
snapmirror release volume_name dstfiler:volume_name (Performed on Destination. Removes a snapmirror destination)

Cluster
cf enable (enable cluster)
cf disable (disable cluster)
cf takeover (take over resources from other controller)
cf giveback (give back controller resources after a take over)

Autosupport
options autosupport.support.enable on (Turns Autosupport on, toggle with off)

Hot Spares
vol status -r (Gives list of spare disks)

Disks
disk show (Show disk information)
disk show -n (Show unowned disks)

Luns
lun setup (runs the cli lun setup wizard)
lun create -s 10g -t windows_2008 -o noreserve /vol/vol1/lun1 (creates a lun of 10GB with type Windows 2008, sets no reservation and places it in the following volume or qtree)
lun offline lun_path (takes a lun offline)
lun online lun_path (brings a lun online)
lun show -v (Verbose listing of luns)

Fiber FCP
fcadmin config -t target 0a (Changes adapter from initiator to target)
fcadmin config (lists adapter state)
fcadmin start (Start the FCP service)
fcadmin stop (Stop the FCP service)
fcp show adapters (Displays adapter type, status, FC Nodename, FC Portname and slot number)
fcp nodename (Displays fiber channel nodename)
fcp show initiators (Show fiber channel initiators)
fcp wwpn-alias set alias_name (Set a fiber channel alias name for the controller)
fcp wwpn-alias remove -a alias_name (Remove a fiber channel alias name for the controller)
igroup show (Displays initiator groups with WWN’s)

Cifs
cifs setup (cifs setup wizard)
cifs restart (restarts cifs)
cifs shares (displays cifs shares)
cifs status (show status of cifs)
cifs domain info (Lists information about the filers connected Windows Domain)
cifs testdc ip_address (Test a specific Windows Domain Controller for connectivity)
cifs prefdc (Displays configured preferred Windows Domain Controllers)
cifs prefdc add domain address_list (Adds a preferred dc for a specific domain i.e. cifs prefdc add netapplab.local 10.10.10.1)
cifs prefdc delete domain (Delete a preferred Windows Domain Controller)
vscan on (Turns virus scanning on)
vscan off (Turns virus scanning off)
vscan reset (Resets virus scanning)

HTTP Admin
options httpd.admin.enable on (enables web admin)

SIS (Deduplication)
sis status (Shows SIS status)
sis config (Shows SIS config)
sis on /vol/vol1 (Turns on deduplication on vol1)
sis start -s /vol/vol1 (Runs deduplication manually on vol1)
sis status -l /vol/vol1 (Displays deduplication status on vol1)
df -s vol1 (View space savings with deduplication)
sis stop /vol/vol1 (Stops deduplication on vol1)
sis off /vol/vol1 (Disables deduplication on vol1)

DNS
dns flush (Flushes the DNS cache)
/etc/resolv.conf (edit this file to change your dns servers)

Problem accessing CSV from passive Cluster Node

I recently had a perplexing problem where Cluster Shared Volumes in a Hyper-V cluster were not working correctly.  The volumes were only accessible from the node currently owning the volume.  Attempts to access the volumes from any of the other nodes resulted Windows Explorer hanging indefinitely.  Enabling maintenance or redirected mode made no difference.

Event ID 5120 was logged:  Cluster Shared Volume 'Volume1' ('Cluster Disk 1') is no longer available on this node because of 'STATUS_BAD_NETWORK_PATH(c00000be)'. All I/O will temporarily be queued until a path to the volume is re-established.
Event ID 5142 also occurred:  Cluster Shared Volume 'Volume1' ('Cluster Disk 1') is no longer accessible from this cluster node because of error 'ERROR_TIMEOUT(1460)'. Please troubleshoot this node's connectivity to the storage device and network connectivity.

I could ping all nodes over both the Production and Heartbeat network links, and I could access file shares from any node on any node.

The Problem
After much troubleshooting I realised I disabled both File and Print Sharing and Client for Microsoft Networks on the Heartbeat NIC on all nodes.  This is a best practice drummed into me since working on Microsoft Clustering when it was still code-named Wolfpack.

The Resolution
I enabled File and Print Sharing and Client for Microsoft Networks and immediately afterwards all my Cluster Shared Volumes started functioning as expected.

The Explanation
It’s documented in MS KB Article 2008795.  When accessing a CSV volume from a passive (non-coordinator) node, the disk I/O to the owning (coordinator) node is routed through a 'preferred' network adapter and requires SMB be enabled on that network adapter. For SMB connections to work on these network adapters, the aforementioned protocols must be enabled.  Ugh.

Setting up a KMS server on Server 2008 R2

Today we’ll deal with setting up a Microsoft Key Management Server (KMS).  a KMS is used to activate Microsoft Volume Licensed products such as Windows 7, Office 2010 and Windows server 2008 R2, amongst others.

a KMS server activates a client for a period of 180 days.  The activated machine will communicate with the KMS every 7 days to renew it’s activation information.  It then resets the license counter back to 180 days if successful.  If not it attempts to background connect to the KMS every 2 hours.

If, after 180 days, the machine has not been able to contact the KMS it will go into the 30 day grace period and notify the user.  After that the machine will enter a reduced functionality mode until it can again connect to a KMS.

That was quite a mouthful – so let’s get down to setting up a KMS on a Windows 2008 R2 host.  In addition we’ll also set it up so that it can activate Office 2010 clients.

Setting up a KMS

1. Activate Windows with a KMS key.  This will automatically configure the server as a KMS
2. Download the Office 2010 KMS Host License Pack
3. Enter your KMS host key when prompted (you will get this key from your Microsoft Volume Licensing website)
4. Make sure to allow the Key Management Service through the Windows Firewall

Verify that KMS is published in DNS

nslookup -type=srv _vlmcs._tcp.<your DNS domain>

Checking the KMS status on your KMS

From an elevated command prompt, type SLMGR.vbs /dlv

Checking the license and activation status on a client

slmgr.vbs –dli

a KMS goes a significant way to easing administrative burden, so go ahead and set it up, it’s as easy-peasy!