Exchange Server 2010 - Server Move (to new hardware)

Here are my notes from when I upgraded our internal Microsoft Exchange 2010 server's hardware. As I was starting a new job; they had a catastrophic Exchange failure about a week before and ended up installing a new server on "adequate" hardware just to get them up and running. I was tasked with specifying new hardware and migrating the environment to the new appropriate server hardware. This is what I did:

NOTE: PLEASE READ THIS ENTIRE ARTICLE BEFORE YOU START. PAY CLOSE ATTENTION TO THE AREA CONCERNING THE CAS ARRAY. This caused me a lot of pain and I found out the hard way. Do NOT move any mailboxes until you have created a CAS ARRAY in your environment.

OLD SERVER:          exchange.mydomain.local

NEW SERVER:         exch01.mydomain.local

4/10/2012

exchange.mydomain.local
IP Address: 10.0.10.11

Exchange Version Information: Version: 14.1 (build 218.15)

Ran powershell command:

GCM exsetup | %{$_.Fileversioninfo}

Result:

ProductVersion   FileVersion      FileName                                                                             
--------------   -----------      --------                                                                             
14.01.0355.002   14.01.0355.002   C:\Program Files\Microsoft\Exchange Server\V14\bin\ExSetup.exe

This translates to Microsoft Exchange 2010 SP1 Rollup 6


In order to bring it to the current patch level, we would need SP2 and SP2 Update Rollup 1. However, after asking a Microsoft Exchange MVP, there is no need to have the old server fully patched/upgraded in order to perform mailbox moves, etc.

4/11/2012

Updated and ran Exchange Best Practices Analyzer Tool on exchange.mydomain.local

3 Critical Issues were found:

1. Organization incoming message size not set
   
2. Organization outgoing message size not set
   
3. The Offline Address Book Site Public Folder Store Was Deleted
   

On item ONE:

Ran the powershell command below to set the limit to 50MB:

Set-TransportConfig -MaxReceiveSize 50MB

On item TWO:

Ran the powershell command below to set the limit to 50MB:

Set-TransportConfig -MaxSendSize 50MB

Then ran this command:

Get-TransportConfig

After running this command we now have the following output for those values:

MaxReceiveSize: 50 MB
MaxSendSize: 50 MB

On item THREE: (The Offline Address Book Site Public Folder Store Was Deleted)

According to this article on TechNet:

http://technet.microsoft.com/en-us/library/7f20b922-7988-4ece-b167-069efe6abcff.aspx

“If your Exchange organization is Exchange Server 2007 or 2010, with no down level (Outlook 2003 or earlier) clients, you can safely disregard this warning or Error.”

Per the article, I am disregarding this item.

EXCHANGE INSTALL

Installing pre-requisite software before starting Exchange 2010 Setup.

• Microsoft Office 2010 Filter Packs
  

The rest is completed using the following PowerShell Commands:

Import-Module ServerManager

Add-WindowsFeature NET-Framework,RSAT-ADDS,Web-Server,Web-Basic-Auth,Web-Windows-Auth,Web-Metabase,Web-Net-Ext,Web-Lgcy-Mgmt-Console,WAS-Process-Model,RSAT-Web-Server,Web-ISAPI-Ext,Web-Digest-Auth,Web-Dyn-Compression,NET-HTTP-Activation,Web-Asp-Net,Web-Client-Auth,Web-Dir-Browsing,Web-Http-Errors,Web-Http-Logging,Web-Http-Redirect,Web-Http-Tracing,Web-ISAPI-Filter,Web-Request-Monitor,Web-Static-Content,Web-WMI,RPC-Over-HTTP-Proxy –Restart

Re-boot the server.

Installed Hotfix for this issue as another pre-req to install Exchange:
http://support.microsoft.com/?kbid=982867


Ran Windows Updates again… Re-booted.


Ran setup again. Passed all pre-requisites for the roles:

Summary: 6 item(s). 6 succeeded, 0 failed.
Elapsed time: 00:00:31


Configuring Prerequisites
Completed
Elapsed Time: 00:00:01

Organization Prerequisites
Completed
Elapsed Time: 00:00:14

Languages Prerequisites
Completed
Elapsed Time: 00:00:02

Hub Transport Role Prerequisites
Completed
Elapsed Time: 00:00:04

Client Access Role Prerequisites
Completed
Elapsed Time: 00:00:04

Mailbox Role Prerequisites
Completed
Elapsed Time: 00:00:04


4/24/2012

Finally running setup. Since when we complete this project, this will be the only Exchange server, we need the following roles installed:

• Hub Transport Role
  
• Client Access Role
  
• Mailbox Role
  

During setup, when installing the “Hub Transport Role” portion, I got the following error:

Hub Transport Role
Failed

The following error was generated when "$error.Clear();

          if ( ($server -eq $null) -and ($RoleIsDatacenter -ne $true) )

          {

            Update-RmsSharedIdentity -ServerName $RoleNetBIOSName

          }

        " was run: "Database is mandatory on UserMailbox. Property Name: Database".

Database is mandatory on UserMailbox. Property Name: Database

After Googling the error, came across this blog post:

http://www.paulbrown.us/2009/12/adding-a-mailbox-server-role-to-exchange-2010-database-is-mandatory-on-usermailbox/

It suggested that the there was an attribute missing on a mailbox called:

FederatedEmail.4c1f4d8b-8179-4148-93bf-00a95fa1e042


The attribute was homeMDB. I just copied this attribute from another mailbox and pasted it in there (since there is only one server currently, ALL mailboxes have to use the same DB).

After doing what he suggested, I re-ran setup and the Hub Transport Role completed without issue.

Client Access Role installed without Issue.
Mailbox Role installed without issue.
Installation completed successfully.

I installed 3 Microsoft Updates after the install (one of which was an Exchange Update Rollup) and rebooted the server.

After the re-boot, I was prompted to enter the license key before a 28 day grace period ends. I hit our Microsoft Open License site and got the code and entered it. I needed to re-start the Information Store for it to take effect.


5/03/2012

Moved my mailbox using the Exchange Management Console (not PowerShell).

Ran report using PowerShell to see mailbox sizes:

Get-MailboxStatistics -Database "Mailbox Database 1584415394" | Sort-Object TotalItemSize –Descending | ft @{label=”User”;expression={$_.DisplayName}},@{label=”Total Size (MB)”;expression={$_.TotalItemSize.Value.ToMB()}},@{label=”Items”;expression={$_.ItemCount}} | out-file C:\MailboxSizeReport.txt

Renamed the Mailbox Database on EXCH01 to:
E:\ExchangeDB1\ExchangeDB1.edb

Moved Logs to:
E:\ExchangeDB1_Logs

(This process caused all Exchange services to restart)


5/11/2012

Moved 2 other user's mailboxes using the Exchange Management Console (not PowerShell).


5/14/2012

Moved the Offline Address Book to EXCH01. Got the following result:

Summary: 1 item(s). 1 succeeded, 0 failed.
Elapsed time: 00:00:01

Default Offline Address Book
Completed

Warning:

Task has copied current web distribution files for offline address book "\Default Offline Address Book" to the target server. You can back up and remove copy of these files from the previous location: \\EXCHANGE\ExchangeOAB\cf2fc5ae-74a1-469e-922f-c85ca5ff2f5b.

Exchange Management Shell command completed:

move-OfflineAddressBook -Identity '\Default Offline Address Book' -Server 'EXCH01'

Elapsed Time: 00:00:01

• Created a new Public Folder Store on exch01. DB Files are located at: E:\Public Folder Databases\Public Store DB1.edb (logs too)
  
• Setup replication of the Public Folders from “exchange” to “exch01”
  
• Moved Public Folder Replicas from “exchange” to “exch01” using this script:
  

.\MoveAllReplicas.ps1 -Server exchange -NewServer exch01

 

• Move all arbitration mailboxes (Note: there were several “Orphaned” system mailboxes that I had to fix before being able to move them)
  

Execute Get-Mailbox -Arbitration | select Alias, to see all available Arbitration mailboxes. I got this error on the Discovery and the Message Approval mailboxes:


WARNING: The object XXXXXXXX.XXXXX/Users/SystemMailbox{e0dc1c29-89c3-4034-b678-e6c29d823ed9} has been corrupted, and it's in an inconsistent state. The following validation errors happened:
WARNING: Database is mandatory on UserMailbox.

WARNING: The object XXXXXXXX.XXXXX/Users/SystemMailbox{1f05a927-xxxx-xxxx-xxxx-xxxxxxxxxxxx} (where x is a random number) has been corrupted, and it's in an inconsistent state. The following validation errors happened:
WARNING: Database is mandatory on UserMailbox.

In order to get this working I had to run Setup.com /PrepareAD and also run:

Enable-Mailbox on the Discovery and message Approval Mailboxes from above. I referenced the following websites to solve this:

http://www.mikepfeiffer.net/2010/04/how-to-recreate-the-system-mailboxes-in-exchange-2010/

http://halfloaded.com/blog/installing-exchange-2010-service-pack-1-fails-at-mailbox-role-database-is-mandatory-on-usermailbox/

I moved all the Arbitration mailboxes to the other mailbox server by executing the cmdlets below:

Get-Mailbox -Arbitration | New-MoveRequest -TargetDatabase "ExchangeDB1"

• Created Exchange Database 2 and 3
  

Database Location: E:\ExchangeDB2\ExchangeDB2.edb
Logs to: E:\ExchangeDB2_Logs

Database Location: E:\ExchangeDB3\ExchangeDB3.edb
Logs to: E:\ExchangeDB3_Logs


5/15/2012

• Removed the Public Folder Database from “exchange”
  

Note: Several Users reports that they got an error message that said, “The Microsoft Exchange administrator has made a change that requires you quit and restart Outlook.”

• Added exch01 to the 2 Send Connectors under:
  

Organization Configuration > Hub Transport:

GFIFaxmaker (Under “Source Server” Tab)
OutboundEmail (Under “Source Server” Tab)


5/16/2012

• Upon arrival this AM, many people were complaining of “Mail Delivery Delayed” messages in their inbox. This was an error on my part. I forgot the ZIX only accepted mail from 10.0.10.11.
  

I Changed the ZIX’s configuration because mail started flowing from exch01 and was not able to relay on the ZIX. The ZIX was only allowing exchange (10.0.10.11) to relay, so I opened it up to allow everything on 10.0.10.0/24 subnet to relay. After this was completed, mail flow returned to normal.

• Later in the day, a mis-configuration was finally found in the Exchange Organization. This must have been from when the migration was done from the old Exchange server. But under “Organization Configuration” > “Hub Transport” on the “Accepted Domains” tab, the myexternaldomain.com domain was set as an “Internal Relay Domain” this was causing loops in the queues because instead of sending an NDR when an address was not found, it returned the message out to the internet. This caused the ZIX to send it BACK – and a loop resulted. I had to set this setting to “Authoritative Domain” to resolve this. This had been an ongoing issue, but we never had the time to resolve it, and the queues never really got too backed up. The downside was that nobody from the outside would get an NDR if they sent a message to the wrong internal address.
  

• After hours today, I attempted to get the SAN SSL Certificate imported to the new Exchange server. I had some difficulty, but finally figured it out. Two things to keep in mind:
  

1. The SAN Certificate is specific to the machine used to create the cert request, it cannot be imported to any other Exchange servers. You must re-key the certificate using a new request from the new server.
   
2. During the 2nd part of IMPORTING the certificate using the “New Exchange Certificate” wizard, you must have the certificate REQUEST file (.req) you created in the beginning in the same folder as the certificate you download from the provider. Exchange uses both to format the cert into the PFX format. Also when you import it, there is a section that asks for a password for the cert. You can just enter a space if you don’t have one.
   

• I also changed the NAT translation in the SonicWall to point to the new Exchange server (exch01)
  

5/17/2012

• Started pointing devices that email on the network to the new server.
  
• Started with the ZIX. This is done by logging onto the web interface, then:
  

• Changed the Help Desk mail setup to the new IP
  
• Changed the copiers SMTP setup to the new IP
  

5/18-20/2012

• Started moving mailboxes (actual users). I did all of them except for a few that had huge messages in them. I had to look at the move logs for failures. In the log it would show the message info (Subject Line, etc.) so it could be found in the mailbox. Most were in user’s dumpsters. But some were in Sent Items too. There were users who had attachments that were 90+ Megabytes in a single message.
  

5/21/2012

• Finalized moving mailboxes. Got all users moved over after deleting some items in their respective dumpsters.
  
• Ran ESEUTIL /d on the Mailbox Database on exchange to try to bring the DB size down from 300GB.
  
• In the evening, I rebooted the OLD exchange server, and most users that were on were disconnected from Outlook. Tomorrow I will try to determine why.
  

5/22/2012

As it turns out, in an environment with more than ONE Exchange server, you need a CAS Array so the Outlook clients are automatically directed to the correct Exchange server.

I was pointed on a forum to these articles:

http://blogs.technet.com/b/exchange/archive/2012/03/23/demystifying-the-cas-array-object-part-1.aspx


http://blogs.technet.com/b/exchange/archive/2012/03/28/demystifying-the-cas-array-object-part-2.aspx

Unfortunately, since I have already created the new mailboxes, and moved users to them, I may not be able to have Outlook “autoconfigure”.

I ran the command below to create a CAS array for the entire Exchange Organization:

New-ClientAccessArray –Fqdn “outlook.mydomain.local” –Site “Default-First-Site-Name”

I ran this next command to associate all the mailbox databases with the newly created CAS Array:

Get-MailboxDatabase | Set-MailboxDatabase -RPCClientAccessServer “outlook.mydomain.local”


7/06/2012

Had a conference call with the ex-senior Exchange Escalation Engineer for Microsoft UK. They could not commit to a “project” but spent an hour looking at our Exchange setup. They went through our send/receive connectors, public folder DB location, etc.

In the “autodiscover settings” they saw several references to the old server. Those need to be changed to the NEW server.

They saw nothing that was glaring that might cause a disruption if the old server were de-commissioned. They actually recommended that we shut the server down during the day to see what happens. They said that would be the quickest way to find out which clients were using the old server.

--------------------------------------------------------------

When I looked at the “Autodiscover Settings” (Ctrl+Right-Click on Outlook System tray icon > “Test E-mail Autoconfiguration…”) there were still entries for the old exchange server in the following:

Protocol: Exchange RPC
Server: outlook.mydomain.local
Login Name: myusername
Availability Service URL: https://exchange.mydomain.local/EWS/Exchange.asmx
OOF URL: https://exchange.mydomain.local/EWS/Exchange.asmx
OAB URL: http://exch01.mydomain.local/OAB/cf2fc5ae-74a1-469e-922f-c85ca5ff2f5b/
Unified Message Service URL: https://exchange.mydomain.local/EWS/UM2007Legacy.asmx
Auth Package: Unspecified

So I ran the following PowerShell Command:

Set-WebServicesVirtualDirectory -Identity "EWS*" -ExternalUrl "https://mail.myexternaldomain.com/EWS/Exchange.asmx" -InternalUrl "https://exch01.mydomain.local/EWS/Exchange.asmx"

After running that command the Autoconfiguration settings now look like this:

Protocol: Exchange RPC
Server: outlook.mydomain.local
Login Name: agarmont
Availability Service URL: https://exchange.mydomain.local/EWS/Exchange.asmx
OOF URL: https:// exch01.mydomain.local/EWS/Exchange.asmx
OAB URL: http://exch01.mydomain.local/OAB/cf2fc5ae-74a1-469e-922f-c85ca5ff2f5b/
Unified Message Service URL: https:// exch01.mydomain.local/EWS/UM2007Legacy.asmx
Auth Package: Unspecified


7/19/2012

Today we are attempting to test Exchange shutdown during the day, per the recommendation of my friend the Exchange MVP. Several users’ Outlook are still showing a connection to the server when a NETSTAT is run from exchange.mydomain.local. There were around 10-12 of them. We determined that this was from “auto-added” mailboxes from when a user has “full” access to a user’s mailbox.

We ended up removing the permissions from the mailboxes temporarily until connections to exchange.local stopped. Then we shut down exchange.local. After it was completely down, we then re-assigned permissions to the mailboxes in question and let them be “re-mapped” to the new server.

This ultimately allowed us to leave exchange.local off for good. We only need to start it back up in order to un-install Exchange and decommission it properly.


7/25/2012 – 9:30am

Attempting to uninstall Exchange services from exchange.mydomain.local today. Started this by going to Control Panel and “Un-installing” Microsoft Exchange Server 2010.

Started by uninstalling each Role one at a time:

1. Hub Transport Role
   
2. Mailbox Role
   
3. Client Access Role & Management Tools
   

Un-install went smoothly with no errors or issues.

Ran the Exchange Best Practices Analyzer to make sure nothing is mis-configured after the un-install. Only 2 Critical Issues were flagged:

1. Organization incoming message size too high
   
2. Organization outgoing message size too high
   

These are set to what we want them at so they can be disregarded.