Real Life Exchange 2010 Disaster Recovery
Last Thursday night I had the distinct privilege of participating in a real life Exchange 2010 disaster recovery scenario. Roughly 41,000 mailboxes on 22 databases had the DAG ripped out from underneath them accidentally. When I say ‘ripped out from underneath them’ I mean literally that; everything was removed, including all the cluster resources and mailbox servers. As far as AD was concerned these servers didn’t even exist. All we had left was the .edb files, catalog indexes and log files. The solution? Database portability.
Even though this was essentially a test environment, there were a number of live mailboxes that simply had to be recovered and we wanted to recover them all. The steps we took were rather straightforward;
1 | Find some live mailbox servers that had the spare capacity to mount 22 databases. Split the list of databases to be mounted among them.
2 | For each mailbox server, copy over the database and log directories so we had the data to use.
3 | Perform a soft recovery with eseutil /r on each database/log set to commit any uncommitted log files and ensure we could actually mount the data later in the process.
4 | Create the new mailbox databases: new-mailboxdatabase -name <name> -server <server> -EdbFilePath <path to recovery folder, e.g. c:\RecoverDBs\RecoverDB1\<name of original edb>.edb> -LogFilePath <path to logs, e.g. c:\RecoverDBs\RecoverLogs1>
Pro Tip: Use a new name for the database. If the old database was named DAG1-DB001, you might use DAG1-RecoveryDB001.
5 | Set the newly created databases to allow file restore: set-mailboxdatabase <db name> -AllowFileRestore:$true
6 | Copy in the database, logs and catalog data to the correct folders (those specified in step 4)
7 | Mount the databases one at a time: mount-database <DatabaseName>
8 | Once the database is mounted we can now re-home all the users with mail data there: get-mailbox -database <OriginalDatabaseName> | ?{$_.ObjectClass -NotMatch ‘(SystemAttendantMailbox|ExOldDbSystemMailbox)’} | set-mailbox -database <RecoveryDatabaseName>
9 | If you’re running with multiple copies then keep in mind that you’ve only got one live copy of the new database. You can either add a copy of the new database or do what we did and move them to databases on your new DAG that (hopefully) has multiple copies already. If you choose to go the route of moving them to existing healthy databases the command is: get-mailbox -database <RecoveryDatabaseName> | new-moverequest -TargetDatabase <HealthyDatabaseName>
Comments and/or questions are welcome in the comments. I just wrote this from memory so if I missed anything along the way please let me know.
Categories: Exchange Server, Systems Administration, Tech
Tags: disaster recovery, exchange, Exchange 2010, Exchange Server
Comments: 7 Comments.
Great information, how long did it take to recovery everything?
.-= Disaster Roy´s last blog ..Catalog Recovery =-.
[Reply]
Nice…. Let me ask you another question. How many people were involved in the recovery and if they’ve done it before ?
Thanks
[Reply]
Jeremy Phillips Reply:
December 19th, 2011 at 4:35 pm
There were two of us directly involved and a few store developers (we were consultants working with the Exchange team at the time) to sanity check our steps.
Nope, we’d never done it with 2010 before. I don’t think many had at this point as it was still in beta.
Feel free to ring us up if you need any Exchange help in the future. =)
[Reply]
Also , it’s really curious how you guys were able to overcome ssid/guid issues since the restoring servers were different names and all that.
I gotta read up more on it but you make it sound really simple, too simple to work in my book . When you try to re-mount/restore a DB belonging to EXCH01 on EXCH02 it’s not gonna work , haven’t done it though.
I am trying it as we speak , installing exchange on a member server and will try to remount the db from a backup . We’ll see……
[Reply]
Jeremy Phillips Reply:
December 19th, 2011 at 5:24 pm
It’s a feature called database portability. Works like a charm if you do it right. Here’s some info about it on TechNet: http://technet.microsoft.com/en-us/library/dd876873.aspx
[Reply]
Hey Jeremy
Still at it man. Trying to recreate the network on two isolated machines , one with the DC and one with the EXCH because on the EXCH that just crashed I had also the dc on it( I know , I know) . I installed the DC and the exchange on the same machine with thoughts that I would move it later on a separate machine or de-promote the dc and leaving only the exch on it.
Got caught up in other stuff , as everyone usually does , postponed it ….. and now I couldn’t restore the machine back so I had to import the pst’s to another exchange and now I’m trying to remove the dead exch from the org , which is also a pain.
Will keep you posted .
[Reply]
Jeremy Phillips Reply:
January 17th, 2010 at 10:19 pm
Six hours.
[Reply]