Last Thursday night I had the distinct privilege of participating in a real life Exchange 2010 disaster recovery scenario. Roughly 41,000 mailboxes on 22 databases had the DAG ripped out from underneath them accidentally. When I say ‘ripped out from underneath them’ I mean literally that; everything was removed, including all the cluster resources and mailbox servers. As far as AD was concerned these servers didn’t even exist. All we had left was the .edb files, catalog indexes and log files. The solution? Database portability.
Even though this was essentially a test environment, there were a number of live mailboxes that simply had to be recovered and we wanted to recover them all. The steps we took were rather straightforward;
1 | Find some live mailbox servers that had the spare capacity to mount 22 databases. Split the list of databases to be mounted among them.
2 | For each mailbox server, copy over the database and log directories so we had the data to use.
3 | Perform a soft recovery with eseutil /r on each database/log set to commit any uncommitted log files and ensure we could actually mount the data later in the process.
4 | Create the new mailbox databases: new-mailboxdatabase -name <name> -server <server> -EdbFilePath <path to recovery folder, e.g. c:\RecoverDBs\RecoverDB1\<name of original edb>.edb> -LogFilePath <path to logs, e.g. c:\RecoverDBs\RecoverLogs1>
Pro Tip: Use a new name for the database. If the old database was named DAG1-DB001, you might use DAG1-RecoveryDB001.
5 | Set the newly created databases to allow file restore: set-mailboxdatabase <db name> -AllowFileRestore:$true
6 | Copy in the database, logs and catalog data to the correct folders (those specified in step 4)
7 | Mount the databases one at a time: mount-database <DatabaseName>
8 | Once the database is mounted we can now re-home all the users with mail data there: get-mailbox -database <OriginalDatabaseName> | ?{$_.ObjectClass -NotMatch ‘(SystemAttendantMailbox|ExOldDbSystemMailbox)’} | set-mailbox -database <RecoveryDatabaseName>
9 | If you’re running with multiple copies then keep in mind that you’ve only got one live copy of the new database. You can either add a copy of the new database or do what we did and move them to databases on your new DAG that (hopefully) has multiple copies already. If you choose to go the route of moving them to existing healthy databases the command is: get-mailbox -database <RecoveryDatabaseName> | new-moverequest -TargetDatabase <HealthyDatabaseName>
Comments and/or questions are welcome in the comments. I just wrote this from memory so if I missed anything along the way please let me know.
Let me know if you see anything obviously wrong with it. Use at your own risk! :)
#################################
#Copyright 2009 Cohesive Logic LLC
#SCR Health Check Script
#Description – Determines if any of the storage groups on the SCR target server are in an unhealthy state.
#################################
#Options – Specify the following options for the target Enviroment
#################################
#SCR Target Server – Specify the FQDN SCR Target server to be analyzed
$SCRTarget = “Target Server”
#Exchange Mailbox Server – Specify the FQDN for the Exchange Mailbox Server Source for your SCR Target
$MailboxServer = “Source Mailbox Server”
#SMTP Server – Specify FQDN of a local SMTP server that can be used to relay the warning email
$SMTPServer = “SMTP SEERVER NAME by DNS or IP” -
#EmailRecipient – Specify the email address where the warning email shoudl be sent.
$EmailRcpt = “JoeDoe@server.com”
#################################
#The email function used to transmit email over using an SMTP relay
function Send-Email([string]$rcpt,$smtpServer,$body) {
$msg = New-Object net.Mail.MailMessage
$smtp = New-Object net.Mail.SmtpClient($smtpServer)
$msg.From = “ExchangeSCR@Domain.com”
$msg.To.Add($rcpt)
$msg.Subject = “SCR HEALTH WARNING”
$msg.Body = $body
$smtp.Send($msg)
}
#The Check-SCR function determines if replication on any of strorage groups of the SCR target server are in an unhealthy state.
#If any storage groups are found to be unheathy
function Check-SCR($MBX,$SCR,$SMTP,$rcpt) {
$MessageBody = $NULL
$ReplicatedSG = Get-StorageGroup -Server $MBX | where {$_.StandbyMachines -ne $NULL} | Select Identity
foreach($SG in $ReplicatedSG) {
if((Get-StorageGroupCopyStatus $SG.Identity -StandbyMachine $SCR).SummaryCopyStatus -ne “Healthy”){
$MessageBody += Get-StorageGroupCopyStatus $SG.Identity -StandbyMachine $SCR | Select Identity,SummaryCopyStatus,CopyQueueLength,ReplayQueueLength LastInspectedLogTime,ServiceDown,Failed,Suspend
}
}
if($MessageBody -ne $NULL){
Send-Email $rcpt $SMTP $MessageBody
}
}
Check-SCR $MailboxServer $SCRTarget $SMTPServer $EmailRcpt
Do you store PSTs on your file server? This is a fairly common issue that most people either aren’t aware of or don’t care about for some reason. If you ask me it’s A Bad Thing that should be avoided at all costs. Microsoft has a KB article on this and I’ve copied and pasted a few choice sections out of it below…
The .pst files are not meant to be a long-term, continuous-use method of storing messages in an enterprise environment.
Other Behaviors of .pst Files over WAN/LAN Links
- All operations take longer.
- Write operations can take approximately four times longer than read operations.
- Outlook has slower performance than the Exchange Client.
When you store .pst files, shares may stop responding. This behavior may cause several client-side problems, such as causing Outlook to stop responding or freezing desktops on client computers. Queuing in the Server service work queues is what causes this temporary condition. The Server service uses work items, such as a request to extend a .pst file, to handle I/O requests that come in over the network. These work items are queued in the Server service work queues. From there, they are handled by the Server service worker threads. The work items are allocated from a kernel resource that is called the nonpaged pool (NPP). The Server service sends these I/O requests to the disk subsystem. If, for reasons that are mentioned above, the disk subsystem does not respond in time, the incoming I/O requests are queued by using work items in the server work queues. Because these work items are allocated from the NPP, this resource eventually runs out. Running out of NPP causes systems to eventually stop responding and to log event ID 2019.
Consider one more scenario that is not specifically called out in the KB: real world file server performance. If you have 400 users with 1GB PST files located on a single file server what happens when they all come in at 8AM and load up Outlook? Outlook is going to try to load the entire PST at once and your file server is going to be asked to deliver 400GB of data to 400 users simultaneously. Yet again, A Bad Thing. These are obviously make believe numbers but hopefully you get the idea.
Microsoft makes a few recommendations in the KB, all of which are feasible, but I would add a big one that is becoming more popular daily – Archiving. I’ll write another post that covers some archiving options in the coming weeks but until then know that it allows you to move your user’s older and less frequently accessed e-mail to slower, cheaper storage with minimal impact to the user experience. It also allows you to provide a “bottomless mailbox” as far as your users are concerned as they will never hit their quotas if your policies are designed properly.
[ LINK ] to Q297019 – Personal folder files are unsupported over a LAN or over a WAN link
With the release of Microsoft Windows Server 2008 with Hyper-V and Microsoft Hyper-V Server 2008, a virtualized Exchange 2007 SP1 server is no longer restricted to the realm of the lab; it can be deployed in a production environment and receive full support from Microsoft. This past August, we published our support policies and recommendations for virtualizing Exchange, but many people have asked us to go beyond that guidance and weigh-in on the more philosophical question: is virtualization is a good idea when it comes to Exchange?
Due to the performance and business requirements of Exchange, most deployments would benefit from deployment on physical servers. However, there are some scenarios in which a virtualized Exchange 2007 infrastructure may allow you to realize real benefits in terms of space, power, and deployment flexibility. Presented here are sample scenarios in which virtualization may make sense, as well as checklists to help you evaluate whether the current load on your infrastructure makes it a good candidate for virtualization.
via You Had Me At EHLO… : Should You Virtualize Your Exchange 2007 SP1 Environment?.
1. Start the Exchange Management Console.
2. In the console tree, click the Recipient Configuration node.
3. In the action pane, click New Mailbox. The New Mailbox wizard appears.
4. On the Introduction page, click User Mailbox, and then click Next.
5. On the User Type page, click New User, and then click Next.
6. On the User Information page, complete the following fields:
· Organizational unit By default, the New Mailbox wizard displays the Users container in Active Directory. To change the default organizational unit (OU), click Browse, and then select the OU you want.
· First name Type the first name of the user. This field is optional.
· Initials Type the initials of the user. This field is optional.
· Last name Type the last name of the user. This field is optional.
· Name By default, this field is populated with the user’s first name, initials, and last name. You can modify the name in this field.
· User logon name (User Principal Name) Type the name that the user will use to log on to the mailbox. The user logon name consists of a user name and a suffix. Typically, the suffix is the domain name in which the user account resides.
· User logon name (pre-Windows 2000) Type the user name for the user that is compatible with the legacy versions of Microsoft Windows (prior to the release of Windows 2000 Server). This field is automatically populated based on the User logon name (User Principal Name) field. This field is required.
· Password Type the password that the user must use to log on to his or her mailbox.
· Confirm password Retype the password that you entered in the Password field.
· User must change password at next logon Select this check box if you want the user to reset the password.
7. Click Next.
8. On the Mailbox Settings page, complete the following fields:
· Alias By default, this field is populated based on the User logon name (User Principal Name) of the user. You can modify the alias in this field. If the user logon name contains any characters that are not valid for the alias field, they will be replaced by underscore characters (_). The alias cannot exceed 64 characters and must be unique in the forest.
· Mailbox database Click Browse to open the Select Mailbox Database dialog box. This dialog box lists all the mailbox databases in your Exchange organization. By default, the mailbox databases are sorted by name. You can also click the title of the corresponding column to sort the databases by storage group name or server name. Select the mailbox database you want to use, and then click OK.
· Managed folder mailbox policy To specify a messaging records management (MRM) policy, select this check box, and then click Browse to select the MRM mailbox policy to be associated with this mailbox. For example, use this option if you want this mailbox to adhere to an MRM policy such as the retention period for the mailbox data. To learn more about managed folder mailbox policies, see Understanding Messaging Records Management. This is an optional field.
· Exchange ActiveSync mailbox policy To specify an Exchange ActiveSync mailbox policy, select this check box, and then click Browse to select the Exchange ActiveSync mailbox policy to be associated with this mailbox. To learn more about Exchange ActiveSync mailbox policies, see Understanding Exchange ActiveSync Mailbox Policies. This is an optional field.
9. Click Next.
10. On the New Mailbox page, review the Configuration Summary. To make any configuration changes, click Back. To create the new mailbox, click New.
11. On the Completion page, the Summary states whether the mailbox was successfully created. The summary also displays the Exchange Management Shell command that was used to create the mailbox.
12. Click Finish.
Courtesy of http://technet.microsoft.com/en-us/library/aa998197.aspx
Michael B. Smith posted a good article on VSS and Exchange recently. If you’re an Exchange admin it’s definitely worth the read.
The Volume Shadow Copy Service (VSS) was originally added to Windows Server 2003. The first version of Exchange Server to support VSS was Exchange Server 2003. The primary reason for the existence of VSS is to improve the performance of backup operations. VSS is only available for NTFS volumes.
I will not cover specific implementation details here, but concepts; some of those concepts will be specific to NTFS and Exchange specific and some will be general to VSS.
via Volume Shadow Copy Services (VSS) and Exchange – The Basics – Michael’s meanderings….