Hi Experts,
I have an Exchange 2010 HA environment with the below specs.
3 Node DAG ( 2 nodes in the Primary site and 1 node in the DR site) , 2 node cluster for HUb/CAS , 1 HUB/CAS in DR site.
Last 2 years our DR test was successful , but after we have upgraded to Exchange 2010 SP2 RU3 we have not tested the DR site. this friday when we tested our DR site , we were unable to mount the mailbox databases on the DR mailbox server. Below are the steps followed for it.
1. Shut down all the DC, Exchange servers in the Primary site.
2. Stop-DatabaseAvailabilityGroup –Identity SME-HO-Vmbx01 –ActiveDirectorySite Default-First-Site-Name –ConfigurationOnly - this command executed on the DR MBX server.
3. Stop-Service ClusSvc - his command executed on the DR MBX server.
4. Restore-DatabaseAvailabilityGroup –Identity SME-HO-Vmbx01 –ActiveDirectorySite DR
5. After the above command waited for almost 1hour, but the mailbox databases were not getting mounted on the DR mBX server. FOund the below events in the application log.
Log Name: ApplicationSource: MSExchangeRepl
Date: 2/22/2013 8:39:13 PM
Event ID: 3154
Task Category: Service
Level: Error
Keywords: Classic
User: N/A
Computer: SME-DR-MBX01.SERCO.AE
Description:
Active Manager failed to mount database Normal-Users on server SME-HO-MBX01.SERCO.AE. Error: An Active Manager operation failed. Error The database action failed. Error: An error occurred while trying to select a database copy for possible activation. Error: The database 'Normal-Users' was not mounted because errors occurred either while validating database copies for possible activation, or while attempting to activate another copy. Detailed error(s):
sme-ho-mbx01:
An Active Manager operation failed. Error Invalid Active Manager configuration. Error: Automount consensus not reached.. [Server: SME-HO-MBX01.SERCO.AE]
sme-ho-mbx02:
Database copy 'Normal-Users' is in a failed state on server 'SME-HO-MBX02.SERCO.AE'. Reason: The Microsoft Exchange Replication service encountered a transient error while attempting to start a replication instance for Normal-Users\SME-HO-MBX02. The copy will be set to failed. Error: The NetworkManager has not yet been initialized. Check the event logs to determine the cause.
sme-dr-mbx01:
Database copy 'Normal-Users' is in a failed state on server 'SME-DR-MBX01.SERCO.AE'. Reason: The Microsoft Exchange Replication service encountered a transient error while attempting to start a replication instance for Normal-Users\SME-DR-MBX01. The copy will be set to failed. Error: The NetworkManager has not yet been initialized. Check the event logs to determine the cause.
.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="MSExchangeRepl" />
<EventID Qualifiers="49156">3154</EventID>
<Level>2</Level>
<Task>1</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2013-02-22T16:39:13.000000000Z" />
<EventRecordID>358399</EventRecordID>
<Channel>Application</Channel>
<Computer>SME-DR-MBX01.SERCO.AE</Computer>
<Security />
</System>
<EventData>
<Data>Normal-Users</Data>
<Data>SME-HO-MBX01.SERCO.AE</Data>
<Data>An Active Manager operation failed. Error The database action failed. Error: An error occurred while trying to select a database copy for possible activation. Error: The database 'Normal-Users' was not mounted because errors occurred either while validating database copies for possible activation, or while attempting to activate another copy. Detailed error(s):
sme-ho-mbx01:
An Active Manager operation failed. Error Invalid Active Manager configuration. Error: Automount consensus not reached.. [Server: SME-HO-MBX01.SERCO.AE]
sme-ho-mbx02:
Database copy 'Normal-Users' is in a failed state on server 'SME-HO-MBX02.SERCO.AE'. Reason: The Microsoft Exchange Replication service encountered a transient error while attempting to start a replication instance for Normal-Users\SME-HO-MBX02. The copy will be set to failed. Error: The NetworkManager has not yet been initialized. Check the event logs to determine the cause.
sme-dr-mbx01:
Database copy 'Normal-Users' is in a failed state on server 'SME-DR-MBX01.SERCO.AE'. Reason: The Microsoft Exchange Replication service encountered a transient error while attempting to start a replication instance for Normal-Users\SME-DR-MBX01. The copy will be set to failed. Error: The NetworkManager has not yet been initialized. Check the event logs to determine the cause.
.</Data>
</EventData>
</Event>
6. Then gave the restore- Dag COMMAND, EVEN that gave the same above error
7. Another event noticed in the application log
Log Name: Application
Date: 2/22/2013 8:17:25 PM
Event ID: 2060
Task Category: Service
Level: Error
Keywords: Classic
User: N/A
Computer: SME-DR-MBX01.SERCO.AE
Description:
The Microsoft Exchange Replication service encountered a transient error while attempting to start a replication instance for Limited Users\SME-DR-MBX01. The copy will be set to failed. Error: The NetworkManager has not yet been initialized. Check the event logs to determine the cause.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="MSExchangeRepl" />
<EventID Qualifiers="49156">2060</EventID>
<Level>2</Level>
<Task>1</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2013-02-22T16:17:25.000000000Z" />
<EventRecordID>358355</EventRecordID>
<Channel>Application</Channel>
<Computer>SME-DR-MBX01.SERCO.AE</Computer>
<Security />
</System>
<EventData>
<Data>Limited Users\SME-DR-MBX01</Data>
<Data>The NetworkManager has not yet been initialized. Check the event logs to determine the cause.</Data>
</EventData>
</Event>
8. Please provide your valuable inputs urgently, as our DR failover was a failure and we need to fix any underlying issue and go for another test.
Md.Abubakar Noorani IT Systems Engineer Serco Ltd.