Good Afternoon,
I have two DAG servers that will not join to the DAG group regardless if I add DAG1 or DAG2 first. The first server that I add will join the DAG successfully, but the next one will not join regardless of the sequence. Please see the following specs below:
OS - Windows 2008 R2 SP1 (Hyper-V Machines) - both
Exchange 2010 SP1 - same patch levels - Exchange Trusted System is added on the servers
No AV or Firewall Enabled, TCP/IP Chimney/Offload Checksum Disabled, Two NICs each with 192.x.x.x and 10.x.x.x
Below is the add-database dagtask. Please advise.
add-databaseavailabiltygroupserver started on machine DAG02.
[2013-03-13T19:21:34] add-dagserver started
[2013-03-13T19:21:34] commandline: $scriptCmd = {& $wrappedCmd @PSBoundParameters }
[2013-03-13T19:21:34] Option 'Identity' = 'DAG-GROUP'.
[2013-03-13T19:21:34] Option 'MailboxServer' = 'DAG02'.
[2013-03-13T19:21:34] Option 'DatabaseAvailabilityGroupIpAddresses' = ''.
[2013-03-13T19:21:34] Option 'WhatIf' = ''.
[2013-03-13T19:21:34] Process: w3wp w3wp.exe:4752.
[2013-03-13T19:21:34] User context = 'NT AUTHORITY\SYSTEM'.
[2013-03-13T19:21:34] Member of group 'Everyone'.
[2013-03-13T19:21:34] Member of group 'BUILTIN\Users'.
[2013-03-13T19:21:34] Member of group 'NT AUTHORITY\SERVICE'.
[2013-03-13T19:21:34] Member of group 'CONSOLE LOGON'.
[2013-03-13T19:21:34] Member of group 'NT AUTHORITY\Authenticated Users'.
[2013-03-13T19:21:34] Member of group 'NT AUTHORITY\This Organization'.
[2013-03-13T19:21:34] Member of group 'BUILTIN\IIS_IUSRS'.
[2013-03-13T19:21:34] Member of group 'LOCAL'.
[2013-03-13T19:21:34] Member of group 'IIS APPPOOL\MSExchangePowerShellAppPool'.
[2013-03-13T19:21:34] Member of group 'BUILTIN\Administrators'.
[2013-03-13T19:21:34] Updated Progress 'Validating the parameters.' 2%.
[2013-03-13T19:21:34] Working
[2013-03-13T19:21:34] Mailbox server: value passed in = DAG02, mailboxServer.Name = DAG02, mailboxServer.Fqdn = DAG02.domain.com
[2013-03-13T19:21:34] LogClussvcState: clussvc is Stopped on DAG02.domain.com. Exception (if any) = none
[2013-03-13T19:21:34] The IP addresses for the DAG are (blank means DHCP): 10.10.4.132
[2013-03-13T19:21:34] Looking up IP addresses for DAG-GROUP.
[2013-03-13T19:21:34] DAG-GROUP = [ 10.10.4.132 ].
[2013-03-13T19:21:34] Looking up IP addresses for DAG02.
[2013-03-13T19:21:34] DAG02 = [ fe80::750d:cd25:bb29:8f58%11, fe80::f50d:77f0:3c65:6388%16, 10.10.4.131, 192.168.1.20 ].
[2013-03-13T19:21:34] Looking up IP addresses for DAG02.domain.com.
[2013-03-13T19:21:34] DAG02.domain.com = [ fe80::750d:cd25:bb29:8f58%11, fe80::f50d:77f0:3c65:6388%16, 10.10.4.131, 192.168.1.20 ].
[2013-03-13T19:21:34] DAG DAG-GROUP has 1 servers:
[2013-03-13T19:21:34] DAG DAG-GROUP contains server DAG01.
[2013-03-13T19:21:34] Updated Progress 'Checking if Mailbox server 'DAG02' is in a database availability group.' 4%.
[2013-03-13T19:21:34] Working
[2013-03-13T19:21:34] GetRemoteCluster() for the mailbox server failed with exception = An Active Manager operation failed. Error: An error occurred while attempting a cluster operation. Error: Cluster API '"OpenCluster
(DAG02.domain.com) failed with 0x6d9. Error: There are no more endpoints available from the endpoint mapper"' failed... This is OK.
[2013-03-13T19:21:34] Ignoring previous error, as it is acceptable if the cluster does not exist yet.
[2013-03-13T19:21:34] DumpClusterTopology: Opening remote cluster DAG-GROUP.
[2013-03-13T19:21:34] Dumping the cluster by connecting to: DAG-GROUP.
[2013-03-13T19:21:34] The cluster's name is: DAG-GROUP.
[2013-03-13T19:21:34] Groups
[2013-03-13T19:21:34] group: Available Storage [not a CMS]
[2013-03-13T19:21:34] OwnerNode: DAG01.domain.com
[2013-03-13T19:21:34] State: Offline
[2013-03-13T19:21:34] group: Cluster Group [Cluster Main Group]
[2013-03-13T19:21:34] OwnerNode: DAG01.domain.com
[2013-03-13T19:21:34] State: Online
[2013-03-13T19:21:34] Resource: Cluster IP Address [Online, type = IP Address, PossibleOwners = DAG01 ]
[2013-03-13T19:21:34] Address = [10.10.4.132]
[2013-03-13T19:21:34] EnableDhcp = [0]
[2013-03-13T19:21:34] Network = [Cluster Network 1]
[2013-03-13T19:21:34] Resource: Cluster Name [Online, type = Network Name, PossibleOwners = DAG01 ]
[2013-03-13T19:21:34] NetName = [DAG-GROUP]
[2013-03-13T19:21:34] Nodes
[2013-03-13T19:21:34] node: DAG01.domain.com [ state = Up ]
[2013-03-13T19:21:34] Subnets
[2013-03-13T19:21:34] Name(Cluster Network 1), Mask(10.10.0.0/20), Role(ClusterNetworkRoleInternalAndClient)
[2013-03-13T19:21:34] NIC 10.10.4.130 on Node DAG01 in State=Up
[2013-03-13T19:21:34] Name(Cluster Network 2), Mask(192.168.1.0/27), Role(ClusterNetworkRoleInternalUse)
[2013-03-13T19:21:34] NIC 192.168.1.10 on Node DAG01 in State=Up
[2013-03-13T19:21:34] Opening the cluster on nodes [DAG01].
[2013-03-13T19:21:34] Other mailbox servers in the DAG are already members of cluster 'DAG-GROUP'
[2013-03-13T19:21:34] The server DAG02 does not belong to a cluster, and the other servers belong to DAG-GROUP.
[2013-03-13T19:21:34] Successfully resolved the servers based on the stopped servers list.
[2013-03-13T19:21:34] The following servers are in the StartedServers list (The list is the StartedServers property of the DAG in AD):
[2013-03-13T19:21:34] The following servers are in the StoppedServers list:
[2013-03-13T19:21:34] Verifiying that the members of database availability group 'DAG-GROUP' are also members of the cluster.
[2013-03-13T19:21:34] Verifying that the members of cluster 'DAG-GROUP' are also members of the database availability group.
[2013-03-13T19:21:34] According to GetNodeClusterState(), the server DAG02 is NotConfigured.
[2013-03-13T19:21:34] The CNO is currently Online.
[2013-03-13T19:21:34] InternalValidate() done.
[2013-03-13T19:21:35] Updated Progress 'Adding server 'DAG02' to database availability group 'DAG-GROUP'.' 6%.
[2013-03-13T19:21:35] Working
[2013-03-13T19:21:35] Updated Progress 'Adding server 'DAG02' to the cluster.' 8%.
[2013-03-13T19:21:35] Working
[2013-03-13T19:27:53] The following log entry comes from a different process that's running on machine 'DAG01.domain.com'. BEGIN
[2013-03-13T19:27:53] [2013-03-13T19:21:35] Opening a local AmCluster handle.
[2013-03-13T19:21:35] Updated Progress 'Adding server 'DAG02' to database availability group 'DAG-GROUP'.' 2%.
[2013-03-13T19:21:35] Working
[2013-03-13T19:21:35] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 12, szObjectName = DAG02,
dwStatus = 0x0 )
[2013-03-13T19:21:35] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 12, szObjectName = , dwStatus =
0x0 )
[2013-03-13T19:21:35] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseGettingCurrentMembership, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 25, szObjectName =
DAG-GROUP, dwStatus = 0x0 )
[2013-03-13T19:21:35] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseGettingCurrentMembership, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 25, szObjectName = ,
dwStatus = 0x0 )
[2013-03-13T19:21:35] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseAddNodeToCluster, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 37, szObjectName = DAG02,
dwStatus = 0x0 )
[2013-03-13T19:21:35] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseAddNodeToCluster, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 37, szObjectName = , dwStatus =
0x0 )
[2013-03-13T19:21:36] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNetft, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 50, szObjectName = DAG02,
dwStatus = 0x0 )
[2013-03-13T19:21:36] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNetft, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 50, szObjectName = , dwStatus = 0x0
)
[2013-03-13T19:21:36] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateClusDisk, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 62, szObjectName = DAG02,
dwStatus = 0x0 )
[2013-03-13T19:21:36] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateClusDisk, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 62, szObjectName = , dwStatus =
0x0 )
[2013-03-13T19:21:36] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseConfigureClusSvc, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 75, szObjectName = DAG02,
dwStatus = 0x0 )
[2013-03-13T19:21:36] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseConfigureClusSvc, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 75, szObjectName = , dwStatus =
0x0 )
[2013-03-13T19:21:36] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseStartingClusSvc, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 87, szObjectName =
DAG02.domain.com, dwStatus = 0x0 )
[2013-03-13T19:21:36] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseStartingClusSvc, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 87, szObjectName = , dwStatus =
0x0 )
[2013-03-13T19:21:36] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseNodeUp, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 100, szObjectName = DAG02, dwStatus =
0x0 )
[2013-03-13T19:24:36] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseNodeUp, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseFatal, dwPercentComplete = 100, szObjectName = DAG02, dwStatus = 0x5b4 )
[2013-03-13T19:24:36] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseFailureCleanup, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 100, szObjectName = DAG02,
dwStatus = 0x0 )
[2013-03-13T19:27:53] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseFailureCleanup, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 100, szObjectName = , dwStatus =
0x0 )
[2013-03-13T19:27:53] The preceding log entry comes from a different process running on computer 'DAG01.domain.com'. END
[2013-03-13T19:27:53] The operation wasn't successful because an error was encountered. You may find more details in log file "C:\ExchangeSetupLogs\DagTasks\dagtask_2013-03-13_19-21-34.631_add-databaseavailabiltygroupserver.log".
[2013-03-13T19:27:53] WriteError! Exception = Microsoft.Exchange.Cluster.Replay.DagTaskOperationFailedException: A server-side database availability group administrative operation failed. Error: The operation failed. CreateCluster errors
may result from incorrectly configured static addresses. Error: An error occurred while attempting a cluster operation. Error: Cluster API '"AddClusterNode() (MaxPercentage=100) failed with 0x5b4. Error: This operation returned because
the timeout period expired"' failed. ---> Microsoft.Exchange.Cluster.Replay.AmClusterApiException: An Active Manager operation failed. Error: An error occurred while attempting a cluster operation. Error: Cluster API '"AddClusterNode()
(MaxPercentage=100) failed with 0x5b4. Error: This operation returned because the timeout period expired"' failed.. ---> System.ComponentModel.Win32Exception: This operation returned because the timeout period expired
--- End of inner exception stack trace ---
at Microsoft.Exchange.Cluster.ClusApi.AmCluster.AddNodeToCluster(AmServerName nodeName, IClusterSetupProgress setupProgress, IntPtr context, Exception& errorException, Boolean throwExceptionOnFailure)
at Microsoft.Exchange.Cluster.Replay.DagHelper.AddDagClusterNode(AmServerName mailboxServerName, String& verboseLog)
--- End of inner exception stack trace (Microsoft.Exchange.Cluster.Replay.AmClusterApiException) ---
at Microsoft.Exchange.Cluster.Replay.DagHelper.ThrowDagTaskOperationWrapper(Exception exception)
at Microsoft.Exchange.Cluster.Replay.DagHelper.AddDagClusterNode(AmServerName mailboxServerName, String& verboseLog)
at Microsoft.Exchange.Cluster.ReplayService.ReplayRpcServer.<>c__DisplayClass34.<RpcsAddNodeToCluster>b__33()
at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.RunRpcServerOperation(String databaseName, RpcServerOperation rpcOperation)
--- End of stack trace on server (DAG01.domain.com) ---
at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.ClientRethrowIfFailed(String databaseName, String serverName, RpcErrorExceptionInfo errorInfo)
at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunRpcOperationDbName(AmServerName serverName, String databaseName, Int32 timeoutMs, IHaRpcExceptionWrapper rpcExceptionWrapperInstance, InternalRpcOperation rpcOperation)
at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunRpcOperation(AmServerName serverName, Nullable`1 dbGuid, Int32 timeoutMs, IHaRpcExceptionWrapper rpcExceptionWrapperInstance, InternalRpcOperation rpcOperation)
at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunAddNodeToCluster(AmServerName serverName, AmServerName newNode, String& verboseLog)
at Microsoft.Exchange.Management.SystemConfigurationTasks.AddDatabaseAvailabilityGroupServer.JoinNodeToCluster()
[2013-03-13T19:27:53] Updated Progress 'Done!' 100%.
[2013-03-13T19:27:53] COMPLETED
add-databaseavailabiltygroupserver explicitly called CloseTempLogFile().