GroupFailoverTest-FiveNodeGroup-1 26
GroupFailoverTest-FiveNodeGroup-2 25
GroupFailoverTest-FiveNodeGroup-3 23
GroupFailoverTest-FiveNodeGroup-4 22
GroupFailoverTest-FiveNodeGroup-5 29

4398 9020218
4301 175
4346 3
4434 5
4357 13
4362 9994
4402 24
4349 2
4437 0
4441 10104
4366 14
4405 4
4308 4
4353 11
4356 10137
4444 24
4369 4
4408 1
4311 6
4316 10113
4361 15
4449 3
4413 30
4373 0

bash-3.1$ ls
5nodes.pl	     FiveNodeGroup-3.cdp  SixNodeGroup-1.cdp	fail.pl		    nm.sh~	 trigger.pl
5nodes1.txt	     FiveNodeGroup-4.cdp  SixNodeGroup-2.cdp	hm-node-1-util.dat  ns.ior
Alpha-client.cdp     FiveNodeGroup-5.cdp  ThreeNodeGroup-1.cdp	hm-node-1.ior	    rm.ior
FiveNodeGroup-1.cdp  FourNodeGroup-1.cdp  TwoNodeGroup-1.cdp	hm.sh		    run_test.pl
FiveNodeGroup-2.cdp  NineNodeGroup-1.cdp  deploy.pl		nm.sh		    start_nm.pl
bash-3.1$ clear
bash-3.1$ ./run_test.pl 
HOST=node-1
start Name_Service ...
start ReplicationManager ...
RM: Registered successfully node-1:ReplicationManager-node-1:ReplicationManager:1 with Replication manager.
start HostMonitor ...
start remote HostMonitor on node node-2 ...
start remote HostMonitor on node node-3 ...
start remote HostMonitor on node node-4 ...
start remote HostMonitor on node node-5 ...
AppSideMonitor_Thread::svc - got heartbeat port 7000 from hm.
AppSideMonitor_Thread: Entering reactor event loop.
HostMonitorImpl::register_process process_id = ReplicationManager-node-1, host = node-1, port = 7000.
start remote HostMonitor on node node-6 ...
start remote NodeManager on node node-2 ...
start remote NodeManager on node node-3 ...
start remote NodeManager on node node-4 ...
start remote NodeManager on node node-5 ...
start remote NodeManager on node node-6 ...
start ExecutionManager ...
starting Node_Manager...
starting Node_Manager...
starting Node_Manager...
starting Node_Manager...
starting Node_Manager...
start FaultCorrelationManager ...
(6663|3074066912) [LM_TRACE] -  10:37:47.995270 - Logging level is set to 10
(6663|3074066912) [LM_TRACE] -  10:37:47.997748 - Module_main.h - initializing ORB
(6663|3074066912) [LM_TRACE] -  10:37:48.008263 - Module_Main.h - initializing module instance(3074066912) calling FaultCorrelationManager_Module::FaultCorrelationManager_Module in file `FaultCorrelationManager_Module.cpp' on line 53
(3074066912) leaving FaultCorrelationManager_Module::FaultCorrelationManager_Module
(6663|3074066912) [LM_TRACE] -  10:37:48.015240 - FaultCorrelationManager_Module::create_object - FCM's orb is nil.
(6663|3074066912) [LM_TRACE] -  10:37:48.017566 - FaultCorrelationManager_Module::parse_args - ignoring unknown option 1
(3074066912) calling DAnCE_FaultCorrelationManager_Module::register_value_factories (void) in file `FaultCorrelationManager_Module.cpp' on line 442
(3074066912) leaving DAnCE_FaultCorrelationManager_Module::register_value_factories (void)
(3074066912) calling DAnCE_FaultCorrelationManager_Module::create_poas in file `FaultCorrelationManager_Module.cpp' on line 454
(6663|3074066912) [LM_TRACE] -  10:37:48.024605 - FaultCorrelationManager_Module::create_poas - Resolving root POA
(6663|3074066912) [LM_TRACE] -  10:37:48.034668 - FaultCorrelationManager_Module::create_poas - Obtaining the POAManager
(6663|3074066912) [LM_TRACE] -  10:37:48.034916 - FaultCorrelationManager_Module::create_poas - FaultCorrelationManager_Module::create_poas - Creating the "Managers" POA.
(3074066912) leaving DAnCE_FaultCorrelationManager_Module::create_poas
(6663|3074066912) [LM_DEBUG] -  10:37:48.041296 - FaultCorrelationManager_Module::create_object - creating FaultCorrelationManager
(6663|3074066912) [LM_TRACE] -  10:37:48.051042 - FaultCorrelationManager_Impl::FaulCorrelationManager_Impl - Binding property exec-mgr provided by caller.
(6663|3074066912) [LM_TRACE] -  10:37:48.051254 - FaultCorrelationManager_Module::create_object - New FaultCorrelationMananger servant instance for NodeManager allocated.
(6663|3074066912) [LM_TRACE] -  10:37:48.056054 - FaultCorrelationManager_Module::create_object - Writing node IOR IOR:010000002600000049444c3a44416e43452f4661756c74436f7272656c6174696f6e4d616e616765723a312e30000000020000000000000098000000010102000f0000006e6f64652d312d6269672d6c616e0000d59900004300000014010f004e5550000000190000000001000000526f6f74504f41004d616e61676572730000000000010000004661756c74436f7272656c6174696f6e4d616e616765720002000000000000000800000001363636004f4154010000001800000001363636010001000100000001000105090101000000000000000000a4000000010102001c000000706333362e697369736c61622e76616e64657262696c742e65647500d59950004300000014010f004e5550000000190000000001000000526f6f74504f41004d616e61676572730000000000010000004661756c74436f7272656c6174696f6e4d616e616765720002000000000000000800000001000000004f41540100000018000000010000000100010001000000010001050901010000000000 to file EM.ior.
(6663|3074066912) [LM_NOTICE] -  10:37:48.082479 - FaultCorrelationManager_Module::create_object - FaultCorrelationManager is running...
(6663|3074066912) [LM_DEBUG] -  10:37:48.082671 - FaultCorrelationManager_Module::create_object - FaultCorrelationManager IOR: IOR:010000002600000049444c3a44416e43452f4661756c74436f7272656c6174696f6e4d616e616765723a312e30000000020000000000000098000000010102000f0000006e6f64652d312d6269672d6c616e0000d59900004300000014010f004e5550000000190000000001000000526f6f74504f41004d616e61676572730000000000010000004661756c74436f7272656c6174696f6e4d616e616765720002000000000000000800000001363636004f4154010000001800000001363636010001000100000001000105090101000000000000000000a4000000010102001c000000706333362e697369736c61622e76616e64657262696c742e65647500d59950004300000014010f004e5550000000190000000001000000526f6f74504f41004d616e61676572730000000000010000004661756c74436f7272656c6174696f6e4d616e616765720002000000000000000800000001000000004f41540100000018000000010000000100010001000000010001050901010000000000
FaultCorrelationManager_Impl::svc (void)
(6663|3074066912) [LM_DEBUG] -  10:37:48.088008 - FaultCorrelationManager_Module::create_object - activated thread for processing failure notifications.
FCM: instance[Alpha-1] -> plan[GroupFailoverTest-FiveNodeGroup-1]
FCM: node[node-2] -> oid[Alpha] -> instance[Alpha-1]
FCM: instance[Beta-1] -> plan[GroupFailoverTest-FiveNodeGroup-1]
FCM: node[node-3] -> oid[Beta] -> instance[Beta-1]
FCM: instance[Gamma-1] -> plan[GroupFailoverTest-FiveNodeGroup-1]
FCM: node[node-4] -> oid[Gamma] -> instance[Gamma-1]
FCM: instance[Delta-1] -> plan[GroupFailoverTest-FiveNodeGroup-1]
FCM: node[node-5] -> oid[Delta] -> instance[Delta-1]
FCM: instance[Epsilon-1] -> plan[GroupFailoverTest-FiveNodeGroup-1]
FCM: node[node-6] -> oid[Epsilon] -> instance[Epsilon-1]
RM: received ranklist constraints:
|	oid Beta:
|		node-3
|	oid Alpha:
|		node-2
|	oid Epsilon:
|		node-6
|	oid Gamma:
|		node-4
|	oid Delta:
|		node-5
HostMonitorImpl::register_process process_id = 4398, host = node-2, port = 7000.
***** vp: 3065705330
***** cc: 3065705330
HostMonitorImpl::register_process process_id = 4357, host = node-3, port = 7000.
***** vp: 3065275250
***** cc: 3065275250
HostMonitorImpl::register_process process_id = 4434, host = node-4, port = 7000.
***** vp: 3065377650
***** cc: 3065377650
HostMonitorImpl::register_process process_id = 4346, host = node-5, port = 7000.
***** vp: 3065164658
***** cc: 3065164658
HostMonitorImpl::register_process process_id = 4301, host = node-6, port = 7000.
***** vp: 3064951666
***** cc: 3064951666
RM: Registered successfully node-6:4301:Epsilon:1 with Replication manager.
RM: Registered successfully node-5:4346:Delta:1 with Replication manager.
RM: Registered successfully node-4:4434:Gamma:1 with Replication manager.
RM: Registered successfully node-3:4357:Beta:1 with Replication manager.
RM: Registered successfully node-2:4398:Alpha:1 with Replication manager.
FCM: instance[Alpha-2] -> plan[GroupFailoverTest-FiveNodeGroup-2]
FCM: node[node-3] -> oid[Alpha] -> instance[Alpha-2]
FCM: instance[Beta-2] -> plan[GroupFailoverTest-FiveNodeGroup-2]
FCM: node[node-4] -> oid[Beta] -> instance[Beta-2]
FCM: instance[Gamma-2] -> plan[GroupFailoverTest-FiveNodeGroup-2]
FCM: node[node-5] -> oid[Gamma] -> instance[Gamma-2]
FCM: instance[Delta-2] -> plan[GroupFailoverTest-FiveNodeGroup-2]
FCM: node[node-6] -> oid[Delta] -> instance[Delta-2]
FCM: instance[Epsilon-2] -> plan[GroupFailoverTest-FiveNodeGroup-2]
FCM: node[node-2] -> oid[Epsilon] -> instance[Epsilon-2]
RM: received ranklist constraints:
|	oid Beta:
|		node-3
|		node-4
|	oid Alpha:
|		node-2
|		node-3
|	oid Epsilon:
|		node-6
|		node-2
|	oid Gamma:
|		node-4
|		node-5
|	oid Delta:
|		node-5
|		node-6
HostMonitorImpl::register_process process_id = 4362, host = node-3, port = 7001.
***** vp: 3065836402
***** cc: 3065836402
HostMonitorImpl::register_process process_id = 4437, host = node-4, port = 7001.
***** vp: 3065197426
***** cc: 3065197426
HostMonitorImpl::register_process process_id = 4349, host = node-5, port = 7001.
***** vp: 3065750386
***** cc: 3065750386
HostMonitorImpl::register_process process_id = 4304, host = node-6, port = 7001.
***** vp: 3064918898
***** cc: 3064918898
HostMonitorImpl::register_process process_id = 4402, host = node-2, port = 7001.
***** vp: 3064927090
***** cc: 3064927090
RM: Registered successfully node-2:4402:Epsilon:2 with Replication manager.
RM: Registered successfully node-6:4304:Delta:2 with Replication manager.
RM: Registered successfully node-5:4349:Gamma:2 with Replication manager.
RM: Registered successfully node-4:4437:Beta:2 with Replication manager.
RM: Registered successfully node-3:4362:Alpha:2 with Replication manager.
FCM: instance[Alpha-3] -> plan[GroupFailoverTest-FiveNodeGroup-3]
FCM: node[node-4] -> oid[Alpha] -> instance[Alpha-3]
FCM: instance[Beta-3] -> plan[GroupFailoverTest-FiveNodeGroup-3]
FCM: node[node-5] -> oid[Beta] -> instance[Beta-3]
FCM: instance[Gamma-3] -> plan[GroupFailoverTest-FiveNodeGroup-3]
FCM: node[node-6] -> oid[Gamma] -> instance[Gamma-3]
FCM: instance[Delta-3] -> plan[GroupFailoverTest-FiveNodeGroup-3]
FCM: node[node-2] -> oid[Delta] -> instance[Delta-3]
FCM: instance[Epsilon-3] -> plan[GroupFailoverTest-FiveNodeGroup-3]
FCM: node[node-3] -> oid[Epsilon] -> instance[Epsilon-3]
RM: received ranklist constraints:
|	oid Beta:
|		node-3
|		node-4
|		node-5
|	oid Alpha:
|		node-2
|		node-3
|		node-4
|	oid Epsilon:
|		node-6
|		node-2
|		node-3
|	oid Gamma:
|		node-4
|		node-5
|		node-6
|	oid Delta:
|		node-5
|		node-6
|		node-2
HostMonitorImpl::register_process process_id = 4441, host = node-4, port = 7002.
***** vp: 3065750386
***** cc: 3065750386
HostMonitorImpl::register_process process_id = 4353, host = node-5, port = 7002.
***** vp: 3065533298
***** cc: 3065533298
HostMonitorImpl::register_process process_id = 4308, host = node-6, port = 7002.
***** vp: 3065615218
***** cc: 3065615218
HostMonitorImpl::register_process process_id = 4405, host = node-2, port = 7002.
***** vp: 3065586546
***** cc: 3065586546
HostMonitorImpl::register_process process_id = 4366, host = node-3, port = 7002.
***** vp: 3065561970
***** cc: 3065561970
RM: Registered successfully node-3:4366:Epsilon:2 with Replication manager.
RM: Registered successfully node-2:4405:Delta:2 with Replication manager.
RM: Registered successfully node-6:4308:Gamma:2 with Replication manager.
RM: Registered successfully node-5:4353:Beta:2 with Replication manager.
RM: Registered successfully node-4:4441:Alpha:2 with Replication manager.
FCM: instance[Alpha-4] -> plan[GroupFailoverTest-FiveNodeGroup-4]
FCM: node[node-5] -> oid[Alpha] -> instance[Alpha-4]
FCM: instance[Beta-4] -> plan[GroupFailoverTest-FiveNodeGroup-4]
FCM: node[node-6] -> oid[Beta] -> instance[Beta-4]
FCM: instance[Gamma-4] -> plan[GroupFailoverTest-FiveNodeGroup-4]
FCM: node[node-2] -> oid[Gamma] -> instance[Gamma-4]
FCM: instance[Delta-4] -> plan[GroupFailoverTest-FiveNodeGroup-4]
FCM: node[node-3] -> oid[Delta] -> instance[Delta-4]
FCM: instance[Epsilon-4] -> plan[GroupFailoverTest-FiveNodeGroup-4]
FCM: node[node-4] -> oid[Epsilon] -> instance[Epsilon-4]
RM: received ranklist constraints:
|	oid Beta:
|		node-3
|		node-4
|		node-5
|		node-6
|	oid Alpha:
|		node-2
|		node-3
|		node-4
|		node-5
|	oid Epsilon:
|		node-6
|		node-2
|		node-3
|		node-4
|	oid Gamma:
|		node-4
|		node-5
|		node-6
|		node-2
|	oid Delta:
|		node-5
|		node-6
|		node-2
|		node-3
HostMonitorImpl::register_process process_id = 4356, host = node-5, port = 7003.
***** vp: 3065635698
***** cc: 3065635698
HostMonitorImpl::register_process process_id = 4311, host = node-6, port = 7003.
***** vp: 3065770866
***** cc: 3065770866
HostMonitorImpl::register_process process_id = 4408, host = node-2, port = 7003.
***** vp: 3064914802
***** cc: 3064914802
HostMonitorImpl::register_process process_id = 4369, host = node-3, port = 7003.
***** vp: 3065803634
***** cc: 3065803634
HostMonitorImpl::register_process process_id = 4444, host = node-4, port = 7003.
***** vp: 3065705330
***** cc: 3065705330
RM: Registered successfully node-4:4444:Epsilon:2 with Replication manager.
RM: Registered successfully node-3:4369:Delta:2 with Replication manager.
RM: Registered successfully node-2:4408:Gamma:2 with Replication manager.
RM: Registered successfully node-6:4311:Beta:2 with Replication manager.
RM: Registered successfully node-5:4356:Alpha:2 with Replication manager.
FCM: instance[Alpha-5] -> plan[GroupFailoverTest-FiveNodeGroup-5]
FCM: node[node-6] -> oid[Alpha] -> instance[Alpha-5]
FCM: instance[Beta-5] -> plan[GroupFailoverTest-FiveNodeGroup-5]
FCM: node[node-2] -> oid[Beta] -> instance[Beta-5]
FCM: instance[Gamma-5] -> plan[GroupFailoverTest-FiveNodeGroup-5]
FCM: node[node-3] -> oid[Gamma] -> instance[Gamma-5]
FCM: instance[Delta-5] -> plan[GroupFailoverTest-FiveNodeGroup-5]
FCM: node[node-4] -> oid[Delta] -> instance[Delta-5]
FCM: instance[Epsilon-5] -> plan[GroupFailoverTest-FiveNodeGroup-5]
FCM: node[node-5] -> oid[Epsilon] -> instance[Epsilon-5]
RM: received ranklist constraints:
|	oid Beta:
|		node-3
|		node-4
|		node-5
|		node-6
|		node-2
|	oid Alpha:
|		node-2
|		node-3
|		node-4
|		node-5
|		node-6
|	oid Epsilon:
|		node-6
|		node-2
|		node-3
|		node-4
|		node-5
|	oid Gamma:
|		node-4
|		node-5
|		node-6
|		node-2
|		node-3
|	oid Delta:
|		node-5
|		node-6
|		node-2
|		node-3
|		node-4
HostMonitorImpl::register_process process_id = 4316, host = node-6, port = 7004.
***** vp: 3065664370
***** cc: 3065664370
HostMonitorImpl::register_process process_id = 4413, host = node-2, port = 7004.
***** vp: 3065574258
***** cc: 3065574258
HostMonitorImpl::register_process process_id = 4373, host = node-3, port = 7004.
***** vp: 3064959858
***** cc: 3064959858
HostMonitorImpl::register_process process_id = 4449, host = node-4, port = 7004.
***** vp: 3065680754
***** cc: 3065680754
HostMonitorImpl::register_process process_id = 4361, host = node-5, port = 7004.
***** vp: 3065185138
***** cc: 3065185138
RM: Registered successfully node-5:4361:Epsilon:2 with Replication manager.
RM: Registered successfully node-4:4449:Delta:2 with Replication manager.
RM: Registered successfully node-3:4373:Gamma:2 with Replication manager.
RM: Registered successfully node-2:4413:Beta:2 with Replication manager.
RM: Registered successfully node-6:4316:Alpha:2 with Replication manager.
It looks like process 4398 has failed.
RM: A state synchronization agent (pid=4398) died.
RM: process_proc_failure (4398)
RM: process 4398 on host node-2 failed
RM: 'Alpha' on 'node-2' failed.
FCM: app_failure ()
FaultCorrelationManager_Impl::app_failure_i (node-2, Alpha)
FCM::app_failure (Alpha, node-2): caused by component 'Alpha-1' in plan 'GroupFailoverTest-FiveNodeGroup-1'.
FCM::stop_failover_unit (GroupFailoverTest-FiveNodeGroup-1)
RM: Removed application node-2:4398:Alpha:1.
RM: Failover host = node-3.
It looks like process 4301 has failed.
It looks like process 4346 has failed.
It looks like process 4434 has failed.
(4396|3072046800) [LM_ERROR] -  10:41:15.490767 - NodeApplication_Impl::~NodeApplication_Impl - Caught CORBA exception while removing container 0 on server 0: system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'
OMG minor code (2), described as 'No usable profile in IOR.', completed = NO

FCM: instance[Alpha-1] removed
FCM: node[node-2] -> oid[Alpha] removed.
FCM: instance[Beta-1] removed
FCM: node[node-3] -> oid[Beta] removed.
FCM: instance[Gamma-1] removed
FCM: node[node-4] -> oid[Gamma] removed.
FCM: instance[Delta-1] removed
FCM: node[node-5] -> oid[Delta] removed.
FCM: instance[Epsilon-1] removed
FCM: node[node-6] -> oid[Epsilon] removed.
(4396|3072046800) [LM_ERROR] -  10:41:15.492025 - NodeApplication_Impl::~NodeApplication_Impl - Caught CORBA exception while removing server 0: system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'
OMG minor code (2), described as 'No usable profile in IOR.', completed = NO

It looks like process 4357 has failed.
RM: received ranklist constraints:
|	oid Beta:
|		node-3
|		node-4
|		node-5
|		node-6
|		node-2
|	oid Alpha:
|		node-2
|		node-3
|		node-4
|		node-5
|		node-6
|	oid Epsilon:
|		node-6
|		node-2
|		node-3
|		node-4
|		node-5
|	oid Gamma:
|		node-4
|		node-5
|		node-6
|		node-2
|		node-3
|	oid Delta:
|		node-5
|		node-6
|		node-2
|		node-3
|		node-4
FCM::stop_failover_unit (GroupFailoverTest-FiveNodeGroup-1): plan was shutdown sucessfully.
RM: A state synchronization agent (pid=4434) died.
RM: A state synchronization agent (pid=4301) died.
RM: A state synchronization agent (pid=4357) died.
RM: A state synchronization agent (pid=4346) died.
RM: A client agent agent died.
RM: A client agent agent died.
RM: A client agent agent died.
RM: A client agent agent died.
RM: A client agent agent died.
RM: process_proc_failure (4301)
RM: process 4301 on host node-6 failed
RM: 'Epsilon' on 'node-6' failed.
FCM: app_failure ()
FaultCorrelationManager_Impl::app_failure_i (node-6, Epsilon)
FCM::app_failure (Epsilon, node-6): plan for component 'Epsilon-1' not found.
RM: Removed application node-6:4301:Epsilon:1.
RM: Failover host = node-2.
RM: process_proc_failure (4346)
RM: process 4346 on host node-5 failed
RM: 'Delta' on 'node-5' failed.
FCM: app_failure ()
FaultCorrelationManager_Impl::app_failure_i (node-5, Delta)
FCM::app_failure (Delta, node-5): plan for component 'Delta-1' not found.
RM: Removed application node-5:4346:Delta:1.
RM: Failover host = node-6.
RM: process_proc_failure (4434)
RM: process 4434 on host node-4 failed
RM: 'Gamma' on 'node-4' failed.
FCM: app_failure ()
FaultCorrelationManager_Impl::app_failure_i (node-4, Gamma)
FCM::app_failure (Gamma, node-4): plan for component 'Gamma-1' not found.
RM: Removed application node-4:4434:Gamma:1.
RM: Failover host = node-5.
RM: process_proc_failure (4357)
RM: process 4357 on host node-3 failed
RM: 'Beta' on 'node-3' failed.
FCM: app_failure ()
FaultCorrelationManager_Impl::app_failure_i (node-3, Beta)
FCM::app_failure (Beta, node-3): plan for component 'Beta-1' not found.
RM: Removed application node-3:4357:Beta:1.
RM: Failover host = node-4.
It looks like process 4362 has failed.
RM: process_proc_failure (4362)
RM: process 4362 on host node-3 failed
RM: 'Alpha' on 'node-3' failed.
FCM: app_failure ()
FaultCorrelationManager_Impl::app_failure_i (node-3, Alpha)
FCM::app_failure (Alpha, node-3): caused by component 'Alpha-2' in plan 'GroupFailoverTest-FiveNodeGroup-2'.
FCM::stop_failover_unit (GroupFailoverTest-FiveNodeGroup-2)
RM: Removed application node-3:4362:Alpha:1.
RM: Failover host = node-4.
It looks like process 4402 has failed.
It looks like process 4304 has failed.
RM: process_proc_failure () : removed state sync agent from list.
It looks like process 4349 has failed.
(4355|3071919824) [LM_ERROR] -  10:41:25.560066 - NodeApplication_Impl::~NodeApplication_Impl - Caught CORBA exception while removing container 0 on server 0: system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'
OMG minor code (2), described as 'No usable profile in IOR.', completed = NO

FCM: instance[Alpha-2] removed
FCM: node[node-3] -> oid[Alpha] removed.
FCM: instance[Beta-2] removed
FCM: node[node-4] -> oid[Beta] removed.
FCM: instance[Gamma-2] removed
FCM: node[node-5] -> oid[Gamma] removed.
FCM: instance[Delta-2] removed
FCM: node[node-6] -> oid[Delta] removed.
FCM: instance[Epsilon-2] removed
FCM: node[node-2] -> oid[Epsilon] removed.
RM: A state synchronization agent (pid=4349) died.
It looks like process 4437 has failed.
(4355|3071919824) [LM_ERROR] -  10:41:25.561342 - NodeApplication_Impl::~NodeApplication_Impl - Caught CORBA exception while removing server 0: system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'
OMG minor code (2), described as 'No usable profile in IOR.', completed = NO

RM: A state synchronization agent (pid=4437) died.
RM: received ranklist constraints:
|	oid Beta:
|		node-3
|		node-4
|		node-5
|		node-6
|		node-2
|	oid Alpha:
|		node-2
|		node-3
|		node-4
|		node-5
|		node-6
|	oid Epsilon:
|		node-6
|		node-2
|		node-3
|		node-4
|		node-5
|	oid Gamma:
|		node-4
|		node-5
|		node-6
|		node-2
|		node-3
|	oid Delta:
|		node-5
|		node-6
|		node-2
|		node-3
|		node-4
FCM::stop_failover_unit (GroupFailoverTest-FiveNodeGroup-2): plan was shutdown sucessfully.
RM: A state synchronization agent (pid=4304) died.
RM: A state synchronization agent (pid=4402) died.
RM: A client agent agent died.
RM: A client agent agent died.
RM: A client agent agent died.
RM: A client agent agent died.
RM: A client agent agent died.
RM: process_proc_failure (4402)
RM: process 4402 on host node-2 failed
RM: 'Epsilon' on 'node-2' failed.
FCM: app_failure ()
FaultCorrelationManager_Impl::app_failure_i (node-2, Epsilon)
FCM::app_failure (Epsilon, node-2): plan for component 'Epsilon-2' not found.
RM: Removed application node-2:4402:Epsilon:1.
RM: Failover host = node-3.
RM: process_proc_failure (4304)
RM: process 4304 on host node-6 failed
RM: 'Delta' on 'node-6' failed.
FCM: app_failure ()
FaultCorrelationManager_Impl::app_failure_i (node-6, Delta)
FCM::app_failure (Delta, node-6): plan for component 'Delta-2' not found.
RM: Removed application node-6:4304:Delta:1.
RM: Failover host = node-2.
RM: process_proc_failure (4349)
RM: process 4349 on host node-5 failed
RM: 'Gamma' on 'node-5' failed.
FCM: app_failure ()
FaultCorrelationManager_Impl::app_failure_i (node-5, Gamma)
FCM::app_failure (Gamma, node-5): plan for component 'Gamma-2' not found.
RM: Removed application node-5:4349:Gamma:1.
RM: Failover host = node-6.
RM: process_proc_failure (4437)
RM: process 4437 on host node-4 failed
RM: 'Beta' on 'node-4' failed.
FCM: app_failure ()
FaultCorrelationManager_Impl::app_failure_i (node-4, Beta)
FCM::app_failure (Beta, node-4): plan for component 'Beta-2' not found.
RM: Removed application node-4:4437:Beta:1.
RM: Failover host = node-5.
It looks like process 4441 has failed.
RM: process_proc_failure (4441)
RM: process 4441 on host node-4 failed
RM: 'Alpha' on 'node-4' failed.
FCM: app_failure ()
FaultCorrelationManager_Impl::app_failure_i (node-4, Alpha)
FCM::app_failure (Alpha, node-4): caused by component 'Alpha-3' in plan 'GroupFailoverTest-FiveNodeGroup-3'.
FCM::stop_failover_unit (GroupFailoverTest-FiveNodeGroup-3)
RM: Removed application node-4:4441:Alpha:1.
RM: Failover host = node-5.
RM: process_proc_failure () : removed state sync agent from list.
It looks like process 4366 has failed.
It looks like process 4405 has failed.
It looks like process 4308 has failed.
(4432|3072186064) [LM_ERROR] -  10:41:35.761729 - NodeApplication_Impl::~NodeApplication_Impl - Caught CORBA exception while removing container 0 on server 0: system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'
OMG minor code (2), described as 'No usable profile in IOR.', completed = NO

FCM: instance[Alpha-3] removed
FCM: node[node-4] -> oid[Alpha] removed.
FCM: instance[Beta-3] removed
FCM: node[node-5] -> oid[Beta] removed.
FCM: instance[Gamma-3] removed
FCM: node[node-6] -> oid[Gamma] removed.
FCM: instance[Delta-3] removed
FCM: node[node-2] -> oid[Delta] removed.
FCM: instance[Epsilon-3] removed
FCM: node[node-3] -> oid[Epsilon] removed.
RM: A state synchronization agent (pid=4308) died.
(4432|3072186064) [LM_ERROR] -  10:41:35.762960 - NodeApplication_Impl::~NodeApplication_Impl - Caught CORBA exception while removing server 0: system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'
OMG minor code (2), described as 'No usable profile in IOR.', completed = NO

It looks like process 4353 has failed.
RM: received ranklist constraints:
|	oid Beta:
|		node-3
|		node-4
|		node-5
|		node-6
|		node-2
|	oid Alpha:
|		node-2
|		node-3
|		node-4
|		node-5
|		node-6
|	oid Epsilon:
|		node-6
|		node-2
|		node-3
|		node-4
|		node-5
|	oid Gamma:
|		node-4
|		node-5
|		node-6
|		node-2
|		node-3
|	oid Delta:
|		node-5
|		node-6
|		node-2
|		node-3
|		node-4
FCM::stop_failover_unit (GroupFailoverTest-FiveNodeGroup-3): plan was shutdown sucessfully.
RM: A state synchronization agent (pid=4366) died.
RM: A state synchronization agent (pid=4405) died.
RM: A client agent agent died.
RM: A client agent agent died.
RM: A client agent agent died.
RM: A client agent agent died.
RM: A client agent agent died.
RM: A state synchronization agent (pid=4353) died.
RM: process_proc_failure (4366)
RM: process 4366 on host node-3 failed
RM: 'Epsilon' on 'node-3' failed.
FCM: app_failure ()
FaultCorrelationManager_Impl::app_failure_i (node-3, Epsilon)
FCM::app_failure (Epsilon, node-3): plan for component 'Epsilon-3' not found.
RM: Removed application node-3:4366:Epsilon:1.
RM: Failover host = node-4.
RM: process_proc_failure (4405)
RM: process 4405 on host node-2 failed
RM: 'Delta' on 'node-2' failed.
FCM: app_failure ()
FaultCorrelationManager_Impl::app_failure_i (node-2, Delta)
FCM::app_failure (Delta, node-2): plan for component 'Delta-3' not found.
RM: Removed application node-2:4405:Delta:1.
RM: Failover host = node-3.
RM: process_proc_failure (4308)
RM: process 4308 on host node-6 failed
RM: 'Gamma' on 'node-6' failed.
FCM: app_failure ()
FaultCorrelationManager_Impl::app_failure_i (node-6, Gamma)
FCM::app_failure (Gamma, node-6): plan for component 'Gamma-3' not found.
RM: Removed application node-6:4308:Gamma:1.
RM: Failover host = node-2.
RM: process_proc_failure (4353)
RM: process 4353 on host node-5 failed
RM: 'Beta' on 'node-5' failed.
FCM: app_failure ()
FaultCorrelationManager_Impl::app_failure_i (node-5, Beta)
FCM::app_failure (Beta, node-5): plan for component 'Beta-3' not found.
RM: Removed application node-5:4353:Beta:1.
RM: Failover host = node-6.
It looks like process 4356 has failed.
RM: process_proc_failure (4356)
RM: process 4356 on host node-5 failed
RM: 'Alpha' on 'node-5' failed.
FCM: app_failure ()
FaultCorrelationManager_Impl::app_failure_i (node-5, Alpha)
FCM::app_failure (Alpha, node-5): caused by component 'Alpha-4' in plan 'GroupFailoverTest-FiveNodeGroup-4'.
FCM::stop_failover_unit (GroupFailoverTest-FiveNodeGroup-4)
RM: Removed application node-5:4356:Alpha:1.
RM: Failover host = node-6.
RM: process_proc_failure () : removed state sync agent from list.
It looks like process 4444 has failed.
It looks like process 4369 has failed.
RM: A state synchronization agent (pid=4444) died.
It looks like process 4408 has failed.
(4344|3072579280) [LM_ERROR] -  10:41:45.846378 - NodeApplication_Impl::~NodeApplication_Impl - Caught CORBA exception while removing container 0 on server 0: system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'
OMG minor code (2), described as 'No usable profile in IOR.', completed = NO

FCM: instance[Alpha-4] removed
FCM: node[node-5] -> oid[Alpha] removed.
FCM: instance[Beta-4] removed
FCM: node[node-6] -> oid[Beta] removed.
FCM: instance[Gamma-4] removed
FCM: node[node-2] -> oid[Gamma] removed.
FCM: instance[Delta-4] removed
FCM: node[node-3] -> oid[Delta] removed.
FCM: instance[Epsilon-4] removed
FCM: node[node-4] -> oid[Epsilon] removed.
It looks like process 4311 has failed.
(4344|3072579280) [LM_ERROR] -  10:41:45.847627 - NodeApplication_Impl::~NodeApplication_Impl - Caught CORBA exception while removing server 0: system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'
OMG minor code (2), described as 'No usable profile in IOR.', completed = NO

RM: A state synchronization agent (pid=4311) died.
RM: A state synchronization agent (pid=4369) died.
RM: A state synchronization agent (pid=4408) died.
RM: received ranklist constraints:
|	oid Beta:
|		node-3
|		node-4
|		node-5
|		node-6
|		node-2
|	oid Alpha:
|		node-2
|		node-3
|		node-4
|		node-5
|		node-6
|	oid Epsilon:
|		node-6
|		node-2
|		node-3
|		node-4
|		node-5
|	oid Gamma:
|		node-4
|		node-5
|		node-6
|		node-2
|		node-3
|	oid Delta:
|		node-5
|		node-6
|		node-2
|		node-3
|		node-4
FCM::stop_failover_unit (GroupFailoverTest-FiveNodeGroup-4): plan was shutdown sucessfully.
RM: A client agent agent died.
RM: A client agent agent died.
RM: A client agent agent died.
RM: A client agent agent died.
RM: A client agent agent died.
RM: process_proc_failure (4444)
RM: process 4444 on host node-4 failed
RM: 'Epsilon' on 'node-4' failed.
FCM: app_failure ()
FaultCorrelationManager_Impl::app_failure_i (node-4, Epsilon)
FCM::app_failure (Epsilon, node-4): plan for component 'Epsilon-4' not found.
RM: Removed application node-4:4444:Epsilon:1.
RM: Failover host = node-5.
RM: process_proc_failure (4369)
RM: process 4369 on host node-3 failed
RM: 'Delta' on 'node-3' failed.
FCM: app_failure ()
FaultCorrelationManager_Impl::app_failure_i (node-3, Delta)
FCM::app_failure (Delta, node-3): plan for component 'Delta-4' not found.
RM: Removed application node-3:4369:Delta:1.
RM: Failover host = node-4.
RM: process_proc_failure (4408)
RM: process 4408 on host node-2 failed
RM: 'Gamma' on 'node-2' failed.
FCM: app_failure ()
FaultCorrelationManager_Impl::app_failure_i (node-2, Gamma)
FCM::app_failure (Gamma, node-2): plan for component 'Gamma-4' not found.
RM: Removed application node-2:4408:Gamma:1.
RM: Failover host = node-3.
RM: process_proc_failure (4311)
RM: process 4311 on host node-6 failed
RM: 'Beta' on 'node-6' failed.
FCM: app_failure ()
FaultCorrelationManager_Impl::app_failure_i (node-6, Beta)
FCM::app_failure (Beta, node-6): plan for component 'Beta-4' not found.
RM: Removed application node-6:4311:Beta:1.
RM: Failover host = node-2.
It looks like process 4316 has failed.
RM: process_proc_failure (4316)
RM: process 4316 on host node-6 failed
RM: 'Alpha' on 'node-6' failed.
FCM: app_failure ()
FaultCorrelationManager_Impl::app_failure_i (node-6, Alpha)
FCM::app_failure (Alpha, node-6): caused by component 'Alpha-5' in plan 'GroupFailoverTest-FiveNodeGroup-5'.
FCM::stop_failover_unit (GroupFailoverTest-FiveNodeGroup-5)
RM: Removed application node-6:4316:Alpha:1.
RM: Can't find failover host for tag=Alpha.
RM: process_proc_failure () : removed state sync agent from list.
It looks like process 4361 has failed.
It looks like process 4449 has failed.
RM: A state synchronization agent (pid=4449) died.
RM: A state synchronization agent (pid=4361) died.
It looks like process 4373 has failed.
(4299|3072141008) [LM_ERROR] -  10:41:55.912199 - NodeApplication_Impl::~NodeApplication_Impl - Caught CORBA exception while removing container 0 on server 0: system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'
OMG minor code (2), described as 'No usable profile in IOR.', completed = NO

FCM: instance[Alpha-5] removed
FCM: node[node-6] -> oid[Alpha] removed.
FCM: instance[Beta-5] removed
FCM: node[node-2] -> oid[Beta] removed.
FCM: instance[Gamma-5] removed
FCM: node[node-3] -> oid[Gamma] removed.
FCM: instance[Delta-5] removed
FCM: node[node-4] -> oid[Delta] removed.
FCM: instance[Epsilon-5] removed
FCM: node[node-5] -> oid[Epsilon] removed.
(4299|3072141008) [LM_ERROR] -  10:41:55.913421 - NodeApplication_Impl::~NodeApplication_Impl - Caught CORBA exception while removing server 0: system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'
OMG minor code (2), described as 'No usable profile in IOR.', completed = NO

It looks like process 4413 has failed.
RM: received ranklist constraints:
|	oid Beta:
|		node-3
|		node-4
|		node-5
|		node-6
|		node-2
|	oid Alpha:
|		node-2
|		node-3
|		node-4
|		node-5
|		node-6
|	oid Epsilon:
|		node-6
|		node-2
|		node-3
|		node-4RM: A state synchronization agent (pid=4373) died.

|		node-5
|	oid Gamma:
|		node-4
|		node-5
|		node-6
|		node-2
|		node-3
|	oid Delta:
|		node-5
|		node-6
|		node-2
|		node-3
|		node-4
FCM::stop_failover_unit (GroupFailoverTest-FiveNodeGroup-5): plan was shutdown sucessfully.
RM: A state synchronization agent (pid=4413) died.
RM: A client agent agent died.
RM: A client agent agent died.
RM: A client agent agent died.
RM: A client agent agent died.
RM: A client agent agent died.
RM: process_proc_failure (4361)
RM: process 4361 on host node-5 failed
RM: 'Epsilon' on 'node-5' failed.
FCM: app_failure ()
FaultCorrelationManager_Impl::app_failure_i (node-5, Epsilon)
FCM::app_failure (Epsilon, node-5): plan for component 'Epsilon-5' not found.
RM: Removed application node-5:4361:Epsilon:1.
RM: Can't find failover host for tag=Epsilon.
RM: process_proc_failure (4449)
RM: process 4449 on host node-4 failed
RM: 'Delta' on 'node-4' failed.
FCM: app_failure ()
FaultCorrelationManager_Impl::app_failure_i (node-4, Delta)
FCM::app_failure (Delta, node-4): plan for component 'Delta-5' not found.
RM: Removed application node-4:4449:Delta:1.
RM: Can't find failover host for tag=Delta.
RM: process_proc_failure (4413)
RM: process 4413 on host node-2 failed
RM: 'Beta' on 'node-2' failed.
FCM: app_failure ()
FaultCorrelationManager_Impl::app_failure_i (node-2, Beta)
FCM::app_failure (Beta, node-2): plan for component 'Beta-5' not found.
RM: Removed application node-2:4413:Beta:1.
RM: Can't find failover host for tag=Beta.
RM: process_proc_failure (4373)
RM: process 4373 on host node-3 failed
RM: 'Gamma' on 'node-3' failed.
FCM: app_failure ()
FaultCorrelationManager_Impl::app_failure_i (node-3, Gamma)
FCM::app_failure (Gamma, node-3): plan for component 'Gamma-5' not found.
RM: Removed application node-3:4373:Gamma:1.
RM: Can't find failover host for tag=Gamma.
d
stopped FaultCorrelationManager.
stop NodeManager on node  ...
stop NodeManager on node  ...
stop NodeManager on node  ...
stop NodeManager on node  ...
stop NodeManager on node  ...
stopped ExecutionManager.
stopped HostMonitor.
stop HostMonitor on node node-2 ...
stop HostMonitor on node node-3 ...
stop HostMonitor on node node-4 ...
stop HostMonitor on node node-5 ...
stop HostMonitor on node node-6 ...
stopped ReplicationManager.
stopped Name_Service.
bash-3.1$ 