Jul 23 16:12:34 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (pcmk_cpg_membership) info: Group attrd event 3: node 2 pid 3296858 joined via cluster join Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (pcmk_cpg_membership) info: Group attrd event 3: an-a02n01 (node 1 pid 121631) is member Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-controld [121633] (pcmk_cpg_membership) info: Group crmd event 3: node 2 pid 3296860 joined via cluster join Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-controld [121633] (pcmk_cpg_membership) info: Group crmd event 3: an-a02n01 (node 1 pid 121633) is member Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-controld [121633] (pcmk_cpg_membership) info: Group crmd event 3: an-a02n02 (node 2 pid 3296860) is member Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-controld [121633] (crm_update_peer_proc) info: pcmk_cpg_membership: Node an-a02n02[2] - corosync-cpg is now online Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-controld [121633] (peer_update_callback) info: Node an-a02n02 is now a peer | DC=true old=0x0000000 new=0x4000000 Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-controld [121633] (te_trigger_stonith_history_sync) info: Fence history will be synchronized cluster-wide within 5 seconds Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (pcmk_cpg_membership) info: Group stonith-ng event 3: node 2 pid 3296856 joined via cluster join Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (pcmk_cpg_membership) info: Group stonith-ng event 3: an-a02n01 (node 1 pid 121629) is member Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-based [121628] (pcmk_cpg_membership) info: Group cib event 3: node 2 pid 3296855 joined via cluster join Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-based [121628] (pcmk_cpg_membership) info: Group cib event 3: an-a02n01 (node 1 pid 121628) is member Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (crm_get_peer) info: Created entry c3ff630f-ae7a-49a5-af89-7aa8d4270273/0x559792685c00 for node an-a02n02/2 (2 total) Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (crm_get_peer) info: Node 2 is now known as an-a02n02 Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (crm_get_peer) info: Created entry 301efa9f-ab21-4758-b392-9cb2155af993/0x564e3948de00 for node an-a02n02/2 (2 total) Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (crm_get_peer) info: Node 2 has uuid 2 Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (pcmk_cpg_membership) info: Group attrd event 3: an-a02n02 (node 2 pid 3296858) is member Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (crm_get_peer) info: Node 2 is now known as an-a02n02 Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (crm_update_peer_proc) info: pcmk_cpg_membership: Node an-a02n02[2] - corosync-cpg is now online Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (update_peer_state_iter) notice: Node an-a02n02 state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (crm_cs_flush) info: Sent 0 CPG messages (1 remaining): Try again (6) Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (crm_get_peer) info: Node 2 has uuid 2 Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (pcmk_cpg_membership) info: Group stonith-ng event 3: an-a02n02 (node 2 pid 3296856) is member Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (crm_update_peer_proc) info: pcmk_cpg_membership: Node an-a02n02[2] - corosync-cpg is now online Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_peer_state_iter) notice: Node an-a02n02 state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (crm_cs_flush) info: Sent 0 CPG messages (1 remaining): Try again (6) Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-controld [121633] (quorum_notification_cb) info: Quorum retained | membership=259 members=2 Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-controld [121633] (update_peer_state_iter) notice: Node an-a02n02 state is now member | nodeid=2 previous=lost source=quorum_notification_cb Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-controld [121633] (peer_update_callback) info: Cluster node an-a02n02 is now member (was lost) Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_INTEGRATION | input=I_NODE_JOIN cause=C_FSA_INTERNAL origin=peer_update_callback Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_one) info: Making join-3 offers to any unconfirmed nodes because an unknown node joined Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Making join-3 offers based on membership event 259 Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-3 offer to an-a02n02 Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Not making join-3 offer to already known node an-a02n01 (confirmed) Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 100 aborted: Peer Halt | source=do_te_invoke:136 complete=true Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-based [121628] (crm_get_peer) info: Created entry b90d6436-ebda-4818-9e86-f7a67e348035/0x5619760a23b0 for node an-a02n02/2 (2 total) Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-based [121628] (crm_get_peer) info: Node 2 is now known as an-a02n02 Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-based [121628] (crm_get_peer) info: Node 2 has uuid 2 Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-based [121628] (pcmk_cpg_membership) info: Group cib event 3: an-a02n02 (node 2 pid 3296855) is member Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-based [121628] (crm_update_peer_proc) info: pcmk_cpg_membership: Node an-a02n02[2] - corosync-cpg is now online Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-based [121628] (update_peer_state_iter) notice: Node an-a02n02 state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/305) Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/306) Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/309) Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/310) Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.28 2 Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.29 (null) Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=29 Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crmd=online, @crm-debug-origin=peer_update_callback Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/305, version=0.337.29) Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/306, version=0.337.29) Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/309, version=0.337.29) Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.29 2 Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.30 (null) Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=30 Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=post_cache_update Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @in_ccm=true, @crm-debug-origin=post_cache_update Jul 23 16:13:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/310, version=0.337.30) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-controld [121633] (pcmk__update_peer_expected) info: do_dc_join_filter_offer: Node an-a02n02[2] - expected state is now member (was down) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_finalize) notice: Finalizing join-3 for 1 node (sync'ing CIB from an-a02n02) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_finalize) notice: Requested CIB version Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_sync operation for section 'all' to an-a02n02 (origin=local/crmd/311) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n02: faa77275e1e8c408f4838d2d2dcdc91c Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replacement 0.337.2 from an-a02n02 not applied to 0.337.30: current num_updates is greater than the replacement Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-controld [121633] (finalize_sync_callback) warning: Could not sync CIB from an-a02n02 in join-3: Update was older than existing configuration Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-controld [121633] (register_fsa_error_adv) info: Resetting the current action list Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) warning: Input I_ELECTION_DC received in state S_FINALIZE_JOIN from finalize_sync_callback Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=finalize_sync_callback Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-4 offer to an-a02n02 Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-4 offer to an-a02n01 Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-4 requests from 2 outstanding nodes Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_check_for_new_writer) notice: Detected another attribute writer (an-a02n02), starting new election Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (process_peer_sync_response) info: Processing sync-response from an-a02n02 Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting #attrd-protocol[an-a02n02]: (unset) -> 3 | from an-a02n02 Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (crm_cs_flush) info: Sent 3 CPG messages (0 remaining): OK (1) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (crm_cs_flush) info: Sent 2 CPG messages (0 remaining): OK (1) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (election_count_vote) info: election-attrd round 2 (owner node ID 2) pass: vote from an-a02n02 (Uptime) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (process_peer_sync_response) info: Processing sync-response from an-a02n02 Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (election_check) info: election-attrd won by local node Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_declare_winner) notice: Recorded local node as attribute writer (was unset) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 14 with 1 change for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 15 with 1 change for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 16 with 1 change for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 17 with 1 change for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 18 with 1 change for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 19 with 1 change for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 20 with 1 change for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 21 with 1 change for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: 5c0ae7a81d266484cd53effd22475581 Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.337.30 with 0.337.30 from an-a02n01 Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: cib_perform_op: Local-only Change: 0.337.30 Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: +~ /cib/status/node_state[@id='1']/lrm[@id='1'] moved to offset 1 Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=post_cache_update Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Local-only Replace: 0.337.30 from an-a02n01 Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/313, version=0.337.30) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/314) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/315) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/14) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/15) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/16) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (xml_patch_version_check) notice: Versions did not change in patch 0.337.30 Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/17) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/18) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node2_an-pdu03#start_0' delayed: update 14 in progress Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node2_an-pdu03#start_0' delayed: update 15 in progress Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node1_an-pdu03#start_0' delayed: update 16 in progress Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node1_an-pdu03#start_0' delayed: update 17 in progress Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node2_an-pdu04#start_0' delayed: update 18 in progress Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node2_an-pdu04#start_0' delayed: update 19 in progress Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node1_an-pdu04#start_0' delayed: update 20 in progress Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/19) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node1_an-pdu04#start_0' delayed: update 21 in progress Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/20) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/21) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/314, version=0.337.30) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/315, version=0.337.30) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4128871] exited with status 0 Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/14, version=0.337.30) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 14 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=1627003019 Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/15, version=0.337.30) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 15 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/16, version=0.337.30) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 16 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/17, version=0.337.30) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 17 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=1627003020 Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/18, version=0.337.30) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 18 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/19, version=0.337.30) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 19 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=1627003034 Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/20, version=0.337.30) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 20 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/21, version=0.337.30) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 21 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=1627003035 Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-24.raw Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.337.0 of the CIB to disk (digest: 1636edea2c3c05c32ce58a90fcd16dcd) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.OJOoxc (digest: /var/lib/pacemaker/cib/cib.uryeTa) Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4128872] exited with status 0 Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4128874] exited with status 0 Jul 23 16:13:12 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4128911] exited with status 0 Jul 23 16:13:13 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4128917] exited with status 0 Jul 23 16:13:13 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4128935] exited with status 0 Jul 23 16:13:13 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4128937] exited with status 0 Jul 23 16:13:13 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4128949] exited with status 0 Jul 23 16:13:13 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4129089] exited with status 0 Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4129227] exited with status 0 Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/318) Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_JOIN_RESULT received in state S_ELECTION from route_message Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/319) Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_JOIN_RESULT received in state S_ELECTION from route_message Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/318, version=0.337.30) Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.30 2 Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.31 (null) Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=31 Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced, @join=member, @expected=member Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/319, version=0.337.31) Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/320, version=0.337.31) Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/321) Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/323) Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/325) Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/327) Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-5 offer to an-a02n02 Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-5 offer to an-a02n01 Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-5 requests from 2 outstanding nodes Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/321, version=0.337.31) Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/323, version=0.337.31) Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/325, version=0.337.31) Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/327, version=0.337.31) Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: 37a1008549014c5b03f3859ebd7e67d9 Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.337.31 with 0.337.31 from an-a02n01 Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/330, version=0.337.31) Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/331) Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/332) Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-25.raw Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/331, version=0.337.31) Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.337.0 of the CIB to disk (digest: 513fd971e2a46a594a89277e208e4395) Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.7UP1Ni (digest: /var/lib/pacemaker/cib/cib.hAzBMk) Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4129388] exited with status 0 Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/332, version=0.337.31) Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4129392] exited with status 0 Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4129395] exited with status 0 Jul 23 16:13:14 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4129406] exited with status 0 Jul 23 16:13:15 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4129412] exited with status 0 Jul 23 16:13:15 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4129422] exited with status 0 Jul 23 16:13:15 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4129425] exited with status 0 Jul 23 16:13:15 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4129431] exited with status 0 Jul 23 16:13:15 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4129433] exited with status 0 Jul 23 16:13:16 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4129434] exited with status 0 Jul 23 16:13:16 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 333) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 16:13:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/333) Jul 23 16:13:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/334) Jul 23 16:13:16 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 335) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 16:13:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.31 2 Jul 23 16:13:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.32 (null) Jul 23 16:13:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 16:13:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=32 Jul 23 16:13:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/333, version=0.337.32) Jul 23 16:13:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.32 2 Jul 23 16:13:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.33 (null) Jul 23 16:13:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=33 Jul 23 16:13:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:13:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 16:13:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/334, version=0.337.33) Jul 23 16:13:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/335) Jul 23 16:13:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.33 2 Jul 23 16:13:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.34 (null) Jul 23 16:13:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 16:13:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=34 Jul 23 16:13:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/335, version=0.337.34) Jul 23 16:13:16 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4129435] exited with status 0 Jul 23 16:13:16 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4129436] exited with status 0 Jul 23 16:13:16 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4129437] exited with status 0 Jul 23 16:13:16 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4129450] exited with status 0 Jul 23 16:13:17 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4129454] exited with status 0 Jul 23 16:13:17 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4129457] exited with status 0 Jul 23 16:13:17 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4129459] exited with status 0 Jul 23 16:13:17 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4129466] exited with status 0 Jul 23 16:13:17 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4129468] exited with status 0 Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4129476] exited with status 0 Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/336) Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 100 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/339) Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/340) Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/341) Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.34 2 Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.35 (null) Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=35 Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/336, version=0.337.35) Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/339, version=0.337.35) Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.35 2 Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.36 (null) Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=36 Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/340, version=0.337.36) Jul 23 16:13:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/341, version=0.337.36) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 100 aborted: History refresh | source=process_resource_updates:268 complete=true Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 22 18:17:00 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 22 18:16:47 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 22 18:16:47 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 22 18:17:00 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n02 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu03 on an-a02n02 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu04 on an-a02n02 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu04 ( an-a02n02 ) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node2_an-pdu03 ( an-a02n02 ) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node2_an-pdu04 ( an-a02n02 ) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n01) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 101, saving inputs in /var/lib/pacemaker/pengine/pe-input-122.bz2 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 101 (ref=pe_calc-dc-1627081999-234) derived from /var/lib/pacemaker/pengine/pe-input-122.bz2 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu03_monitor_0 on an-a02n02 | action 11 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu04_monitor_0 on an-a02n02 | action 12 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node2_an-pdu03_monitor_0 on an-a02n02 | action 13 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node2_an-pdu04_monitor_0 on an-a02n02 | action 14 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation ipmilan_node2_monitor_0 on an-a02n02 | action 15 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation delay_node2_monitor_0 on an-a02n02 | action 16 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation ipmilan_node1_monitor_0 on an-a02n02 | action 17 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation delay_node1_monitor_0 on an-a02n02 | action 18 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation srv13-psql_monitor_0 on an-a02n02 | action 19 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation srv14-file-processor_monitor_0 on an-a02n02 | action 20 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation srv15-tickets_monitor_0 on an-a02n02 | action 21 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation srv16-machine-interface_monitor_0 on an-a02n02 | action 22 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation srv17-order-processing_monitor_0 on an-a02n02 | action 23 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation srv18-inventory_monitor_0 on an-a02n02 | action 24 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation srv19-uls_monitor_0 on an-a02n02 | action 25 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation srv20-fileprocessor_monitor_0 on an-a02n02 | action 26 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation srv21-tickets_monitor_0 on an-a02n02 | action 27 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation srv22-machineinterface_monitor_0 on an-a02n02 | action 28 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation srv23-orderprocessing_monitor_0 on an-a02n02 | action 29 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation srv24-inventory_monitor_0 on an-a02n02 | action 30 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.36 2 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.37 (null) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=37 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/11, version=0.337.37) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.37 2 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.38 (null) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=38 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/12, version=0.337.38) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.38 2 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.39 (null) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=39 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:7;11:101:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=5, @rc-code=7, @op-status=0, @exec-time=1 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/13, version=0.337.39) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 11 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=5 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 on an-a02n02 | action 31 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.39 2 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.40 (null) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=40 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:7;12:101:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=9, @rc-code=7, @op-status=0 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/14, version=0.337.40) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 12 (apc_snmp_node1_an-pdu04_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=9 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu04_start_0 on an-a02n02 | action 33 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.40 2 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.41 (null) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=41 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/15, version=0.337.41) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.41 2 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.42 (null) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=42 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:7;13:101:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=13, @rc-code=7, @op-status=0 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/16, version=0.337.42) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 13 (apc_snmp_node2_an-pdu03_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=13 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node2_an-pdu03_start_0 on an-a02n02 | action 35 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.42 2 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.43 (null) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=43 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/17, version=0.337.43) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.43 2 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.44 (null) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=44 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:7;14:101:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=17, @rc-code=7, @op-status=0 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/18, version=0.337.44) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 14 (apc_snmp_node2_an-pdu04_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=17 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node2_an-pdu04_start_0 on an-a02n02 | action 37 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.44 2 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.45 (null) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=45 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/19, version=0.337.45) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.45 2 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.46 (null) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=46 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ipmilan_node2']/lrm_rsc_op[@id='ipmilan_node2_last_0']: @transition-magic=0:7;15:101:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=21, @rc-code=7, @op-status=0 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/20, version=0.337.46) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 15 (ipmilan_node2_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=21 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.46 2 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.47 (null) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=47 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/23, version=0.337.49) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.49 2 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.50 (null) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=50 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ipmilan_node1']/lrm_rsc_op[@id='ipmilan_node1_last_0']: @transition-magic=0:7;17:101:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=29, @rc-code=7, @op-status=0 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/24, version=0.337.50) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 17 (ipmilan_node1_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=29 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.50 2 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.51 (null) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=51 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/28, version=0.337.54) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.54 2 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.55 (null) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=55 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/29, version=0.337.55) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.55 2 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.56 (null) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=56 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/30, version=0.337.56) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.56 2 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.57 (null) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=57 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/31, version=0.337.57) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.57 2 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.58 (null) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=58 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/32, version=0.337.58) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.58 2 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.59 (null) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=59 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/33, version=0.337.59) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.59 2 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.60 (null) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=60 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/34, version=0.337.60) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.60 2 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.61 (null) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=61 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/35, version=0.337.61) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.61 2 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.62 (null) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=62 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/36, version=0.337.62) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.62 2 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.63 (null) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=63 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/37, version=0.337.63) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.63 2 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.64 (null) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=64 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/38, version=0.337.64) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.64 2 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.65 (null) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=65 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @transition-key=31:101:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;31:101:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @exec-time=0 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/39, version=0.337.65) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.65 2 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.66 (null) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=66 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_start_0, @operation=start, @transition-key=33:101:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;33:101:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/40, version=0.337.66) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.66 2 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.67 (null) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=67 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @operation_key=apc_snmp_node2_an-pdu03_start_0, @operation=start, @transition-key=35:101:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;35:101:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/41, version=0.337.67) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.67 2 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.68 (null) Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=68 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @operation_key=apc_snmp_node2_an-pdu04_start_0, @operation=start, @transition-key=37:101:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;37:101:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1 Jul 23 16:13:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/42, version=0.337.68) Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.68 2 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.69 (null) Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=69 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='srv18-inventory']/lrm_rsc_op[@id='srv18-inventory_last_0']: @transition-magic=0:7;24:101:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=57, @rc-code=7, @op-status=0, @exec-time=377 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/43, version=0.337.69) Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 24 (srv18-inventory_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=57 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.69 2 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.70 (null) Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=70 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='srv16-machine-interface']/lrm_rsc_op[@id='srv16-machine-interface_last_0']: @transition-magic=0:7;22:101:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=49, @rc-code=7, @op-status=0, @exec-time=380 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/44, version=0.337.70) Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 22 (srv16-machine-interface_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=49 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.70 2 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.71 (null) Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=71 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='srv17-order-processing']/lrm_rsc_op[@id='srv17-order-processing_last_0']: @transition-magic=0:7;23:101:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=53, @rc-code=7, @op-status=0, @exec-time=381 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/45, version=0.337.71) Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 23 (srv17-order-processing_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=53 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.71 2 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.72 (null) Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=72 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='srv24-inventory']/lrm_rsc_op[@id='srv24-inventory_last_0']: @transition-magic=0:7;30:101:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=81, @rc-code=7, @op-status=0, @exec-time=379 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/46, version=0.337.72) Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 30 (srv24-inventory_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=81 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.72 2 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.73 (null) Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=73 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='srv22-machineinterface']/lrm_rsc_op[@id='srv22-machineinterface_last_0']: @transition-magic=0:7;28:101:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=73, @rc-code=7, @op-status=0, @exec-time=389 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/47, version=0.337.73) Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 28 (srv22-machineinterface_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=73 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.73 2 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.74 (null) Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=74 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='srv20-fileprocessor']/lrm_rsc_op[@id='srv20-fileprocessor_last_0']: @transition-magic=0:7;26:101:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=65, @rc-code=7, @op-status=0, @exec-time=391 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/48, version=0.337.74) Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 26 (srv20-fileprocessor_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=65 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.74 2 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.75 (null) Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=75 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='srv19-uls']/lrm_rsc_op[@id='srv19-uls_last_0']: @transition-magic=0:7;25:101:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=61, @rc-code=7, @op-status=0, @exec-time=393 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/49, version=0.337.75) Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 25 (srv19-uls_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=61 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.75 2 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.76 (null) Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=76 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='srv14-file-processor']/lrm_rsc_op[@id='srv14-file-processor_last_0']: @transition-magic=0:7;20:101:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=41, @rc-code=7, @op-status=0, @exec-time=398 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/50, version=0.337.76) Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 20 (srv14-file-processor_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=41 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.76 2 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.77 (null) Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=77 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='srv21-tickets']/lrm_rsc_op[@id='srv21-tickets_last_0']: @transition-magic=0:7;27:101:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=69, @rc-code=7, @op-status=0, @exec-time=396 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/51, version=0.337.77) Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 27 (srv21-tickets_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=69 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.77 2 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.78 (null) Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=78 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='srv13-psql']/lrm_rsc_op[@id='srv13-psql_last_0']: @transition-magic=0:7;19:101:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=37, @rc-code=7, @op-status=0, @exec-time=402 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/52, version=0.337.78) Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 19 (srv13-psql_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=37 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.78 2 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.79 (null) Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=79 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='srv15-tickets']/lrm_rsc_op[@id='srv15-tickets_last_0']: @transition-magic=0:7;21:101:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=45, @rc-code=7, @op-status=0, @exec-time=401 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/53, version=0.337.79) Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 21 (srv15-tickets_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=45 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.79 2 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.80 (null) Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=80 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='srv23-orderprocessing']/lrm_rsc_op[@id='srv23-orderprocessing_last_0']: @transition-magic=0:7;29:101:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=77, @rc-code=7, @op-status=0, @exec-time=396 Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/54, version=0.337.80) Jul 23 16:13:20 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 29 (srv23-orderprocessing_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=77 Jul 23 16:13:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: e4cfea920e17146a260e162cc41143a4 for 0.337.80 (0x5619761712c0 0) Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.80 2 Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.81 (null) Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=81 Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:1;33:101:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=83, @rc-code=1, @op-status=0, @exec-time=12332 Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/23, version=0.337.82) Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 23 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 101 aborted by transient_attributes.2 'create': Transient attribute change | cib=0.337.82 source=abort_unless_down:313 path=/cib/status/node_state[@id='2'] complete=false Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/24) Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.82 2 Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.83 (null) Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=83 Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/24, version=0.337.83) Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 24 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=1627003035 Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627082012 Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 101 aborted by status-2-last-failure-apc_snmp_node1_an-pdu04.start_0 doing create last-failure-apc_snmp_node1_an-pdu04#start_0=1627082012: Transient attribute change | cib=0.337.83 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=false Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.83 2 Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.84 (null) Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=84 Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:1;31:101:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=82, @rc-code=1, @op-status=0, @exec-time=12345 Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/25, version=0.337.85) Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 25 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 101 aborted by status-2-fail-count-apc_snmp_node1_an-pdu03.start_0 doing create fail-count-apc_snmp_node1_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.337.85 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=false Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.85 2 Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.86 (null) Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=86 Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/26, version=0.337.86) Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 101 aborted by status-2-last-failure-apc_snmp_node1_an-pdu03.start_0 doing create last-failure-apc_snmp_node1_an-pdu03#start_0=1627082012: Transient attribute change | cib=0.337.86 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=false Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 26 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=1627003020 Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627082012 Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.86 2 Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.87 (null) Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=87 Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:1;37:101:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=85, @rc-code=1, @op-status=0, @exec-time=12344 Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/27, version=0.337.88) Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 27 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 101 aborted by status-2-fail-count-apc_snmp_node2_an-pdu04.start_0 doing create fail-count-apc_snmp_node2_an-pdu04#start_0=INFINITY: Transient attribute change | cib=0.337.88 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=false Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.88 2 Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.89 (null) Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=89 Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/28, version=0.337.89) Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 28 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=1627003034 Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 101 aborted by status-2-last-failure-apc_snmp_node2_an-pdu04.start_0 doing create last-failure-apc_snmp_node2_an-pdu04#start_0=1627082012: Transient attribute change | cib=0.337.89 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=false Jul 23 16:13:32 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=1627082012 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.89 2 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.90 (null) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=90 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:1;35:101:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=84, @rc-code=1, @op-status=0, @exec-time=13351 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/29, version=0.337.91) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 29 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.91 2 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.92 (null) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=92 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/30, version=0.337.92) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 30 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=1627003019 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=1627082013 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 101 aborted by status-2-fail-count-apc_snmp_node2_an-pdu03.start_0 doing create fail-count-apc_snmp_node2_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.337.91 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 101 aborted by status-2-last-failure-apc_snmp_node2_an-pdu03.start_0 doing create last-failure-apc_snmp_node2_an-pdu03#start_0=1627082013: Transient attribute change | cib=0.337.92 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 22 18:17:00 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 22 18:16:47 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 22 18:16:47 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 22 18:17:00 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu03 on an-a02n02 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for ipmilan_node1 on an-a02n02 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu03 ( an-a02n02 ) due to node availability Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu04 ( an-a02n02 ) due to node availability Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node2_an-pdu03 ( an-a02n02 ) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node2_an-pdu04 ( an-a02n02 ) due to node availability Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n01) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Move ipmilan_node1 ( an-a02n01 -> an-a02n02 ) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 102, saving inputs in /var/lib/pacemaker/pengine/pe-input-123.bz2 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627082013-259 is obsolete Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 22 18:17:00 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 22 18:16:47 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 22 18:16:47 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 22 18:17:00 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for delay_node2 on an-a02n02 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for delay_node1 on an-a02n02 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu03 ( an-a02n02 ) due to node availability Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu04 ( an-a02n02 ) due to node availability Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node2_an-pdu03 ( an-a02n02 ) due to node availability Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node2_an-pdu04 ( an-a02n02 ) due to node availability Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Move delay_node2 ( an-a02n01 -> an-a02n02 ) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Move delay_node1 ( an-a02n01 -> an-a02n02 ) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 103, saving inputs in /var/lib/pacemaker/pengine/pe-input-124.bz2 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 103 (ref=pe_calc-dc-1627082013-260) derived from /var/lib/pacemaker/pengine/pe-input-124.bz2 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 on an-a02n02 | action 11 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu04_stop_0 on an-a02n02 | action 12 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node2_an-pdu03_stop_0 on an-a02n02 | action 13 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node2_an-pdu04_stop_0 on an-a02n02 | action 14 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation delay_node2_stop_0 locally on an-a02n01 | action 17 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for delay_node2 on an-a02n01 | transition_key=17:103:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=delay_node2_stop_0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:delay_node2 action:stop call_id:119 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation delay_node1_stop_0 locally on an-a02n01 | action 22 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/347) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: delay_node2 stop (call 119) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for delay_node1 on an-a02n01 | transition_key=22:103:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=delay_node1_stop_0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:delay_node1 action:stop call_id:121 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) info: Result of monitor operation for delay_node2 on an-a02n01: Cancelled | call=117 key=delay_node2_monitor_60000 confirmed=true Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.92 2 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.93 (null) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=93 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='delay_node2']/lrm_rsc_op[@id='delay_node2_last_0']: @operation_key=delay_node2_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=17:103:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;17:103:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change= Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: delay_node1 stop (call 121) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for delay_node2 on an-a02n01: ok | rc=0 call=119 key=delay_node2_stop_0 confirmed=true cib-update=349 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) info: Result of monitor operation for delay_node1 on an-a02n01: Cancelled | call=116 key=delay_node1_monitor_60000 confirmed=true Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/347, version=0.337.93) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for delay_node1 on an-a02n01: ok | rc=0 call=121 key=delay_node1_stop_0 confirmed=true cib-update=350 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/348) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/349) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/350) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.93 2 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.94 (null) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=94 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @transition-key=11:103:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;11:103:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/59, version=0.337.94) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.94 2 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.95 (null) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=95 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='delay_node1']/lrm_rsc_op[@id='delay_node1_last_0']: @operation_key=delay_node1_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=22:103:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;22:103:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change= Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/348, version=0.337.95) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.95 2 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.96 (null) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=96 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='delay_node2']/lrm_rsc_op[@id='delay_node2_last_0']: @transition-magic=0:0;17:103:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=119, @rc-code=0, @op-status=0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/349, version=0.337.96) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 103 action 17 (delay_node2_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=119 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation delay_node2_start_0 on an-a02n02 | action 18 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.96 2 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.97 (null) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=97 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='delay_node1']/lrm_rsc_op[@id='delay_node1_last_0']: @transition-magic=0:0;22:103:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=121, @rc-code=0, @op-status=0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/350, version=0.337.97) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 103 action 22 (delay_node1_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=121 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation delay_node1_start_0 on an-a02n02 | action 23 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.97 2 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.98 (null) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=98 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_stop_0, @operation=stop, @transition-key=12:103:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;12:103:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/60, version=0.337.98) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.98 2 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.99 (null) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=99 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;11:103:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=86, @rc-code=0, @op-status=0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/61, version=0.337.99) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 103 action 11 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=86 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.99 2 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.100 (null) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=100 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:0;12:103:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=87, @rc-code=0, @op-status=0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/62, version=0.337.100) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 103 action 12 (apc_snmp_node1_an-pdu04_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=87 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.100 2 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.101 (null) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=101 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @operation_key=apc_snmp_node2_an-pdu03_stop_0, @operation=stop, @transition-key=13:103:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;13:103:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/63, version=0.337.101) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.101 2 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.102 (null) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=102 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @operation_key=apc_snmp_node2_an-pdu04_stop_0, @operation=stop, @transition-key=14:103:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;14:103:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/64, version=0.337.102) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.102 2 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.103 (null) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=103 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:0;13:103:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=88, @rc-code=0, @op-status=0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/65, version=0.337.103) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 103 action 13 (apc_snmp_node2_an-pdu03_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=88 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.103 2 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.104 (null) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=104 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:0;14:103:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=89, @rc-code=0, @op-status=0 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/66, version=0.337.104) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 103 action 14 (apc_snmp_node2_an-pdu04_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=89 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.104 2 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.105 (null) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=105 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='delay_node2']/lrm_rsc_op[@id='delay_node2_last_0']: @operation_key=delay_node2_start_0, @operation=start, @transition-key=18:103:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;18:103:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1627082013 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/67, version=0.337.105) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.105 2 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.106 (null) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=106 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='delay_node1']/lrm_rsc_op[@id='delay_node1_last_0']: @operation_key=delay_node1_start_0, @operation=start, @transition-key=23:103:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;23:103:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1627082013 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/68, version=0.337.106) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.106 2 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.107 (null) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=107 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='delay_node2']/lrm_rsc_op[@id='delay_node2_last_0']: @transition-magic=0:0;18:103:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=90, @rc-code=0, @op-status=0, @exec-time=9 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/69, version=0.337.107) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 103 action 18 (delay_node2_start_0 on an-a02n02) confirmed: ok | rc=0 call-id=90 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation delay_node2_monitor_60000 on an-a02n02 | action 19 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.107 2 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.108 (null) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=108 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='delay_node1']/lrm_rsc_op[@id='delay_node1_last_0']: @transition-magic=0:0;23:103:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=91, @rc-code=0, @op-status=0, @exec-time=10 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/70, version=0.337.108) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 103 action 23 (delay_node1_start_0 on an-a02n02) confirmed: ok | rc=0 call-id=91 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation delay_node1_monitor_60000 on an-a02n02 | action 24 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.108 2 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.337.109 (null) Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=109 Jul 23 16:13:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='delay_node2']: S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:13:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: fbde84a1ab201d656041be5ee9534f46 for 0.337.112 (0x5619761712c0 0) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.337.112 2 Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.338.0 6a24af66df3d9f825cc2681222f8f5d6 Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=338, @num_updates=0 Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03']/instance_attributes[@id='apc_snmp_node1_an-pdu03-instance_attributes']/nvpair[@id='apc_snmp_node1_an-pdu03-instance_attributes-ip']: @value=10.201.2.3 Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.337.112 -> 0.338.0 from an-a02n02 Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.338.0) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 103 aborted by apc_snmp_node1_an-pdu03-instance_attributes-ip doing modify ip=10.201.2.3: Configuration change | cib=0.338.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03']/instance_attributes[@id='apc_snmp_node1_an-pdu03-instance_attributes']/nvpair[@id='apc_snmp_node1_an-pdu03-instance_attributes-ip'] complete=true Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 31 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 32 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 33 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 34 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 35 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 36 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 37 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 38 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/31) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/353) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/354) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: modify nvpair[@id='apc_snmp_node1_an-pdu03-instance_attributes-ip'] Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.338.0 Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/32) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-26.raw Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/33) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/34) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/35) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/36) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/31, version=0.338.0) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 31 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=1627003019 Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=1627082013 Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.338.0 of the CIB to disk (digest: caa9c9d11084bba7f1ad0b3668a8e153) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.0AuvQm (digest: /var/lib/pacemaker/cib/cib.CdApcl) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node1_an-pdu03' to device list (8 active devices) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/353, version=0.338.0) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.338.0 2 Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.338.1 (null) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/354, version=0.338.1) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/32, version=0.338.1) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 32 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/33, version=0.338.1) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 33 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/34, version=0.338.1) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 34 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=1627003020 Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627082012 Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/35, version=0.338.1) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 35 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/36, version=0.338.1) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 36 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=1627003034 Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=1627082012 Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/37) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/38) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/37, version=0.338.1) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 37 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/38, version=0.338.1) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 38 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=1627003035 Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627082012 Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/356, version=0.338.1) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/357) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/359) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/361) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-6 offer to an-a02n02 Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-6 offer to an-a02n01 Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-6 requests from 2 outstanding nodes Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/357, version=0.338.1) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/359, version=0.338.1) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/361, version=0.338.1) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/363) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/363, version=0.338.1) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: a2d0c684b380b002faa4969d747b5985 Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.338.1 with 0.338.1 from an-a02n01 Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/366, version=0.338.1) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/367) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/368) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-27.raw Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/367, version=0.338.1) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.338.0 of the CIB to disk (digest: caa9c9d11084bba7f1ad0b3668a8e153) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.LWA0ou (digest: /var/lib/pacemaker/cib/cib.V5Ncly) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/368, version=0.338.1) Jul 23 16:22:27 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157918] exited with status 0 Jul 23 16:22:28 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157920] exited with status 0 Jul 23 16:22:28 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157921] exited with status 0 Jul 23 16:22:28 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4157922] exited with status 0 Jul 23 16:22:28 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4157924] exited with status 0 Jul 23 16:22:28 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157926] exited with status 0 Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157927] exited with status 0 Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157928] exited with status 0 Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 369) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/369) Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 371) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/370) Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/371) Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.338.1 2 Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.338.2 (null) Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/369, version=0.338.2) Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.338.2 2 Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.338.3 (null) Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/370, version=0.338.3) Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.338.3 2 Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.338.4 (null) Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/371, version=0.338.4) Jul 23 16:22:29 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157929] exited with status 0 Jul 23 16:22:30 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157930] exited with status 0 Jul 23 16:22:30 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157931] exited with status 0 Jul 23 16:22:30 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4157932] exited with status 0 Jul 23 16:22:30 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4157934] exited with status 0 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157936] exited with status 0 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157937] exited with status 0 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157938] exited with status 0 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/372) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 103 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.338.4 2 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.338.5 (null) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/372, version=0.338.5) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 103 aborted: History refresh | source=process_resource_updates:268 complete=true Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/375) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/376) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/377) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/375, version=0.338.5) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.338.5 2 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.338.6 (null) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/376, version=0.338.6) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/377, version=0.338.6) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 22 18:17:00 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n01 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now b6d660e5416057f7575eac6369636a4f (reload:3.10.2) 0:1;41:0:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n01 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 22 18:16:47 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 22 18:16:47 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 22 18:17:00 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n02 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now b6d660e5416057f7575eac6369636a4f (reload:3.10.2) 0:1;31:101:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n01 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for ipmilan_node2 on an-a02n02 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu03 ( an-a02n01 ) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Move ipmilan_node2 ( an-a02n01 -> an-a02n02 ) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n02) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 104, saving inputs in /var/lib/pacemaker/pengine/pe-input-125.bz2 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 104 (ref=pe_calc-dc-1627082551-278) derived from /var/lib/pacemaker/pengine/pe-input-125.bz2 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 locally on an-a02n01 | action 13 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=13:104:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_start_0 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu03 action:start call_id:122 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-controld [121633] (te_crm_command) info: Executing crm-event (12) without waiting: clear_failcount on an-a02n02 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/380) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-controld [121633] (te_crm_command) info: Executing crm-event (1) locally without waiting: clear_failcount on an-a02n01 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation ipmilan_node2_stop_0 locally on an-a02n01 | action 15 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for ipmilan_node2 on an-a02n01 | transition_key=15:104:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=ipmilan_node2_stop_0 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:ipmilan_node2 action:stop call_id:124 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) info: Result of monitor operation for ipmilan_node2 on an-a02n01: Cancelled | call=113 key=ipmilan_node2_monitor_60000 confirmed=true Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.338.6 2 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.338.7 (null) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=13:104:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;13:104:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/380, version=0.338.7) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-controld [121633] (update_attrd_clear_failures) info: Asking pacemaker-attrd to clear failure of all operations for apc_snmp_node1_an-pdu03 on cluster node an-a02n01 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/381) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section /cib/status/node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0'] to all (origin=local/crmd/382) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.338.7 2 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.338.8 (null) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ipmilan_node2']/lrm_rsc_op[@id='ipmilan_node2_last_0']: @operation_key=ipmilan_node2_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=15:104:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;15:104:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-c Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/381, version=0.338.8) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 40 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: 1627003020 -> (unset) | from an-a02n01 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 41 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.338.8 2 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.338.9 (null) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0'] Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: ipmilan_node2 stop (call 124) exited with status 0 (execution time 1ms, queue time 0ms) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for ipmilan_node2 on an-a02n01: ok | rc=0 call=124 key=ipmilan_node2_stop_0 confirmed=true cib-update=383 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n02 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node1_an-pdu03#start_0' delayed: update 40 in progress Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: 1627082012 -> (unset) | from an-a02n02 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node1_an-pdu03#start_0' delayed: update 41 in progress Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section /cib/status/node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0']: OK (rc=0, origin=an-a02n01/crmd/382, version=0.338.9) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 104 aborted by deletion of lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0']: Resource operation removal | cib=0.338.9 source=process_op_deletion:358 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0'] complete=false Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/383) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/40) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/41) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.338.9 2 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.338.10 (null) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ipmilan_node2']/lrm_rsc_op[@id='ipmilan_node2_last_0']: @transition-magic=0:0;15:104:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=124, @rc-code=0, @op-status=0, @exec-time=1 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/383, version=0.338.10) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 104 action 15 (ipmilan_node2_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=124 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.338.10 2 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.338.11 (null) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0'] Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/40, version=0.338.11) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 40 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 104 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.338.11 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0'] complete=false Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 42 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.338.11 2 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.338.12 (null) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0'] Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/41, version=0.338.12) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 41 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627082012 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 104 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.338.12 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0'] complete=false Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 43 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.338.12 2 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.338.13 (null) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0'] Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section /cib/status/node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0']: OK (rc=0, origin=an-a02n02/crmd/77, version=0.338.13) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 104 aborted by deletion of lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0']: Resource operation removal | cib=0.338.13 source=process_op_deletion:358 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0'] complete=false Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/42) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/43) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.338.13 2 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.338.14 (null) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/42, version=0.338.14) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 42 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 104 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.338.14 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] complete=false Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.338.14 2 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.338.15 (null) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=15 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/43, version=0.338.15) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 43 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 104 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.338.15 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] complete=false Jul 23 16:22:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu03 start (call 122) exited with status 0 (execution time 379ms, queue time 0ms) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_apc_snmp_metadata_1[4157943] exited with status 0 Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for apc_snmp_node1_an-pdu03 on an-a02n01: ok | rc=0 call=122 key=apc_snmp_node1_an-pdu03_start_0 confirmed=true cib-update=384 Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/384) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.338.15 2 Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.338.16 (null) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=16 Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;13:104:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=122, @rc-code=0, @op-status=0, @exec-time=379 Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/384, version=0.338.16) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 104 action 13 (apc_snmp_node1_an-pdu03_start_0 on an-a02n01) confirmed: ok | rc=0 call-id=122 Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 104 (Complete=4, Pending=0, Fired=0, Skipped=2, Incomplete=3, Source=/var/lib/pacemaker/pengine/pe-input-125.bz2): Stopped Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 22 18:17:00 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 22 18:16:47 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 22 18:17:00 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n01 Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for ipmilan_node2 on an-a02n02 Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Started an-a02n01) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start ipmilan_node2 ( an-a02n02 ) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n02) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 105, saving inputs in /var/lib/pacemaker/pengine/pe-input-126.bz2 Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 105 (ref=pe_calc-dc-1627082552-283) derived from /var/lib/pacemaker/pengine/pe-input-126.bz2 Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu03_monitor_60000 locally on an-a02n01 | action 12 Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of monitor operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=12:105:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_monitor_60000 Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation ipmilan_node2_start_0 on an-a02n02 | action 13 Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/386) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.338.16 2 Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.338.17 (null) Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=17 Jul 23 16:22:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.338.22 2 Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.339.0 (null) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=339, @num_updates=0 Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu03']/instance_attributes[@id='apc_snmp_node2_an-pdu03-instance_attributes']/nvpair[@id='apc_snmp_node2_an-pdu03-instance_attributes-ip']: @value=10.201.2.3 Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.338.22 -> 0.339.0 from an-a02n02 Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.339.0) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 105 aborted by apc_snmp_node2_an-pdu03-instance_attributes-ip doing modify ip=10.201.2.3: Configuration change | cib=0.339.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu03']/instance_attributes[@id='apc_snmp_node2_an-pdu03-instance_attributes']/nvpair[@id='apc_snmp_node2_an-pdu03-instance_attributes-ip'] complete=true Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 44 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 45 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/44) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 46 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/45) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 47 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/46) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 48 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 49 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 50 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 51 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/44, version=0.339.0) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: modify nvpair[@id='apc_snmp_node2_an-pdu03-instance_attributes-ip'] Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.339.0 Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 44 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=1627003019 Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=1627082013 Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/390) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/391) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/47) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/48) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/49) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/50) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/51) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/45, version=0.339.0) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 45 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node2_an-pdu03' to device list (8 active devices) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.339.0 2 Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.339.1 (null) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/46, version=0.339.1) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 46 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/390, version=0.339.1) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.339.1 2 Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.339.2 (null) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/391, version=0.339.2) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.339.2 2 Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.339.3 (null) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/47, version=0.339.3) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 47 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/48, version=0.339.3) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 48 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/49, version=0.339.3) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 49 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=1627003034 Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=1627082012 Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/50, version=0.339.3) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 50 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/51, version=0.339.3) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 51 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=1627003035 Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627082012 Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-28.raw Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.339.0 of the CIB to disk (digest: 49133dea2a2bc8449f565905153ef22f) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/393, version=0.339.3) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.aAM8uO (digest: /var/lib/pacemaker/cib/cib.DTcgIN) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/394) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/396) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/398) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/400) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-7 offer to an-a02n02 Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-7 offer to an-a02n01 Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-7 requests from 2 outstanding nodes Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/394, version=0.339.3) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/396, version=0.339.3) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/398, version=0.339.3) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/400, version=0.339.3) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: ed782acbb5ff659690d78c049f31bffb Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.339.3 with 0.339.3 from an-a02n01 Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/403, version=0.339.3) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/404) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/405) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-29.raw Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/404, version=0.339.3) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.339.0 of the CIB to disk (digest: 49133dea2a2bc8449f565905153ef22f) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.87jfCW (digest: /var/lib/pacemaker/cib/cib.1I6N1V) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/405, version=0.339.3) Jul 23 16:22:33 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_apc_snmp_metadata_1[4157950] exited with status 0 Jul 23 16:22:34 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157953] exited with status 0 Jul 23 16:22:34 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157954] exited with status 0 Jul 23 16:22:34 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157955] exited with status 0 Jul 23 16:22:34 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4157956] exited with status 0 Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157958] exited with status 0 Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157959] exited with status 0 Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157960] exited with status 0 Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 406) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/406) Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 408) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/407) Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/408) Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.339.3 2 Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.339.4 (null) Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/406, version=0.339.4) Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.339.4 2 Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.339.5 (null) Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/407, version=0.339.5) Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.339.5 2 Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.339.6 (null) Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/408, version=0.339.6) Jul 23 16:22:35 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_apc_snmp_metadata_1[4157961] exited with status 0 Jul 23 16:22:36 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157963] exited with status 0 Jul 23 16:22:36 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157964] exited with status 0 Jul 23 16:22:36 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157965] exited with status 0 Jul 23 16:22:36 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4157966] exited with status 0 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157968] exited with status 0 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157969] exited with status 0 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157970] exited with status 0 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/409) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 105 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.339.6 2 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.339.7 (null) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/409, version=0.339.7) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 105 aborted: History refresh | source=process_resource_updates:268 complete=true Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/412) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/413) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/414) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/412, version=0.339.7) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.339.7 2 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.339.8 (null) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/413, version=0.339.8) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/414, version=0.339.8) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 22 18:17:00 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node2_an-pdu03 on an-a02n01 changed: hash was b65fbd52b0bbfad83728e2328fb84a09 vs. now e7b5280893307b461d430ebe126191cd (reload:3.10.2) 0:1;45:0:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node2_an-pdu03 on an-a02n01 because resource parameters have changed | apc_snmp_node2_an-pdu03_clear_failcount_0 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 22 18:16:47 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 22 18:17:00 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node2_an-pdu03 on an-a02n02 changed: hash was b65fbd52b0bbfad83728e2328fb84a09 vs. now e7b5280893307b461d430ebe126191cd (reload:3.10.2) 0:1;35:101:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node2_an-pdu03 on an-a02n02 because resource parameters have changed | apc_snmp_node2_an-pdu03_clear_failcount_0 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu03 on an-a02n02 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for delay_node2 on an-a02n01 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Started an-a02n01) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node2_an-pdu03 ( an-a02n02 ) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n02) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Move delay_node2 ( an-a02n02 -> an-a02n01 ) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n02) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 106, saving inputs in /var/lib/pacemaker/pengine/pe-input-127.bz2 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 106 (ref=pe_calc-dc-1627082557-294) derived from /var/lib/pacemaker/pengine/pe-input-127.bz2 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node2_an-pdu03_start_0 on an-a02n02 | action 16 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-controld [121633] (te_crm_command) info: Executing crm-event (13) without waiting: clear_failcount on an-a02n02 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-controld [121633] (te_crm_command) info: Executing crm-event (7) locally without waiting: clear_failcount on an-a02n01 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation delay_node2_stop_0 on an-a02n02 | action 20 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-controld [121633] (update_attrd_clear_failures) info: Asking pacemaker-attrd to clear failure of all operations for apc_snmp_node2_an-pdu03 on cluster node an-a02n01 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]: 1627003019 -> (unset) | from an-a02n01 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section /cib/status/node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_failure_0'] to all (origin=local/crmd/417) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 53 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/53) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 54 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/54) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]: 1627082013 -> (unset) | from an-a02n02 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node2_an-pdu03#start_0' delayed: update 53 in progress Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n02 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node2_an-pdu03#start_0' delayed: update 54 in progress Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.339.8 2 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.339.9 (null) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_failure_0'] Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section /cib/status/node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_failure_0']: OK (rc=0, origin=an-a02n01/crmd/417, version=0.339.9) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 106 aborted by deletion of lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_failure_0']: Resource operation removal | cib=0.339.9 source=process_op_deletion:358 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_failure_0'] complete=false Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.339.9 2 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.339.10 (null) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu03.start_0'] Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/53, version=0.339.10) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 53 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 106 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu03.start_0']: Transient attribute change | cib=0.339.10 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu03.start_0'] complete=false Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=1627082013 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 55 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.339.10 2 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.339.11 (null) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu03.start_0'] Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/54, version=0.339.11) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 54 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 106 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu03.start_0']: Transient attribute change | cib=0.339.11 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu03.start_0'] complete=false Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 56 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.339.11 2 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.339.12 (null) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @operation_key=apc_snmp_node2_an-pdu03_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=16:106:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;16:106:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/84, version=0.339.12) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.339.12 2 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.339.13 (null) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_failure_0'] Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section /cib/status/node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_failure_0']: OK (rc=0, origin=an-a02n02/crmd/85, version=0.339.13) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 106 aborted by deletion of lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_failure_0']: Resource operation removal | cib=0.339.13 source=process_op_deletion:358 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_failure_0'] complete=false Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/55) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/56) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.339.13 2 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.339.14 (null) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu03.start_0'] Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/55, version=0.339.14) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 55 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 106 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu03.start_0']: Transient attribute change | cib=0.339.14 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu03.start_0'] complete=false Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.339.14 2 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.339.15 (null) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu03.start_0'] Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=15 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/56, version=0.339.15) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 56 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 106 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu03.start_0']: Transient attribute change | cib=0.339.15 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu03.start_0'] complete=false Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.339.15 2 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.339.16 (null) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=16 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='delay_node2']/lrm_rsc_op[@id='delay_node2_last_0']: @operation_key=delay_node2_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=20:106:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;20:106:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change= Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/86, version=0.339.16) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.339.16 2 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.339.17 (null) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=17 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='delay_node2']/lrm_rsc_op[@id='delay_node2_last_0']: @transition-magic=0:0;20:106:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=98, @rc-code=0, @op-status=0 Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/87, version=0.339.17) Jul 23 16:22:37 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 106 action 20 (delay_node2_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=98 Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.339.17 2 Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.339.18 (null) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=18 Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:0;16:106:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=96, @rc-code=0, @op-status=0, @exec-time=328 Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/88, version=0.339.18) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 106 action 16 (apc_snmp_node2_an-pdu03_start_0 on an-a02n02) confirmed: ok | rc=0 call-id=96 Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 106 (Complete=4, Pending=0, Fired=0, Skipped=2, Incomplete=3, Source=/var/lib/pacemaker/pengine/pe-input-127.bz2): Stopped Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 22 18:17:00 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 22 18:17:00 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu03 on an-a02n02 Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for delay_node2 on an-a02n01 Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Started an-a02n01) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Started an-a02n02) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n02) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start delay_node2 ( an-a02n01 ) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n02) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 107, saving inputs in /var/lib/pacemaker/pengine/pe-input-128.bz2 Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 107 (ref=pe_calc-dc-1627082558-299) derived from /var/lib/pacemaker/pengine/pe-input-128.bz2 Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node2_an-pdu03_monitor_60000 on an-a02n02 | action 15 Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation delay_node2_start_0 locally on an-a02n01 | action 18 Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for delay_node2 on an-a02n01 | transition_key=18:107:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=delay_node2_start_0 Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:delay_node2 action:start call_id:126 Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/419) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.339.18 2 Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.339.19 (null) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=19 Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='delay_node2']/lrm_rsc_op[@id='delay_node2_last_0']: @operation_key=delay_node2_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=18:107:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;18:107:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-chang Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/419, version=0.339.19) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.339.19 2 Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.339.20 (null) Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=20 Jul 23 16:22:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.339.24 2 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.340.0 (null) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=340, @num_updates=0 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu04']/instance_attributes[@id='apc_snmp_node1_an-pdu04-instance_attributes']/nvpair[@id='apc_snmp_node1_an-pdu04-instance_attributes-ip']: @value=10.201.2.4 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.339.24 -> 0.340.0 from an-a02n02 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.340.0) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 107 aborted by apc_snmp_node1_an-pdu04-instance_attributes-ip doing modify ip=10.201.2.4: Configuration change | cib=0.340.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu04']/instance_attributes[@id='apc_snmp_node1_an-pdu04-instance_attributes']/nvpair[@id='apc_snmp_node1_an-pdu04-instance_attributes-ip'] complete=true Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 57 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 58 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 59 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 60 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 61 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 62 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 63 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 64 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: modify nvpair[@id='apc_snmp_node1_an-pdu04-instance_attributes-ip'] Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.340.0 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/425) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/426) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/57) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/58) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/59) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/60) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/61) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node1_an-pdu04' to device list (8 active devices) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-30.raw Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.340.0 of the CIB to disk (digest: 779459e6f05c9b21dbb3bcd14e315945) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.5PEVCG (digest: /var/lib/pacemaker/cib/cib.FHgXwI) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/425, version=0.340.0) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.340.0 2 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.340.1 (null) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/426, version=0.340.1) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.340.1 2 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.340.2 (null) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/57, version=0.340.2) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 57 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.340.2 2 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.340.3 (null) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/58, version=0.340.3) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 58 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.340.3 2 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.340.4 (null) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/59, version=0.340.4) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 59 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.340.4 2 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.340.5 (null) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/60, version=0.340.5) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 60 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/61, version=0.340.5) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 61 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/428, version=0.340.5) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/429) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/62) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/63) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/64) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/431) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/429, version=0.340.5) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/62, version=0.340.5) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 62 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=1627003034 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=1627082012 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/63, version=0.340.5) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 63 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/64, version=0.340.5) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 64 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=1627003035 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627082012 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/431, version=0.340.5) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/433) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-8 offer to an-a02n02 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-8 offer to an-a02n01 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-8 requests from 2 outstanding nodes Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/433, version=0.340.5) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/435) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/435, version=0.340.5) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: 563dcf7cb8af63f6a58a32a598202099 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.340.5 with 0.340.5 from an-a02n01 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/438, version=0.340.5) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/439) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/440) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/439, version=0.340.5) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/440, version=0.340.5) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4157975] exited with status 0 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-31.raw Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.340.0 of the CIB to disk (digest: 779459e6f05c9b21dbb3bcd14e315945) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.FO7rMP (digest: /var/lib/pacemaker/cib/cib.AB0dMP) Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_apc_snmp_metadata_1[4157977] exited with status 0 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157979] exited with status 0 Jul 23 16:22:40 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157980] exited with status 0 Jul 23 16:22:41 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157981] exited with status 0 Jul 23 16:22:41 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4157982] exited with status 0 Jul 23 16:22:41 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157984] exited with status 0 Jul 23 16:22:41 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157985] exited with status 0 Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157986] exited with status 0 Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 441) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/441) Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 443) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/442) Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/443) Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.340.5 2 Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.340.6 (null) Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/441, version=0.340.6) Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.340.6 2 Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.340.7 (null) Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/442, version=0.340.7) Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.340.7 2 Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.340.8 (null) Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/443, version=0.340.8) Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4157987] exited with status 0 Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_apc_snmp_metadata_1[4157988] exited with status 0 Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157990] exited with status 0 Jul 23 16:22:42 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157991] exited with status 0 Jul 23 16:22:43 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157992] exited with status 0 Jul 23 16:22:43 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4157993] exited with status 0 Jul 23 16:22:43 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157995] exited with status 0 Jul 23 16:22:43 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157996] exited with status 0 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4157997] exited with status 0 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/444) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 107 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.340.8 2 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.340.9 (null) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/444, version=0.340.9) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/447) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/448) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/449) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 107 aborted: History refresh | source=process_resource_updates:268 complete=true Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/447, version=0.340.9) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.340.9 2 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.340.10 (null) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/448, version=0.340.10) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/449, version=0.340.10) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu04 on an-a02n01 changed: hash was dcfe282eb9c3d7f7189c538ed5e16421 vs. now 953d17264336f1467fe1409a4ee17bd5 (reload:3.10.2) 0:1;11:2:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu04 on an-a02n01 because resource parameters have changed | apc_snmp_node1_an-pdu04_clear_failcount_0 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 22 18:17:00 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 22 18:17:00 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu04 on an-a02n02 changed: hash was dcfe282eb9c3d7f7189c538ed5e16421 vs. now 953d17264336f1467fe1409a4ee17bd5 (reload:3.10.2) 0:1;33:101:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu04 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu04_clear_failcount_0 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n02 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for ipmilan_node2 on an-a02n01 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for ipmilan_node1 on an-a02n02 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Started an-a02n01) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu04 ( an-a02n02 ) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Started an-a02n02) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Move ipmilan_node2 ( an-a02n02 -> an-a02n01 ) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n01) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Move ipmilan_node1 ( an-a02n01 -> an-a02n02 ) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n02) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 108, saving inputs in /var/lib/pacemaker/pengine/pe-input-129.bz2 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 108 (ref=pe_calc-dc-1627082564-310) derived from /var/lib/pacemaker/pengine/pe-input-129.bz2 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu04_start_0 on an-a02n02 | action 17 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (te_crm_command) info: Executing crm-event (12) without waiting: clear_failcount on an-a02n02 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (te_crm_command) info: Executing crm-event (2) locally without waiting: clear_failcount on an-a02n01 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation ipmilan_node2_stop_0 on an-a02n02 | action 21 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation ipmilan_node1_stop_0 locally on an-a02n01 | action 26 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for ipmilan_node1 on an-a02n01 | transition_key=26:108:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=ipmilan_node1_stop_0 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:ipmilan_node1 action:stop call_id:129 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) info: Result of monitor operation for ipmilan_node1 on an-a02n01: Cancelled | call=109 key=ipmilan_node1_monitor_60000 confirmed=true Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/452) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: ipmilan_node1 stop (call 129) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for ipmilan_node1 on an-a02n01: ok | rc=0 call=129 key=ipmilan_node1_stop_0 confirmed=true cib-update=453 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/453) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (update_attrd_clear_failures) info: Asking pacemaker-attrd to clear failure of all operations for apc_snmp_node1_an-pdu04 on cluster node an-a02n01 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section /cib/status/node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_failure_0'] to all (origin=local/crmd/454) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.340.10 2 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.340.11 (null) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ipmilan_node1']/lrm_rsc_op[@id='ipmilan_node1_last_0']: @operation_key=ipmilan_node1_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=26:108:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;26:108:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-c Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/452, version=0.340.11) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 66 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]: 1627003035 -> (unset) | from an-a02n01 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.340.11 2 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.340.12 (null) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 67 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ipmilan_node1']/lrm_rsc_op[@id='ipmilan_node1_last_0']: @transition-magic=0:0;26:108:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=129, @rc-code=0, @op-status=0 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/453, version=0.340.12) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 108 action 26 (ipmilan_node1_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=129 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation ipmilan_node1_start_0 on an-a02n02 | action 27 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n02 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node1_an-pdu04#start_0' delayed: update 66 in progress Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]: 1627082012 -> (unset) | from an-a02n02 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node1_an-pdu04#start_0' delayed: update 67 in progress Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.340.12 2 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.340.13 (null) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_failure_0'] Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section /cib/status/node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_failure_0']: OK (rc=0, origin=an-a02n01/crmd/454, version=0.340.13) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 108 aborted by deletion of lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_failure_0']: Resource operation removal | cib=0.340.13 source=process_op_deletion:358 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_failure_0'] complete=false Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.340.13 2 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.340.14 (null) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=17:108:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;17:108:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/93, version=0.340.14) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/66) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/67) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.340.14 2 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.340.15 (null) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu04.start_0'] Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=15 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/66, version=0.340.15) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 66 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 108 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.340.15 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu04.start_0'] complete=false Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 68 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.340.15 2 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.340.16 (null) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu04.start_0'] Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=16 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/67, version=0.340.16) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 108 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.340.16 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu04.start_0'] complete=false Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 67 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627082012 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 69 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.340.16 2 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.340.17 (null) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_failure_0'] Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=17 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section /cib/status/node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_failure_0']: OK (rc=0, origin=an-a02n02/crmd/94, version=0.340.17) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 108 aborted by deletion of lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_failure_0']: Resource operation removal | cib=0.340.17 source=process_op_deletion:358 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_failure_0'] complete=false Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.340.17 2 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.340.18 (null) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=18 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ipmilan_node2']/lrm_rsc_op[@id='ipmilan_node2_last_0']: @operation_key=ipmilan_node2_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=21:108:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;21:108:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-c Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/95, version=0.340.18) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.340.18 2 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.340.19 (null) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=19 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ipmilan_node1']/lrm_rsc_op[@id='ipmilan_node1_last_0']: @operation_key=ipmilan_node1_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=27:108:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;27:108:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/96, version=0.340.19) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.340.19 2 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.340.20 (null) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=20 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ipmilan_node2']/lrm_rsc_op[@id='ipmilan_node2_last_0']: @transition-magic=0:0;21:108:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=102, @rc-code=0, @op-status=0, @exec-time=1 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/97, version=0.340.20) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 108 action 21 (ipmilan_node2_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=102 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/68) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/69) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.340.20 2 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.340.21 (null) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu04.start_0'] Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=21 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/68, version=0.340.21) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 68 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 108 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.340.21 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu04.start_0'] complete=false Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.340.21 2 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.340.22 (null) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu04.start_0'] Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=22 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/69, version=0.340.22) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 69 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 108 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.340.22 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu04.start_0'] complete=false Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.340.22 2 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.340.23 (null) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=23 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ipmilan_node1']/lrm_rsc_op[@id='ipmilan_node1_last_0']: @transition-magic=0:0;27:108:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=103, @rc-code=0, @op-status=0, @exec-time=284 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/98, version=0.340.23) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 108 action 27 (ipmilan_node1_start_0 on an-a02n02) confirmed: ok | rc=0 call-id=103 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.340.23 2 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.340.24 (null) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=24 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:0;17:108:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=100, @rc-code=0, @op-status=0, @exec-time=376 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/99, version=0.340.24) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 108 action 17 (apc_snmp_node1_an-pdu04_start_0 on an-a02n02) confirmed: ok | rc=0 call-id=100 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 108 (Complete=6, Pending=0, Fired=0, Skipped=3, Incomplete=4, Source=/var/lib/pacemaker/pengine/pe-input-129.bz2): Stopped Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 22 18:17:00 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n02 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for ipmilan_node2 on an-a02n01 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for ipmilan_node1 on an-a02n02 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Started an-a02n01) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Started an-a02n02) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Started an-a02n02) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start ipmilan_node2 ( an-a02n01 ) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n01) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n02) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 109, saving inputs in /var/lib/pacemaker/pengine/pe-input-130.bz2 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 109 (ref=pe_calc-dc-1627082564-317) derived from /var/lib/pacemaker/pengine/pe-input-130.bz2 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu04_monitor_60000 on an-a02n02 | action 15 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation ipmilan_node2_start_0 locally on an-a02n01 | action 18 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for ipmilan_node2 on an-a02n01 | transition_key=18:109:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=ipmilan_node2_start_0 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:ipmilan_node2 action:start call_id:130 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation ipmilan_node1_monitor_60000 on an-a02n02 | action 24 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/456) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.340.24 2 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.340.25 (null) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=25 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ipmilan_node2']/lrm_rsc_op[@id='ipmilan_node2_last_0']: @operation_key=ipmilan_node2_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=18:109:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;18:109:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/456, version=0.340.25) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.340.25 2 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.340.26 (null) Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=26 Jul 23 16:22:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.340.32 2 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.341.0 (null) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=341, @num_updates=0 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu04']/instance_attributes[@id='apc_snmp_node2_an-pdu04-instance_attributes']/nvpair[@id='apc_snmp_node2_an-pdu04-instance_attributes-ip']: @value=10.201.2.4 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.340.32 -> 0.341.0 from an-a02n02 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.341.0) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 70 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 71 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/70) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 72 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 109 aborted by apc_snmp_node2_an-pdu04-instance_attributes-ip doing modify ip=10.201.2.4: Configuration change | cib=0.341.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu04']/instance_attributes[@id='apc_snmp_node2_an-pdu04-instance_attributes']/nvpair[@id='apc_snmp_node2_an-pdu04-instance_attributes-ip'] complete=true Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 73 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/71) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 74 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/72) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 75 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 76 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/73) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 77 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/74) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/75) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.341.0 2 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.341.1 (null) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: modify nvpair[@id='apc_snmp_node2_an-pdu04-instance_attributes-ip'] Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.341.0 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/70, version=0.341.1) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 70 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.341.1 2 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.341.2 (null) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/71, version=0.341.2) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 71 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.341.2 2 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.341.3 (null) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/72, version=0.341.3) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 72 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.341.3 2 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.341.4 (null) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/73, version=0.341.4) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 73 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/74, version=0.341.4) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 74 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/75, version=0.341.4) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 75 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=1627003034 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=1627082012 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/462) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/463) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node2_an-pdu04' to device list (8 active devices) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/76) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/77) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/462, version=0.341.4) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.341.4 2 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.341.5 (null) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/463, version=0.341.5) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.341.5 2 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.341.6 (null) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/76, version=0.341.6) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 76 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.341.6 2 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.341.7 (null) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/77, version=0.341.7) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 77 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-32.raw Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.341.0 of the CIB to disk (digest: fdcb7a1f3a0d9dd650bdfae41a9f478b) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.1W4n42 (digest: /var/lib/pacemaker/cib/cib.9zkawZ) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/465, version=0.341.7) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/466) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/468) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/470) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/472) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-9 offer to an-a02n02 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-9 offer to an-a02n01 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-9 requests from 2 outstanding nodes Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/466, version=0.341.7) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/468, version=0.341.7) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/470, version=0.341.7) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/472, version=0.341.7) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: 0226e2d0923cf1c7328bdd2bb69480ba Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.341.7 with 0.341.7 from an-a02n01 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/475, version=0.341.7) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/476) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/477) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-33.raw Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/476, version=0.341.7) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.341.0 of the CIB to disk (digest: fdcb7a1f3a0d9dd650bdfae41a9f478b) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.wDOXTd (digest: /var/lib/pacemaker/cib/cib.5aCr2e) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/477, version=0.341.7) Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4158613] exited with status 0 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_apc_snmp_metadata_1[4158623] exited with status 0 Jul 23 16:22:46 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4158663] exited with status 0 Jul 23 16:22:47 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4158678] exited with status 0 Jul 23 16:22:47 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4158688] exited with status 0 Jul 23 16:22:47 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4158887] exited with status 0 Jul 23 16:22:47 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4158898] exited with status 0 Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4158908] exited with status 0 Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4158909] exited with status 0 Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 478) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/478) Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 480) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/479) Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/480) Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.341.7 2 Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.341.8 (null) Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/478, version=0.341.8) Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.341.8 2 Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.341.9 (null) Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/479, version=0.341.9) Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.341.9 2 Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.341.10 (null) Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/480, version=0.341.10) Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4158910] exited with status 0 Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_apc_snmp_metadata_1[4158911] exited with status 0 Jul 23 16:22:48 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4158913] exited with status 0 Jul 23 16:22:49 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4158914] exited with status 0 Jul 23 16:22:49 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4158917] exited with status 0 Jul 23 16:22:49 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4158919] exited with status 0 Jul 23 16:22:49 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4158921] exited with status 0 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.341.10 2 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.342.0 (null) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=342, @num_updates=0 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2']: @devices=apc_snmp_node1_an-pdu04 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.341.10 -> 0.342.0 from an-a02n02 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.342.0) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 79 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 2 from fencing topology for an-a02n01 (2 active levels remaining) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 80 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 81 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 82 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 83 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 84 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 85 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 86 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/79) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/80) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/81) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/82) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/83) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/84) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.342.0 2 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.342.1 (null) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/79, version=0.342.1) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 79 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.342.1 2 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.342.2 (null) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-34.raw Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/80, version=0.342.2) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 80 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.342.2 2 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.342.3 (null) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/81, version=0.342.3) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 81 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.342.3 2 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.342.4 (null) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/82, version=0.342.4) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 82 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/83, version=0.342.4) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 83 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/84, version=0.342.4) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 84 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=1627003034 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=1627082012 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/85) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/86) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.342.0 of the CIB to disk (digest: bb364a28474464cfeca75e0f497d0f7e) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.342.4 2 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.342.5 (null) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/85, version=0.342.5) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.Pp7RLB (digest: /var/lib/pacemaker/cib/cib.furjPz) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 85 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.342.5 2 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.342.6 (null) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/86, version=0.342.6) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 86 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.342.6 2 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.343.0 (null) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03'] Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=343, @num_updates=0 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.343.0) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-35.raw Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.343.0 of the CIB to disk (digest: 7a8ef0f45eadab4fdb5aede58f233e19) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.uSfL8J (digest: /var/lib/pacemaker/cib/cib.8ZRYbK) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4158924] exited with status 0 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4158941] exited with status 0 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/481) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 109 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.343.0 2 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.343.1 (null) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/481, version=0.343.1) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/484) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/485) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/488) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/489) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/490) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/484, version=0.343.1) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.343.1 2 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.343.2 (null) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/485, version=0.343.2) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/488, version=0.343.2) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.343.2 2 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.343.3 (null) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/489, version=0.343.3) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/490, version=0.343.3) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/491, version=0.343.3) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/492) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/494) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/496) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/498) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-10 offer to an-a02n02 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-10 offer to an-a02n01 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-10 requests from 2 outstanding nodes Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/492, version=0.343.3) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/494, version=0.343.3) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/496, version=0.343.3) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/498, version=0.343.3) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: cb8789aaa7c654b5cdbe44da82772e05 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.343.3 with 0.343.3 from an-a02n01 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/501, version=0.343.3) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/502) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/503) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-36.raw Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/502, version=0.343.3) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.343.0 of the CIB to disk (digest: 7a8ef0f45eadab4fdb5aede58f233e19) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.0ptvJV (digest: /var/lib/pacemaker/cib/cib.EFJHEX) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/503, version=0.343.3) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4158942] exited with status 0 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.343.3 2 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.344.0 (null) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=344, @num_updates=0 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources: Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.344.0) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create resources Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.344.0 Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) info: Overwriting existing entry for apc_snmp_node1_an-pdu03 from CIB Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node1_an-pdu03' to device list (8 active devices) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-37.raw Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.344.0 of the CIB to disk (digest: 0bb3754bf209f5091faf20d077fd54ea) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.XkjLF5 (digest: /var/lib/pacemaker/cib/cib.WBBzJ6) Jul 23 16:22:50 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_apc_snmp_metadata_1[4158944] exited with status 0 Jul 23 16:22:51 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4158947] exited with status 0 Jul 23 16:22:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.344.0 2 Jul 23 16:22:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.0 (null) Jul 23 16:22:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=345 Jul 23 16:22:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu04']: Jul 23 16:22:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.345.0) Jul 23 16:22:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create primitive[@id='apc_snmp_node1_an-pdu04'] Jul 23 16:22:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.345.0 Jul 23 16:22:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu04 has been disabled Jul 23 16:22:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-38.raw Jul 23 16:22:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.345.0 of the CIB to disk (digest: 2fd8a84ca2e0bff215c4b12561f5f467) Jul 23 16:22:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.gw3mEo (digest: /var/lib/pacemaker/cib/cib.OL6ttp) Jul 23 16:22:51 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4158948] exited with status 0 Jul 23 16:22:51 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4158950] exited with status 0 Jul 23 16:22:51 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4158963] exited with status 0 Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4158965] exited with status 0 Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4158967] exited with status 0 Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4158975] exited with status 0 Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 504) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/504) Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/505) Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 506) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.0 2 Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.1 (null) Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/504, version=0.345.1) Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.1 2 Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.2 (null) Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/505, version=0.345.2) Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/506) Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.2 2 Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.3 (null) Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/506, version=0.345.3) Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4158976] exited with status 0 Jul 23 16:22:52 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_apc_snmp_metadata_1[4158977] exited with status 0 Jul 23 16:22:53 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4158979] exited with status 0 Jul 23 16:22:53 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4158980] exited with status 0 Jul 23 16:22:53 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4158981] exited with status 0 Jul 23 16:22:53 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4158982] exited with status 0 Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4158984] exited with status 0 Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4158999] exited with status 0 Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4159001] exited with status 0 Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/507) Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 109 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.3 2 Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.4 (null) Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/507, version=0.345.4) Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 109 aborted: History refresh | source=process_resource_updates:268 complete=true Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/510) Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/511) Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/512) Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/510, version=0.345.4) Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.4 2 Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.5 (null) Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/511, version=0.345.5) Jul 23 16:22:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/512, version=0.345.5) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node2_an-pdu04 on an-a02n01 changed: hash was 09508b690d4ef39a75f68a8014ee9848 vs. now 25e19c24d758eebc99cf83d00d559588 (reload:3.10.2) 0:1;15:2:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node2_an-pdu04 on an-a02n01 because resource parameters have changed | apc_snmp_node2_an-pdu04_clear_failcount_0 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 22 18:17:00 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node2_an-pdu04 on an-a02n02 changed: hash was 09508b690d4ef39a75f68a8014ee9848 vs. now 25e19c24d758eebc99cf83d00d559588 (reload:3.10.2) 0:1;37:101:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node2_an-pdu04 on an-a02n02 because resource parameters have changed | apc_snmp_node2_an-pdu04_clear_failcount_0 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:13:19 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n01 changed: hash was b6d660e5416057f7575eac6369636a4f vs. now 3fd89635ef735220dd13344790fbf853 (reload:3.10.2) 0:0;13:104:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_action_definition) info: params:reload Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 60000ms-interval monitor action for apc_snmp_node1_an-pdu03 on an-a02n01 changed: hash was 5e7bff0e01ad15b6456cdc9f08b6b6ff vs. now df3699fd58da6b53bb2ebd50f24d2a95 (reschedule:3.10.2) 0:0;12:105:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_action_definition) info: params:reload Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu04 on an-a02n01 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for delay_node2 on an-a02n02 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for delay_node1 on an-a02n01 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu04 ( an-a02n02 ) due to node availability Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Started an-a02n02) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node2_an-pdu04 ( an-a02n01 ) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Move delay_node2 ( an-a02n01 -> an-a02n02 ) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Move delay_node1 ( an-a02n02 -> an-a02n01 ) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Move apc_snmp_node1_an-pdu03 ( an-a02n01 -> an-a02n02 ) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 110, saving inputs in /var/lib/pacemaker/pengine/pe-input-131.bz2 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 110 (ref=pe_calc-dc-1627082575-336) derived from /var/lib/pacemaker/pengine/pe-input-131.bz2 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu04_stop_0 on an-a02n02 | action 17 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node2_an-pdu04_start_0 locally on an-a02n01 | action 20 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node2_an-pdu04 on an-a02n01 | transition_key=20:110:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu04_start_0 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-controld [121633] (te_crm_command) info: Executing crm-event (15) without waiting: clear_failcount on an-a02n02 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node2_an-pdu04 action:start call_id:132 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-controld [121633] (te_crm_command) info: Executing crm-event (8) locally without waiting: clear_failcount on an-a02n01 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/516) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation delay_node2_stop_0 locally on an-a02n01 | action 24 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for delay_node2 on an-a02n01 | transition_key=24:110:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=delay_node2_stop_0 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:delay_node2 action:stop call_id:134 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation delay_node1_stop_0 on an-a02n02 | action 29 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 locally on an-a02n01 | action 44 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.5 2 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.6 (null) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @operation_key=apc_snmp_node2_an-pdu04_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=20:110:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;20:110:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/516, version=0.345.6) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/517) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.6 2 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.7 (null) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='delay_node2']/lrm_rsc_op[@id='delay_node2_last_0']: @operation_key=delay_node2_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=24:110:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;24:110:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change= Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/517, version=0.345.7) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.7 2 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.8 (null) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=17:110:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;17:110:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code= Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n02 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 88 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]: 1627082012 -> (unset) | from an-a02n02 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/108, version=0.345.8) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 89 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.8 2 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.9 (null) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:0;17:110:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=107, @rc-code=0, @op-status=0 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/109, version=0.345.9) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/88) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/89) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.9 2 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.10 (null) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu04.start_0'] Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/88, version=0.345.10) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 88 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.10 2 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.11 (null) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu04.start_0'] Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/89, version=0.345.11) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 89 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=1627003034 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: delay_node2 stop (call 134) exited with status 0 (execution time 2ms, queue time 0ms) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=44:110:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_stop_0 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu03 action:stop call_id:136 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.11 2 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.12 (null) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_failure_0'] Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu03 stop (call 136) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 110 action 17 (apc_snmp_node1_an-pdu04_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=107 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 110 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu04.start_0']: Transient attribute change | cib=0.345.10 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu04.start_0'] complete=false Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 110 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu04.start_0']: Transient attribute change | cib=0.345.11 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu04.start_0'] complete=false Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) info: Result of monitor operation for delay_node2 on an-a02n01: Cancelled | call=127 key=delay_node2_monitor_60000 confirmed=true Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for delay_node2 on an-a02n01: ok | rc=0 call=134 key=delay_node2_stop_0 confirmed=true cib-update=519 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) info: Result of monitor operation for apc_snmp_node1_an-pdu03 on an-a02n01: Cancelled | call=125 key=apc_snmp_node1_an-pdu03_monitor_60000 confirmed=true Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01: ok | rc=0 call=136 key=apc_snmp_node1_an-pdu03_stop_0 confirmed=true cib-update=520 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-controld [121633] (update_attrd_clear_failures) info: Asking pacemaker-attrd to clear failure of all operations for apc_snmp_node2_an-pdu04 on cluster node an-a02n01 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 90 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section /cib/status/node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_failure_0']: OK (rc=0, origin=an-a02n02/crmd/110, version=0.345.12) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]: 1627003034 -> (unset) | from an-a02n01 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 91 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 110 aborted by deletion of lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_failure_0']: Resource operation removal | cib=0.345.12 source=process_op_deletion:358 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_failure_0'] complete=false Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.12 2 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.13 (null) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='delay_node1']/lrm_rsc_op[@id='delay_node1_last_0']: @operation_key=delay_node1_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=29:110:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;29:110:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change= Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/111, version=0.345.13) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.13 2 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.14 (null) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='delay_node1']/lrm_rsc_op[@id='delay_node1_last_0']: @transition-magic=0:0;29:110:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=109, @rc-code=0, @op-status=0 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/112, version=0.345.14) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 110 action 29 (delay_node1_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=109 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/518) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/519) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/520) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section /cib/status/node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_failure_0'] to all (origin=local/crmd/521) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/90) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/91) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.14 2 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.15 (null) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=15 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=44:110:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;44:110:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code= Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/518, version=0.345.15) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.15 2 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.16 (null) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=16 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='delay_node2']/lrm_rsc_op[@id='delay_node2_last_0']: @transition-magic=0:0;24:110:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=134, @rc-code=0, @op-status=0, @exec-time=2 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/519, version=0.345.16) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 110 action 24 (delay_node2_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=134 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.16 2 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.17 (null) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=17 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;44:110:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=136, @rc-code=0, @op-status=0 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/520, version=0.345.17) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 110 action 44 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=136 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.17 2 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.18 (null) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_failure_0'] Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=18 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section /cib/status/node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_failure_0']: OK (rc=0, origin=an-a02n01/crmd/521, version=0.345.18) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 110 aborted by deletion of lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_failure_0']: Resource operation removal | cib=0.345.18 source=process_op_deletion:358 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_failure_0'] complete=false Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.18 2 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.19 (null) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu04.start_0'] Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=19 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/90, version=0.345.19) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 90 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 110 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu04.start_0']: Transient attribute change | cib=0.345.19 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu04.start_0'] complete=false Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.19 2 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.20 (null) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu04.start_0'] Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=20 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/91, version=0.345.20) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 91 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:22:55 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 110 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu04.start_0']: Transient attribute change | cib=0.345.20 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu04.start_0'] complete=false Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node2_an-pdu04 start (call 132) exited with status 0 (execution time 331ms, queue time 0ms) Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_apc_snmp_metadata_1[4159019] exited with status 0 Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for apc_snmp_node2_an-pdu04 on an-a02n01: ok | rc=0 call=132 key=apc_snmp_node2_an-pdu04_start_0 confirmed=true cib-update=522 Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/522) Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.20 2 Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.21 (null) Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=21 Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:0;20:110:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=132, @rc-code=0, @op-status=0, @exec-time=331 Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/522, version=0.345.21) Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 110 action 20 (apc_snmp_node2_an-pdu04_start_0 on an-a02n01) confirmed: ok | rc=0 call-id=132 Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 110 (Complete=7, Pending=0, Fired=0, Skipped=4, Incomplete=7, Source=/var/lib/pacemaker/pengine/pe-input-131.bz2): Stopped Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu04 on an-a02n01 Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for delay_node2 on an-a02n02 Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for delay_node1 on an-a02n01 Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Started an-a02n02) Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Started an-a02n01) Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start delay_node2 ( an-a02n02 ) Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start delay_node1 ( an-a02n01 ) Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 111, saving inputs in /var/lib/pacemaker/pengine/pe-input-132.bz2 Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 111 (ref=pe_calc-dc-1627082576-344) derived from /var/lib/pacemaker/pengine/pe-input-132.bz2 Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node2_an-pdu04_monitor_60000 locally on an-a02n01 | action 14 Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of monitor operation for apc_snmp_node2_an-pdu04 on an-a02n01 | transition_key=14:111:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu04_monitor_60000 Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation delay_node2_start_0 on an-a02n02 | action 17 Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation delay_node1_start_0 locally on an-a02n01 | action 21 Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/524) Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for delay_node1 on an-a02n01 | transition_key=21:111:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=delay_node1_start_0 Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/525) Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:delay_node1 action:start call_id:138 Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 on an-a02n02 | action 35 Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.21 2 Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.22 (null) Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=22 Jul 23 16:22:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/92, version=0.345.34) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 92 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.34 2 Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.35 (null) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=35 Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/93, version=0.345.35) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 93 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627082589 Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 111 aborted by status-2-fail-count-apc_snmp_node1_an-pdu03.start_0 doing create fail-count-apc_snmp_node1_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.345.34 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 111 aborted by status-2-last-failure-apc_snmp_node1_an-pdu03.start_0 doing create last-failure-apc_snmp_node1_an-pdu03#start_0=1627082589: Transient attribute change | cib=0.345.35 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:22:56 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:22:56 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Started an-a02n02) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Started an-a02n01) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 112, saving inputs in /var/lib/pacemaker/pengine/pe-input-133.bz2 Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627082589-351 is obsolete Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:22:56 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:22:56 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n01 Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Started an-a02n02) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Started an-a02n01) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n02 -> an-a02n01 ) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 113, saving inputs in /var/lib/pacemaker/pengine/pe-input-134.bz2 Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 113 (ref=pe_calc-dc-1627082589-352) derived from /var/lib/pacemaker/pengine/pe-input-134.bz2 Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 on an-a02n02 | action 11 Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.35 2 Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.36 (null) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=36 Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @transition-key=11:113:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;11:113:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/119, version=0.345.36) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.36 2 Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.37 (null) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=37 Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;11:113:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=113, @rc-code=0, @op-status=0 Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/120, version=0.345.37) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 113 action 11 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=113 Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 locally on an-a02n01 | action 38 Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=38:113:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_start_0 Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/533) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu03 action:start call_id:140 Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.37 2 Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.38 (null) Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=38 Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @transition-key=38:113:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;38:113:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change= Jul 23 16:23:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/533, version=0.345.38) Jul 23 16:23:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: c47a6b4e95d7431909917c2bdffe04f4 for 0.345.38 (0x56197640dcd0 0) Jul 23 16:23:15 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4159153] error output [ 2021-07-23 16:23:15,635 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:23:15 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4159153] error output [ ] Jul 23 16:23:15 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4159153] error output [ ] Jul 23 16:23:15 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4159153] error output [ 2021-07-23 16:23:15,635 ERROR: Please use '-h' for usage ] Jul 23 16:23:15 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4159153] error output [ ] Jul 23 16:23:15 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4159153] stderr: [ 2021-07-23 16:23:15,635 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:23:15 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4159153] stderr: [ ] Jul 23 16:23:15 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4159153] stderr: [ ] Jul 23 16:23:15 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4159153] stderr: [ 2021-07-23 16:23:15,635 ERROR: Please use '-h' for usage ] Jul 23 16:23:15 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4159153] stderr: [ ] Jul 23 16:23:15 an-a02n01.alteeve.com pacemaker-fenced [121629] (internal_stonith_action_execute) info: Attempt 2 to execute fence_apc_snmp (monitor). remaining timeout is 14 Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n01 Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 94 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: 1627082589 -> (unset) | from an-a02n01 Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 95 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/94) Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/95) Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.38 2 Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.39 (null) Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=39 Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/94, version=0.345.39) Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 94 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 113 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.345.39 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] complete=false Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.39 2 Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.40 (null) Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=40 Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/95, version=0.345.40) Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 95 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 113 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.345.40 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] complete=false Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.40 2 Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.41 (null) Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=41 Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/121, version=0.345.40) Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.40 2 Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.345.41 (null) Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=41 Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/122, version=0.345.41) Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 113 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu03']: Resource state removal | cib=0.345.41 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] complete=false Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.345.41 2 Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.346.0 (null) Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=346, @num_updates=0 Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']: @value=1627082596 Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n02/crmd/124, version=0.346.0) Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 113 aborted by cib-bootstrap-options-last-lrm-refresh doing modify last-lrm-refresh=1627082596: Configuration change | cib=0.346.0 source=te_update_diff_v2:474 path=/cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh'] complete=false Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-39.raw Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.346.0 of the CIB to disk (digest: eae429533562bb1e20d7929bf35529d1) Jul 23 16:23:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.Oz803D (digest: /var/lib/pacemaker/cib/cib.t2pcAF) Jul 23 16:23:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: c2c07cd045b9d490d650e6b5d05df5c3 for 0.346.0 (0x5619760ad450 0) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4159224] error output [ 2021-07-23 16:23:22,793 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4159224] error output [ ] Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4159224] error output [ ] Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4159224] error output [ 2021-07-23 16:23:22,793 ERROR: Please use '-h' for usage ] Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4159224] error output [ ] Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4159224] stderr: [ 2021-07-23 16:23:22,793 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4159224] stderr: [ ] Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4159224] stderr: [ ] Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4159224] stderr: [ 2021-07-23 16:23:22,793 ERROR: Please use '-h' for usage ] Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4159224] stderr: [ ] Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_remaining_timeout) info: Attempted to execute agent fence_apc_snmp (monitor) the maximum number of times (2) allowed Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_operation) notice: Operation 'monitor' [4159224] using apc_snmp_node1_an-pdu03 returned -201 (Generic Pacemaker error) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu03 start (call 140) exited with status 1 (execution time 13319ms, queue time 0ms) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for apc_snmp_node1_an-pdu03 on an-a02n01: error | rc=1 call=140 key=apc_snmp_node1_an-pdu03_start_0 confirmed=true cib-update=535 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/535) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.346.0 2 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.346.1 (null) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:1;38:113:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=140, @rc-code=1, @op-status=0, @exec-time=13319 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/96, version=0.346.2) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 96 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/97) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.346.2 2 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.346.3 (null) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/97, version=0.346.3) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 97 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=1627082602 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 113 aborted by status-1-fail-count-apc_snmp_node1_an-pdu03.start_0 doing create fail-count-apc_snmp_node1_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.346.2 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 113 aborted by status-1-last-failure-apc_snmp_node1_an-pdu03.start_0 doing create last-failure-apc_snmp_node1_an-pdu03#start_0=1627082602: Transient attribute change | cib=0.346.3 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:23:09 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:23:09 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Started an-a02n02) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Started an-a02n01) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n01 -> an-a02n02 ) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 114, saving inputs in /var/lib/pacemaker/pengine/pe-input-135.bz2 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627082602-355 is obsolete Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:23:09 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:23:09 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Started an-a02n02) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Started an-a02n01) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n01 -> an-a02n02 ) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 115, saving inputs in /var/lib/pacemaker/pengine/pe-input-136.bz2 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 115 (ref=pe_calc-dc-1627082602-356) derived from /var/lib/pacemaker/pengine/pe-input-136.bz2 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu03_monitor_0 on an-a02n02 | action 14 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 locally on an-a02n01 | action 2 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=2:115:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_stop_0 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/539) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu03 action:stop call_id:141 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu03 stop (call 141) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.346.3 2 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01: ok | rc=0 call=141 key=apc_snmp_node1_an-pdu03_stop_0 confirmed=true cib-update=540 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.346.4 (null) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @transition-key=2:115:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;2:115:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1627 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/539, version=0.346.4) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/540) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.346.4 2 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.346.5 (null) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;2:115:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=141, @rc-code=0, @op-status=0 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/540, version=0.346.5) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 115 action 2 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=141 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.346.5 2 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.346.6 (null) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/126, version=0.346.6) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.346.6 2 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.346.7 (null) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:7;14:115:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=118, @rc-code=7, @op-status=0 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/127, version=0.346.7) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 115 action 14 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=118 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 on an-a02n02 | action 39 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.346.7 2 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.346.8 (null) Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @transition-key=39:115:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;39:115:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1 Jul 23 16:23:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/128, version=0.346.8) Jul 23 16:23:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 05ad8f8cf3fe23da74845387b2d1f9fd for 0.346.8 (0x5619760ad450 0) Jul 23 16:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.346.8 2 Jul 23 16:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.347.0 a28da2a03bddb39f0d3e971cd6640c86 Jul 23 16:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu04']/meta_attributes[@id='apc_snmp_node1_an-pdu04-meta_attributes']/nvpair[@id='apc_snmp_node1_an-pdu04-meta_attributes-target-role'] Jul 23 16:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=347, @num_updates=0 Jul 23 16:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.347.0) Jul 23 16:23:28 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 115 aborted by deletion of nvpair[@id='apc_snmp_node1_an-pdu04-meta_attributes-target-role']: Configuration change | cib=0.347.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu04']/meta_attributes[@id='apc_snmp_node1_an-pdu04-meta_attributes']/nvpair[@id='apc_snmp_node1_an-pdu04-meta_attributes-target-role'] complete=false Jul 23 16:23:28 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: (meta) attribute deleted from resource Jul 23 16:23:28 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.347.0 Jul 23 16:23:28 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node1_an-pdu04' to device list (8 active devices) Jul 23 16:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-40.raw Jul 23 16:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.347.0 of the CIB to disk (digest: 71dd588bb0d8988144ae131c7ba8d738) Jul 23 16:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.cyn1Ry (digest: /var/lib/pacemaker/cib/cib.WqI9JA) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.347.0 2 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.348.0 (null) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=348 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Local-only Replace: 0.348.0 from an-a02n01 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.348.0) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by deletion of fencing-level[@id='fl-an-a02n01-2']: Configuration change | cib=0.348.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] complete=false Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 98 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 99 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 100 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 101 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 102 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 103 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 104 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 105 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/543) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/544) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.348.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/98) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/99) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/100) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/101) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/102) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/543, version=0.348.0) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.348.0 2 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.348.1 (null) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/544, version=0.348.1) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.348.1 2 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.348.2 (null) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/98, version=0.348.2) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 98 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.348.2 2 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.348.3 (null) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/99, version=0.348.3) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 99 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.348.3 2 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.348.4 (null) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/100, version=0.348.4) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 100 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.348.4 2 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.348.5 (null) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/101, version=0.348.5) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 101 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=1627082602 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.348.5 2 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.348.6 (null) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/102, version=0.348.6) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 102 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/103) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/104) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/105) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.348.6 2 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.348.7 (null) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/103, version=0.348.7) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 103 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.348.7 2 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.348.8 (null) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/104, version=0.348.8) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 104 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.348.8 2 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.348.9 (null) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/105, version=0.348.9) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 105 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-41.raw Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.348.0 of the CIB to disk (digest: 91f943c1dac5b98e6d799f6aeef67d12) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.SF2t32 (digest: /var/lib/pacemaker/cib/cib.RU21Y2) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/545, version=0.348.9) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/546) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/546, version=0.348.9) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/548) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/550) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-11 offer to an-a02n02 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-11 offer to an-a02n01 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-11 requests from 2 outstanding nodes Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/548, version=0.348.9) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/550, version=0.348.9) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/552) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/552, version=0.348.9) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: ad47c887bc4d7e3e1ab7a73466ae4e44 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.348.9 with 0.348.9 from an-a02n01 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/555, version=0.348.9) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/556) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/557) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/556, version=0.348.9) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/557, version=0.348.9) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4160057] exited with status 0 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-42.raw Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.348.0 of the CIB to disk (digest: 91f943c1dac5b98e6d799f6aeef67d12) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.zeq7Nf (digest: /var/lib/pacemaker/cib/cib.wz8AWg) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu04'] to all (origin=local/cibadmin/2) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.348.9 2 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.349.0 (null) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu04'] Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=349, @num_updates=0 Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.349.0) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Removed 'apc_snmp_node1_an-pdu04' from device list (7 active devices) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-43.raw Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.349.0 of the CIB to disk (digest: 93282e109411f51fa6f057dc01739e27) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.49BT2x (digest: /var/lib/pacemaker/cib/cib.Eeuzxy) Jul 23 16:23:31 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4160060] exited with status 0 Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.349.0 2 Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.350.0 (null) Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=350 Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources: Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.350.0) Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create resources Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.350.0 Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-44.raw Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node1_an-pdu04' to device list (8 active devices) Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.350.0 of the CIB to disk (digest: 881f2d2253092b9bddefbde756f44702) Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.0yTI8s (digest: /var/lib/pacemaker/cib/cib.0QW8Xt) Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4160066] exited with status 0 Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.350.0 2 Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.351.0 (null) Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=351 Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu03']: Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.351.0) Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create primitive[@id='apc_snmp_node2_an-pdu03'] Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.351.0 Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-45.raw Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.351.0 of the CIB to disk (digest: 1c7d1d25634db84aa9743aceef9de1a9) Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.yHKqcA (digest: /var/lib/pacemaker/cib/cib.SuVKiz) Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4160078] exited with status 0 Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4160104] exited with status 0 Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4160106] exited with status 0 Jul 23 16:23:32 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_apc_snmp_metadata_1[4160107] exited with status 0 Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4160109] exited with status 0 Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4160110] exited with status 0 Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by deletion of primitive[@id='apc_snmp_node1_an-pdu04']: Configuration change | cib=0.349.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu04'] complete=false Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by primitive.apc_snmp_node1_an-pdu04 'create': Configuration change | cib=0.350.0 source=te_update_diff_v2:474 path=/cib/configuration/resources complete=false Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by meta_attributes.apc_snmp_node2_an-pdu03-meta_attributes 'create': Configuration change | cib=0.351.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu03'] complete=false Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 558) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/558) Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/559) Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 560) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.351.0 2 Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.351.1 (null) Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/558, version=0.351.1) Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.351.1 2 Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.351.2 (null) Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/559, version=0.351.2) Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/560) Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.351.2 2 Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.351.3 (null) Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/560, version=0.351.3) Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4160112] exited with status 0 Jul 23 16:23:33 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4160113] exited with status 0 Jul 23 16:23:34 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4160116] exited with status 0 Jul 23 16:23:34 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4160117] exited with status 0 Jul 23 16:23:34 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4160118] exited with status 0 Jul 23 16:23:34 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4160120] exited with status 0 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_apc_snmp_metadata_1[4160136] exited with status 0 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4160138] exited with status 0 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4160139] exited with status 0 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/561) Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by deletion of lrm[@id='2']: Resource state removal | cib=0.351.1 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2'] complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv14-file-processor_monitor_0 'create' on an-a02n02: Old event | magic=0:7;20:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.2 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 20 (srv14-file-processor_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=41 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation delay_node1_stop_0 'create' on an-a02n02: Old event | magic=0:0;29:110:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.2 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 110 action 29 (delay_node1_stop_0 on an-a02n02): arrived really late | rc=0 target-rc=0 call-id=109 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation delay_node2_start_0 'create' on an-a02n02: Old event | magic=0:0;17:111:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.2 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 111 action 17 (delay_node2_start_0 on an-a02n02): arrived really late | rc=0 target-rc=0 call-id=110 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation delay_node2_monitor_60000 'create' on an-a02n02: Change in recurring result | magic=0:0;18:111:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.2 source=process_graph_event:408 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 111 action 18 (delay_node2_monitor_60000 on an-a02n02): arrived after initial scheduling | rc=0 target-rc=0 call-id=112 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation apc_snmp_node1_an-pdu04_stop_0 'create' on an-a02n02: Old event | magic=0:0;17:110:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.2 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 110 action 17 (apc_snmp_node1_an-pdu04_stop_0 on an-a02n02): arrived really late | rc=0 target-rc=0 call-id=107 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv22-machineinterface_monitor_0 'create' on an-a02n02: Old event | magic=0:7;28:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.2 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 28 (srv22-machineinterface_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=73 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv13-psql_monitor_0 'create' on an-a02n02: Old event | magic=0:7;19:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.2 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 19 (srv13-psql_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=37 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv24-inventory_monitor_0 'create' on an-a02n02: Old event | magic=0:7;30:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.2 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 30 (srv24-inventory_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=81 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv15-tickets_monitor_0 'create' on an-a02n02: Old event | magic=0:7;21:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.2 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 21 (srv15-tickets_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=45 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv18-inventory_monitor_0 'create' on an-a02n02: Old event | magic=0:7;24:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.2 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 24 (srv18-inventory_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=57 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv21-tickets_monitor_0 'create' on an-a02n02: Old event | magic=0:7;27:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.2 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 27 (srv21-tickets_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=69 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv17-order-processing_monitor_0 'create' on an-a02n02: Old event | magic=0:7;23:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.2 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 23 (srv17-order-processing_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=53 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation ipmilan_node1_start_0 'create' on an-a02n02: Old event | magic=0:0;27:108:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.2 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 108 action 27 (ipmilan_node1_start_0 on an-a02n02): arrived really late | rc=0 target-rc=0 call-id=103 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation ipmilan_node1_monitor_60000 'create' on an-a02n02: Change in recurring result | magic=0:0;24:109:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.2 source=process_graph_event:408 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 109 action 24 (ipmilan_node1_monitor_60000 on an-a02n02): arrived after initial scheduling | rc=0 target-rc=0 call-id=105 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation ipmilan_node2_stop_0 'create' on an-a02n02: Old event | magic=0:0;21:108:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.2 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 108 action 21 (ipmilan_node2_stop_0 on an-a02n02): arrived really late | rc=0 target-rc=0 call-id=102 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv19-uls_monitor_0 'create' on an-a02n02: Old event | magic=0:7;25:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.2 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 25 (srv19-uls_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=61 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation apc_snmp_node2_an-pdu03_start_0 'create' on an-a02n02: Old event | magic=0:0;16:106:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.2 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 106 action 16 (apc_snmp_node2_an-pdu03_start_0 on an-a02n02): arrived really late | rc=0 target-rc=0 call-id=96 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation apc_snmp_node2_an-pdu03_monitor_60000 'create' on an-a02n02: Change in recurring result | magic=0:0;15:107:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.2 source=process_graph_event:408 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 107 action 15 (apc_snmp_node2_an-pdu03_monitor_60000 on an-a02n02): arrived after initial scheduling | rc=0 target-rc=0 call-id=99 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation apc_snmp_node2_an-pdu04_stop_0 'create' on an-a02n02: Old event | magic=0:0;14:103:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.2 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 103 action 14 (apc_snmp_node2_an-pdu04_stop_0 on an-a02n02): arrived really late | rc=0 target-rc=0 call-id=89 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv20-fileprocessor_monitor_0 'create' on an-a02n02: Old event | magic=0:7;26:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.2 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 26 (srv20-fileprocessor_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=65 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv16-machine-interface_monitor_0 'create' on an-a02n02: Old event | magic=0:7;22:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.2 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 22 (srv16-machine-interface_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=49 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv23-orderprocessing_monitor_0 'create' on an-a02n02: Old event | magic=0:7;29:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.2 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 29 (srv23-orderprocessing_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=77 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by deletion of lrm[@id='1']: Resource state removal | cib=0.351.3 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1'] complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted: Peer Cancelled | source=do_te_invoke:129 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.351.3 2 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.351.4 (null) Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/561, version=0.351.4) Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv14-file-processor_monitor_0 'create' on an-a02n01: Old event | magic=0:7;10:0:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.4 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/564) Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 0 action 10 (srv14-file-processor_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=41 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation delay_node1_start_0 'create' on an-a02n01: Old event | magic=0:0;21:111:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.4 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 111 action 21 (delay_node1_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=138 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation delay_node1_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;22:111:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.4 source=process_graph_event:408 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 111 action 22 (delay_node1_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=139 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation delay_node2_stop_0 'create' on an-a02n01: Old event | magic=0:0;24:110:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.4 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/565) Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 110 action 24 (delay_node2_stop_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=134 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation apc_snmp_node1_an-pdu04_stop_0 'create' on an-a02n01: Old event | magic=0:0;1:4:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.4 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 4 action 1 (apc_snmp_node1_an-pdu04_stop_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=92 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation apc_snmp_node1_an-pdu03_start_0 'create' on an-a02n01: Old event | magic=0:1;38:113:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.4 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 113 action 38 (apc_snmp_node1_an-pdu03_start_0 on an-a02n01): arrived really late | rc=1 target-rc=0 call-id=140 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/566) Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv13-psql_monitor_0 'create' on an-a02n01: Old event | magic=0:7;9:0:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.4 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 0 action 9 (srv13-psql_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=37 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv22-machineinterface_start_0 'create' on an-a02n01: Old event | magic=0:0;22:8:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.4 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 8 action 22 (srv22-machineinterface_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=100 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv22-machineinterface_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;24:9:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.4 source=process_graph_event:408 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 9 action 24 (srv22-machineinterface_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=101 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv24-inventory_start_0 'create' on an-a02n01: Old event | magic=0:0;28:10:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.4 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 10 action 28 (srv24-inventory_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=104 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv24-inventory_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;29:10:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.4 source=process_graph_event:408 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 10 action 29 (srv24-inventory_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=105 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv15-tickets_monitor_0 'create' on an-a02n01: Old event | magic=0:7;11:0:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.4 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 0 action 11 (srv15-tickets_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=45 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv18-inventory_monitor_0 'create' on an-a02n01: Old event | magic=0:7;14:0:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.4 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 0 action 14 (srv18-inventory_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=57 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv21-tickets_start_0 'create' on an-a02n01: Old event | magic=0:0;19:7:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.4 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 7 action 19 (srv21-tickets_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=98 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv21-tickets_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;21:8:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.4 source=process_graph_event:408 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 8 action 21 (srv21-tickets_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=99 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv17-order-processing_monitor_0 'create' on an-a02n01: Old event | magic=0:7;13:0:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.4 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 0 action 13 (srv17-order-processing_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=53 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation ipmilan_node1_stop_0 'create' on an-a02n01: Old event | magic=0:0;26:108:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.4 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 108 action 26 (ipmilan_node1_stop_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=129 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation ipmilan_node2_start_0 'create' on an-a02n01: Old event | magic=0:0;18:109:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.4 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 109 action 18 (ipmilan_node2_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=130 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation ipmilan_node2_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;19:109:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.4 source=process_graph_event:408 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 109 action 19 (ipmilan_node2_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=131 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv19-uls_start_0 'create' on an-a02n01: Old event | magic=0:0;13:5:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.4 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 5 action 13 (srv19-uls_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=94 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv19-uls_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;15:6:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.4 source=process_graph_event:408 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 6 action 15 (srv19-uls_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=95 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation apc_snmp_node2_an-pdu03_stop_0 'create' on an-a02n01: Old event | magic=0:0;2:2:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.4 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 2 action 2 (apc_snmp_node2_an-pdu03_stop_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=89 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation apc_snmp_node2_an-pdu04_start_0 'create' on an-a02n01: Old event | magic=0:0;20:110:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.4 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 110 action 20 (apc_snmp_node2_an-pdu04_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=132 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation apc_snmp_node2_an-pdu04_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;14:111:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.4 source=process_graph_event:408 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 111 action 14 (apc_snmp_node2_an-pdu04_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=137 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv20-fileprocessor_start_0 'create' on an-a02n01: Old event | magic=0:0;16:6:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.4 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 6 action 16 (srv20-fileprocessor_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=96 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv20-fileprocessor_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;18:7:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.4 source=process_graph_event:408 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 7 action 18 (srv20-fileprocessor_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=97 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv23-orderprocessing_start_0 'create' on an-a02n01: Old event | magic=0:0;25:9:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.4 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 9 action 25 (srv23-orderprocessing_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=102 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv23-orderprocessing_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;27:10:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.4 source=process_graph_event:408 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 10 action 27 (srv23-orderprocessing_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=103 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by operation srv16-machine-interface_monitor_0 'create' on an-a02n01: Old event | magic=0:7;12:0:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.351.4 source=process_graph_event:412 complete=false Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 0 action 12 (srv16-machine-interface_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=49 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/564, version=0.351.4) Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.351.4 2 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.351.5 (null) Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/565, version=0.351.5) Jul 23 16:23:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/566, version=0.351.5) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.351.5 2 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.351.6 (null) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=39:115:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=0:1;39:115:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=119, @rc-code= Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/107, version=0.351.7) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 107 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.351.7 2 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.351.8 (null) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/108, version=0.351.8) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 108 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=1627082602 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627082616 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by status-2-fail-count-apc_snmp_node1_an-pdu03.start_0 doing create fail-count-apc_snmp_node1_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.351.7 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 115 aborted by status-2-last-failure-apc_snmp_node1_an-pdu03.start_0 doing create last-failure-apc_snmp_node1_an-pdu03#start_0=1627082616: Transient attribute change | cib=0.351.8 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:23:09 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:23:22 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:23:22 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for ipmilan_node2 on an-a02n02 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for ipmilan_node1 on an-a02n01 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n02 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node2_an-pdu03 ( an-a02n02 ) due to node availability Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Started an-a02n01) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Move ipmilan_node2 ( an-a02n01 -> an-a02n02 ) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Move ipmilan_node1 ( an-a02n02 -> an-a02n01 ) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu04 ( an-a02n02 ) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 116, saving inputs in /var/lib/pacemaker/pengine/pe-input-137.bz2 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627082616-367 is obsolete Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:23:09 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:23:22 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:23:22 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for ipmilan_node2 on an-a02n02 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for ipmilan_node1 on an-a02n01 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n02 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node2_an-pdu03 ( an-a02n02 ) due to node availability Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Started an-a02n01) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Move ipmilan_node2 ( an-a02n01 -> an-a02n02 ) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Move ipmilan_node1 ( an-a02n02 -> an-a02n01 ) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu03 ( an-a02n02 ) due to node availability Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu04 ( an-a02n02 ) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 117, saving inputs in /var/lib/pacemaker/pengine/pe-input-138.bz2 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 117 (ref=pe_calc-dc-1627082616-368) derived from /var/lib/pacemaker/pengine/pe-input-138.bz2 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node2_an-pdu03_stop_0 on an-a02n02 | action 14 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation ipmilan_node2_stop_0 locally on an-a02n01 | action 17 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for ipmilan_node2 on an-a02n01 | transition_key=17:117:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=ipmilan_node2_stop_0 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:ipmilan_node2 action:stop call_id:143 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation ipmilan_node1_stop_0 on an-a02n02 | action 22 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 on an-a02n02 | action 11 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu04_start_0 on an-a02n02 | action 39 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: ipmilan_node2 stop (call 143) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/571) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) info: Result of monitor operation for ipmilan_node2 on an-a02n01: Cancelled | call=131 key=ipmilan_node2_monitor_60000 confirmed=true Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for ipmilan_node2 on an-a02n01: ok | rc=0 call=143 key=ipmilan_node2_stop_0 confirmed=true cib-update=572 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.351.8 2 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.351.9 (null) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ipmilan_node2']/lrm_rsc_op[@id='ipmilan_node2_last_0']: @operation_key=ipmilan_node2_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=17:117:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;17:117:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-c Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/571, version=0.351.9) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/572) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.351.9 2 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.351.10 (null) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ipmilan_node2']/lrm_rsc_op[@id='ipmilan_node2_last_0']: @transition-magic=0:0;17:117:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=143, @rc-code=0, @op-status=0 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/572, version=0.351.10) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 117 action 17 (ipmilan_node2_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=143 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation ipmilan_node2_start_0 on an-a02n02 | action 18 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.351.10 2 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.351.11 (null) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @operation_key=apc_snmp_node2_an-pdu03_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=14:117:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;14:117:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code= Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/132, version=0.351.11) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.351.11 2 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.351.12 (null) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ipmilan_node1']/lrm_rsc_op[@id='ipmilan_node1_last_0']: @operation_key=ipmilan_node1_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=22:117:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;22:117:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-c Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/133, version=0.351.12) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.351.12 2 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.351.13 (null) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:0;14:117:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=121, @rc-code=0, @op-status=0 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/134, version=0.351.13) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 117 action 14 (apc_snmp_node2_an-pdu03_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=121 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.351.13 2 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.351.14 (null) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ipmilan_node1']/lrm_rsc_op[@id='ipmilan_node1_last_0']: @transition-magic=0:0;22:117:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=123, @rc-code=0, @op-status=0 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/135, version=0.351.14) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 117 action 22 (ipmilan_node1_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=123 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation ipmilan_node1_start_0 locally on an-a02n01 | action 23 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for ipmilan_node1 on an-a02n01 | transition_key=23:117:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=ipmilan_node1_start_0 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:ipmilan_node1 action:start call_id:144 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.351.14 2 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.351.15 (null) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=15 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @transition-key=11:117:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;11:117:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/136, version=0.351.15) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.351.15 2 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.351.16 (null) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=16 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=39:117:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;39:117:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/137, version=0.351.16) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.351.16 2 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.351.17 (null) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=17 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;11:117:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=124, @rc-code=0, @op-status=0 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/138, version=0.351.17) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 117 action 11 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=124 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.351.17 2 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.351.18 (null) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=18 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ipmilan_node2']/lrm_rsc_op[@id='ipmilan_node2_last_0']: @operation_key=ipmilan_node2_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=18:117:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;18:117:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/139, version=0.351.18) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/573) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.351.18 2 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.351.19 (null) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=19 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ipmilan_node1']/lrm_rsc_op[@id='ipmilan_node1_last_0']: @operation_key=ipmilan_node1_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=23:117:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;23:117:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/573, version=0.351.19) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: ipmilan_node1 start (call 144) exited with status 0 (execution time 280ms, queue time 0ms) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.351.19 2 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.351.20 (null) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=20 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ipmilan_node2']/lrm_rsc_op[@id='ipmilan_node2_last_0']: @transition-magic=0:0;18:117:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=126, @rc-code=0, @op-status=0, @exec-time=281 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4160197] exited with status 0 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/140, version=0.351.20) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/574) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for ipmilan_node1 on an-a02n01: ok | rc=0 call=144 key=ipmilan_node1_start_0 confirmed=true cib-update=574 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.351.20 2 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.351.21 (null) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=21 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ipmilan_node1']/lrm_rsc_op[@id='ipmilan_node1_last_0']: @transition-magic=0:0;23:117:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=144, @rc-code=0, @op-status=0, @exec-time=280 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 117 action 18 (ipmilan_node2_start_0 on an-a02n02) confirmed: ok | rc=0 call-id=126 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation ipmilan_node2_monitor_60000 on an-a02n02 | action 19 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/574, version=0.351.21) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 117 action 23 (ipmilan_node1_start_0 on an-a02n01) confirmed: ok | rc=0 call-id=144 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation ipmilan_node1_monitor_60000 locally on an-a02n01 | action 24 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of monitor operation for ipmilan_node1 on an-a02n01 | transition_key=24:117:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=ipmilan_node1_monitor_60000 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/575) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.351.21 2 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.351.22 (null) Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=22 Jul 23 16:23:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ipmilan_node1']: Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/109, version=0.351.27) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 109 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.351.27 2 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.351.28 (null) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=28 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/110, version=0.351.28) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 110 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627082629 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 117 aborted by status-2-fail-count-apc_snmp_node1_an-pdu04.start_0 doing create fail-count-apc_snmp_node1_an-pdu04#start_0=INFINITY: Transient attribute change | cib=0.351.27 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 117 aborted by status-2-last-failure-apc_snmp_node1_an-pdu04.start_0 doing create last-failure-apc_snmp_node1_an-pdu04#start_0=1627082629: Transient attribute change | cib=0.351.28 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:23:09 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:23:36 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:23:36 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:23:22 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n02 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Started an-a02n01) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n02) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu04 ( an-a02n02 ) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 118, saving inputs in /var/lib/pacemaker/pengine/pe-input-139.bz2 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627082629-378 is obsolete Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:23:09 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:23:36 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:23:36 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:23:22 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n01 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Started an-a02n01) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n02) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu04 ( an-a02n02 -> an-a02n01 ) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 119, saving inputs in /var/lib/pacemaker/pengine/pe-input-140.bz2 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 119 (ref=pe_calc-dc-1627082629-379) derived from /var/lib/pacemaker/pengine/pe-input-140.bz2 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu04_stop_0 on an-a02n02 | action 11 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.351.28 2 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.351.29 (null) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=29 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_stop_0, @operation=stop, @transition-key=11:119:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;11:119:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/144, version=0.351.29) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.351.29 2 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.351.30 (null) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=30 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:0;11:119:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=128, @rc-code=0, @op-status=0 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/145, version=0.351.30) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 119 action 11 (apc_snmp_node1_an-pdu04_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=128 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu04_start_0 locally on an-a02n01 | action 35 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node1_an-pdu04 on an-a02n01 | transition_key=35:119:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu04_start_0 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/580) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu04 action:start call_id:146 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.351.30 2 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.351.31 (null) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=31 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=35:119:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;35:119:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/580, version=0.351.31) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.351.31 2 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.352.0 (null) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu03']/meta_attributes[@id='apc_snmp_node2_an-pdu03-meta_attributes']/nvpair[@id='apc_snmp_node2_an-pdu03-meta_attributes-target-role'] Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=352, @num_updates=0 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.352.0) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 119 aborted by deletion of nvpair[@id='apc_snmp_node2_an-pdu03-meta_attributes-target-role']: Configuration change | cib=0.352.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu03']/meta_attributes[@id='apc_snmp_node2_an-pdu03-meta_attributes']/nvpair[@id='apc_snmp_node2_an-pdu03-meta_attributes-target-role'] complete=false Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: (meta) attribute deleted from resource Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.352.0 Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node2_an-pdu03' to device list (8 active devices) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-46.raw Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.352.0 of the CIB to disk (digest: 034e1af9317bc5065394204374d1e37d) Jul 23 16:23:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.O9VZWx (digest: /var/lib/pacemaker/cib/cib.e0DYvA) Jul 23 16:23:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 0102f40f964e86fc343ccb59bd571c29 for 0.352.0 (0x5619760950e0 0) Jul 23 16:23:55 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4160914] error output [ 2021-07-23 16:23:55,695 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 16:23:55 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4160914] error output [ ] Jul 23 16:23:55 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4160914] error output [ ] Jul 23 16:23:55 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4160914] error output [ 2021-07-23 16:23:55,696 ERROR: Please use '-h' for usage ] Jul 23 16:23:55 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4160914] error output [ ] Jul 23 16:23:55 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4160914] stderr: [ 2021-07-23 16:23:55,695 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 16:23:55 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4160914] stderr: [ ] Jul 23 16:23:55 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4160914] stderr: [ ] Jul 23 16:23:55 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4160914] stderr: [ 2021-07-23 16:23:55,696 ERROR: Please use '-h' for usage ] Jul 23 16:23:55 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4160914] stderr: [ ] Jul 23 16:23:55 an-a02n01.alteeve.com pacemaker-fenced [121629] (internal_stonith_action_execute) info: Attempt 2 to execute fence_apc_snmp (monitor). remaining timeout is 14 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4161223] error output [ 2021-07-23 16:24:02,856 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4161223] error output [ ] Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4161223] error output [ ] Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4161223] error output [ 2021-07-23 16:24:02,856 ERROR: Please use '-h' for usage ] Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4161223] error output [ ] Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4161223] stderr: [ 2021-07-23 16:24:02,856 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4161223] stderr: [ ] Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4161223] stderr: [ ] Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4161223] stderr: [ 2021-07-23 16:24:02,856 ERROR: Please use '-h' for usage ] Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4161223] stderr: [ ] Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_remaining_timeout) info: Attempted to execute agent fence_apc_snmp (monitor) the maximum number of times (2) allowed Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_operation) notice: Operation 'monitor' [4161223] using apc_snmp_node1_an-pdu04 returned -201 (Generic Pacemaker error) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu04 start (call 146) exited with status 1 (execution time 13323ms, queue time 0ms) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for apc_snmp_node1_an-pdu04 on an-a02n01: error | rc=1 call=146 key=apc_snmp_node1_an-pdu04_start_0 confirmed=true cib-update=581 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/581) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.352.0 2 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.352.1 (null) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:1;35:119:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=146, @rc-code=1, @op-status=0, @exec-time=13323 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/111, version=0.352.2) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 111 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.352.2 2 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.352.3 (null) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/112, version=0.352.3) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 112 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=1627082642 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627082629 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 119 aborted by status-1-fail-count-apc_snmp_node1_an-pdu04.start_0 doing create fail-count-apc_snmp_node1_an-pdu04#start_0=INFINITY: Transient attribute change | cib=0.352.2 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 119 aborted by status-1-last-failure-apc_snmp_node1_an-pdu04.start_0 doing create last-failure-apc_snmp_node1_an-pdu04#start_0=1627082642: Transient attribute change | cib=0.352.3 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 16:23:49 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 16:23:49 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:23:09 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:23:36 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:23:22 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu03 on an-a02n01 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu04 on an-a02n02 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for delay_node2 on an-a02n01 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for delay_node1 on an-a02n02 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n01 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node2_an-pdu03 ( an-a02n01 ) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Move apc_snmp_node2_an-pdu04 ( an-a02n01 -> an-a02n02 ) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n02) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Move delay_node2 ( an-a02n02 -> an-a02n01 ) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Move delay_node1 ( an-a02n01 -> an-a02n02 ) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu04 ( an-a02n01 ) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 120, saving inputs in /var/lib/pacemaker/pengine/pe-input-141.bz2 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627082642-382 is obsolete Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 16:23:49 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 16:23:49 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:23:09 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:23:36 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:23:22 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu03 on an-a02n01 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu04 on an-a02n02 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for delay_node2 on an-a02n01 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for delay_node1 on an-a02n02 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node2_an-pdu03 ( an-a02n01 ) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Move apc_snmp_node2_an-pdu04 ( an-a02n01 -> an-a02n02 ) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n02) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Move delay_node2 ( an-a02n02 -> an-a02n01 ) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Move delay_node1 ( an-a02n01 -> an-a02n02 ) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu04 ( an-a02n01 ) due to node availability Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 121, saving inputs in /var/lib/pacemaker/pengine/pe-input-142.bz2 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 121 (ref=pe_calc-dc-1627082642-383) derived from /var/lib/pacemaker/pengine/pe-input-142.bz2 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node2_an-pdu03_start_0 locally on an-a02n01 | action 13 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node2_an-pdu03 on an-a02n01 | transition_key=13:121:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu03_start_0 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node2_an-pdu04_stop_0 locally on an-a02n01 | action 15 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node2_an-pdu03 action:start call_id:147 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/585) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node2_an-pdu04 on an-a02n01 | transition_key=15:121:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu04_stop_0 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node2_an-pdu04 action:stop call_id:149 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation delay_node2_stop_0 on an-a02n02 | action 20 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation delay_node1_stop_0 locally on an-a02n01 | action 25 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.352.3 2 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.352.4 (null) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @operation_key=apc_snmp_node2_an-pdu03_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=13:121:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;13:121:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/585, version=0.352.4) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/586) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.352.4 2 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.352.5 (null) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @operation_key=apc_snmp_node2_an-pdu04_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=15:121:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;15:121:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code= Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/586, version=0.352.5) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.352.5 2 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.352.6 (null) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='delay_node2']/lrm_rsc_op[@id='delay_node2_last_0']: @operation_key=delay_node2_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=20:121:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;20:121:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change= Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/146, version=0.352.6) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node2_an-pdu04 stop (call 149) exited with status 0 (execution time 2ms, queue time 0ms) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.352.6 2 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.352.7 (null) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='delay_node2']/lrm_rsc_op[@id='delay_node2_last_0']: @transition-magic=0:0;20:121:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=130, @rc-code=0, @op-status=0 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for delay_node1 on an-a02n01 | transition_key=25:121:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=delay_node1_stop_0 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/147, version=0.352.7) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/587) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:delay_node1 action:stop call_id:151 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu04_stop_0 locally on an-a02n01 | action 2 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node1_an-pdu04 on an-a02n01 | transition_key=2:121:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu04_stop_0 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/588) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: delay_node1 stop (call 151) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu04 action:stop call_id:152 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.352.7 2 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.352.8 (null) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='delay_node1']/lrm_rsc_op[@id='delay_node1_last_0']: @operation_key=delay_node1_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=25:121:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;25:121:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change= Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu04 stop (call 152) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/587, version=0.352.8) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 121 action 20 (delay_node2_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=130 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) info: Result of monitor operation for apc_snmp_node2_an-pdu04 on an-a02n01: Cancelled | call=137 key=apc_snmp_node2_an-pdu04_monitor_60000 confirmed=true Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node2_an-pdu04 on an-a02n01: ok | rc=0 call=149 key=apc_snmp_node2_an-pdu04_stop_0 confirmed=true cib-update=589 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.352.8 2 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.352.9 (null) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_stop_0, @operation=stop, @transition-key=2:121:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;2:121:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1627 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) info: Result of monitor operation for delay_node1 on an-a02n01: Cancelled | call=139 key=delay_node1_monitor_60000 confirmed=true Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for delay_node1 on an-a02n01: ok | rc=0 call=151 key=delay_node1_stop_0 confirmed=true cib-update=590 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/588, version=0.352.9) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node1_an-pdu04 on an-a02n01: ok | rc=0 call=152 key=apc_snmp_node1_an-pdu04_stop_0 confirmed=true cib-update=591 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/589) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/590) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/591) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation delay_node2_start_0 locally on an-a02n01 | action 21 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for delay_node2 on an-a02n01 | transition_key=21:121:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=delay_node2_start_0 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/592) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:delay_node2 action:start call_id:153 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.352.9 2 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.352.10 (null) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:0;15:121:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=149, @rc-code=0, @op-status=0, @exec-time=2 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/589, version=0.352.10) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 121 action 15 (apc_snmp_node2_an-pdu04_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=149 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node2_an-pdu04_start_0 on an-a02n02 | action 16 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.352.10 2 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.352.11 (null) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='delay_node1']/lrm_rsc_op[@id='delay_node1_last_0']: @transition-magic=0:0;25:121:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=151, @rc-code=0, @op-status=0 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/590, version=0.352.11) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 121 action 25 (delay_node1_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=151 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation delay_node1_start_0 on an-a02n02 | action 26 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.352.11 2 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.352.12 (null) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:0;2:121:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=152, @rc-code=0, @op-status=0 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/591, version=0.352.12) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 121 action 2 (apc_snmp_node1_an-pdu04_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=152 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.352.12 2 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.352.13 (null) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='delay_node2']/lrm_rsc_op[@id='delay_node2_last_0']: @operation_key=delay_node2_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=21:121:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;21:121:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-chang Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/592, version=0.352.13) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.352.13 2 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.352.14 (null) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @operation_key=apc_snmp_node2_an-pdu04_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=16:121:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;16:121:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/148, version=0.352.14) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.352.14 2 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.352.15 (null) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=15 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='delay_node1']/lrm_rsc_op[@id='delay_node1_last_0']: @operation_key=delay_node1_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=26:121:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;26:121:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-chang Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/149, version=0.352.15) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: delay_node2 start (call 153) exited with status 0 (execution time 8ms, queue time 0ms) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4161823] exited with status 0 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for delay_node2 on an-a02n01: ok | rc=0 call=153 key=delay_node2_start_0 confirmed=true cib-update=593 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/593) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.352.15 2 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.352.16 (null) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=16 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='delay_node2']/lrm_rsc_op[@id='delay_node2_last_0']: @transition-magic=0:0;21:121:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=153, @rc-code=0, @op-status=0, @exec-time=8 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/593, version=0.352.16) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 121 action 21 (delay_node2_start_0 on an-a02n01) confirmed: ok | rc=0 call-id=153 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation delay_node2_monitor_60000 locally on an-a02n01 | action 22 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of monitor operation for delay_node2 on an-a02n01 | transition_key=22:121:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=delay_node2_monitor_60000 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/594) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.352.16 2 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.352.17 (null) Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=17 Jul 23 16:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='delay_node2']: Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.353.0) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 121 aborted by apc_snmp_node2_an-pdu03-meta_attributes-target-role doing create target-role=Stopped: Configuration change | cib=0.353.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu03']/meta_attributes[@id='apc_snmp_node2_an-pdu03-meta_attributes'] complete=true Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create meta_attributes[@id='apc_snmp_node2_an-pdu03-meta_attributes'] Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.353.0 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-47.raw Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.353.0 of the CIB to disk (digest: 8e79c756a2f37c403017e41ffdcaf699) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.eYUeRx (digest: /var/lib/pacemaker/cib/cib.kzYTmA) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 16:23:49 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:23:09 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:23:36 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:23:22 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for ipmilan_node2 on an-a02n01 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for ipmilan_node1 on an-a02n02 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node2_an-pdu03 ( an-a02n01 ) due to node availability Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Started an-a02n02) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Move ipmilan_node2 ( an-a02n02 -> an-a02n01 ) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n01) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Move ipmilan_node1 ( an-a02n01 -> an-a02n02 ) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n02) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 122, saving inputs in /var/lib/pacemaker/pengine/pe-input-143.bz2 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 122 (ref=pe_calc-dc-1627082685-396) derived from /var/lib/pacemaker/pengine/pe-input-143.bz2 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node2_an-pdu03_stop_0 locally on an-a02n01 | action 13 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node2_an-pdu03 on an-a02n01 | transition_key=13:122:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu03_stop_0 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node2_an-pdu03 action:stop call_id:157 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation ipmilan_node2_stop_0 on an-a02n02 | action 16 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation ipmilan_node1_stop_0 locally on an-a02n01 | action 21 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/600) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Device 'apc_snmp_node2_an-pdu03' not found (7 active devices) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node2_an-pdu03 stop (call 157) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for ipmilan_node1 on an-a02n01 | transition_key=21:122:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=ipmilan_node1_stop_0 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/601) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:ipmilan_node1 action:stop call_id:159 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) info: Result of monitor operation for apc_snmp_node2_an-pdu03 on an-a02n01: Cancelled | call=155 key=apc_snmp_node2_an-pdu03_monitor_60000 confirmed=true Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: ipmilan_node1 stop (call 159) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node2_an-pdu03 on an-a02n01: ok | rc=0 call=157 key=apc_snmp_node2_an-pdu03_stop_0 confirmed=true cib-update=602 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) info: Result of monitor operation for ipmilan_node1 on an-a02n01: Cancelled | call=145 key=ipmilan_node1_monitor_60000 confirmed=true Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for ipmilan_node1 on an-a02n01: ok | rc=0 call=159 key=ipmilan_node1_stop_0 confirmed=true cib-update=603 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.353.0 2 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.353.1 (null) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @operation_key=apc_snmp_node2_an-pdu03_stop_0, @operation=stop, @transition-key=13:122:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;13:122:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/600, version=0.353.1) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.353.1 2 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.353.2 (null) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ipmilan_node1']/lrm_rsc_op[@id='ipmilan_node1_last_0']: @operation_key=ipmilan_node1_stop_0, @operation=stop, @transition-key=21:122:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;21:122:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1627082685, @exec-time=0 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/601, version=0.353.2) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/602) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/603) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.353.2 2 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.353.3 (null) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:0;13:122:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=157, @rc-code=0, @op-status=0 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/602, version=0.353.3) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 122 action 13 (apc_snmp_node2_an-pdu03_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=157 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.353.3 2 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.353.4 (null) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ipmilan_node1']/lrm_rsc_op[@id='ipmilan_node1_last_0']: @transition-magic=0:0;21:122:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=159, @rc-code=0, @op-status=0 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/603, version=0.353.4) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 122 action 21 (ipmilan_node1_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=159 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation ipmilan_node1_start_0 on an-a02n02 | action 22 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.353.4 2 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.353.5 (null) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ipmilan_node2']/lrm_rsc_op[@id='ipmilan_node2_last_0']: @operation_key=ipmilan_node2_stop_0, @operation=stop, @transition-key=16:122:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;16:122:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1627082685, @exec-time=0 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/156, version=0.353.5) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.353.5 2 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.353.6 (null) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ipmilan_node2']/lrm_rsc_op[@id='ipmilan_node2_last_0']: @transition-magic=0:0;16:122:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=136, @rc-code=0, @op-status=0 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/157, version=0.353.6) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 122 action 16 (ipmilan_node2_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=136 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation ipmilan_node2_start_0 locally on an-a02n01 | action 17 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for ipmilan_node2 on an-a02n01 | transition_key=17:122:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=ipmilan_node2_start_0 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.353.6 2 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.353.7 (null) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ipmilan_node1']/lrm_rsc_op[@id='ipmilan_node1_last_0']: @operation_key=ipmilan_node1_start_0, @operation=start, @transition-key=22:122:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;22:122:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1627082685 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/158, version=0.353.7) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:ipmilan_node2 action:start call_id:160 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/604) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.353.7 2 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.353.8 (null) Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ipmilan_node2']/lrm_rsc_op[@id='ipmilan_node2_last_0']: @operation_key=ipmilan_node2_start_0, @operation=start, @transition-key=17:122:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;17:122:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1627082685 Jul 23 16:24:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/604, version=0.353.8) Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: ipmilan_node2 start (call 160) exited with status 0 (execution time 271ms, queue time 0ms) Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4163911] exited with status 0 Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/605) Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for ipmilan_node2 on an-a02n01: ok | rc=0 call=160 key=ipmilan_node2_start_0 confirmed=true cib-update=605 Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.353.8 2 Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.353.9 (null) Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ipmilan_node2']/lrm_rsc_op[@id='ipmilan_node2_last_0']: @transition-magic=0:0;17:122:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=160, @rc-code=0, @op-status=0, @exec-time=271 Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/605, version=0.353.9) Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 122 action 17 (ipmilan_node2_start_0 on an-a02n01) confirmed: ok | rc=0 call-id=160 Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation ipmilan_node2_monitor_60000 locally on an-a02n01 | action 18 Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of monitor operation for ipmilan_node2 on an-a02n01 | transition_key=18:122:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=ipmilan_node2_monitor_60000 Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/606) Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.353.9 2 Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.353.10 (null) Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ipmilan_node2']/lrm_rsc_op[@id='ipmilan_node2_monitor_60000']: @crm-debug-origin=do_update_resource, @transition-key=18:122:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;18:122:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1627082686, @exec-time=0 Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/606, version=0.353.10) Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.353.10 2 Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.353.11 (null) Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ipmilan_node1']/lrm_rsc_op[@id='ipmilan_node1_last_0']: @transition-magic=0:0;22:122:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=137, @rc-code=0, @op-status=0, @exec-time=278 Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/159, version=0.353.11) Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 122 action 22 (ipmilan_node1_start_0 on an-a02n02) confirmed: ok | rc=0 call-id=137 Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation ipmilan_node1_monitor_60000 on an-a02n02 | action 23 Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.353.11 2 Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.353.12 (null) Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ipmilan_node1']/lrm_rsc_op[@id='ipmilan_node1_monitor_60000']: @crm-debug-origin=do_update_resource, @transition-key=23:122:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;23:122:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1627082686, @exec-time=0 Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/160, version=0.353.12) Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of monitor operation for ipmilan_node2 on an-a02n01: ok | rc=0 call=161 key=ipmilan_node2_monitor_60000 confirmed=false cib-update=607 Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/607) Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.353.12 2 Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.353.13 (null) Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ipmilan_node2']/lrm_rsc_op[@id='ipmilan_node2_monitor_60000']: @transition-magic=0:0;18:122:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=161, @rc-code=0, @op-status=0, @exec-time=594 Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/607, version=0.353.13) Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 122 action 18 (ipmilan_node2_monitor_60000 on an-a02n01) confirmed: ok | rc=0 call-id=161 Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.353.13 2 Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.353.14 (null) Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ipmilan_node1']/lrm_rsc_op[@id='ipmilan_node1_monitor_60000']: @transition-magic=0:0;23:122:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=138, @rc-code=0, @op-status=0, @exec-time=590 Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/161, version=0.353.14) Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 122 action 23 (ipmilan_node1_monitor_60000 on an-a02n02) confirmed: ok | rc=0 call-id=138 Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 122 (Complete=7, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-143.bz2): Complete Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 16:24:46 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.353.14 2 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.354.0 (null) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=354, @num_updates=0 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-2']: @devices=apc_snmp_node2_an-pdu04 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.353.14 -> 0.354.0 from an-a02n02 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.354.0) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 122 aborted by fencing-level.fl-an-a02n02-2 'modify': Configuration change | cib=0.354.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-2'] complete=true Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 113 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 114 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 115 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 116 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 117 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 118 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 119 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 120 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 2 from fencing topology for an-a02n02 (2 active levels remaining) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/610) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/611) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/113) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-48.raw Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/114) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/115) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/116) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/117) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.354.0 of the CIB to disk (digest: 8e7d337e7bd15ea4a9d6d4f0eb233672) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.evR838 (digest: /var/lib/pacemaker/cib/cib.HzVcn8) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/610, version=0.354.0) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.354.0 2 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.354.1 (null) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/611, version=0.354.1) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.354.1 2 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.354.2 (null) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/113, version=0.354.2) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 113 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.354.2 2 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.354.3 (null) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/114, version=0.354.3) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 114 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/115, version=0.354.3) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 115 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/116, version=0.354.3) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 116 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=1627082602 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627082616 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.354.3 2 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.354.4 (null) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/117, version=0.354.4) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 117 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/118) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/119) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/120) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.354.4 2 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.354.5 (null) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/118, version=0.354.5) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 118 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/119, version=0.354.5) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 119 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/120, version=0.354.5) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 120 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=1627082642 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627082629 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/613, version=0.354.5) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/614) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/614, version=0.354.5) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/616) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/616, version=0.354.5) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/618) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-12 offer to an-a02n02 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-12 offer to an-a02n01 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-12 requests from 2 outstanding nodes Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/618, version=0.354.5) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/620) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/620, version=0.354.5) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: ee44214fbca50d14070cc0f088da35b2 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.354.5 with 0.354.5 from an-a02n01 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/623, version=0.354.5) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/624) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/625) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-49.raw Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/624, version=0.354.5) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.354.0 of the CIB to disk (digest: 8e7d337e7bd15ea4a9d6d4f0eb233672) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.y4Cinh (digest: /var/lib/pacemaker/cib/cib.PrzsOh) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/625, version=0.354.5) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4163919] exited with status 0 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.354.5 2 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.355.0 (null) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu03'] Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=355, @num_updates=0 Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //primitive[@id='apc_snmp_node2_an-pdu03']: OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.355.0) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Device 'apc_snmp_node2_an-pdu03' not found (7 active devices) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-50.raw Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.355.0 of the CIB to disk (digest: d3b0f83350564e995bb845811b106381) Jul 23 16:24:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.Wh3wSt (digest: /var/lib/pacemaker/cib/cib.oOcrMs) Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4163921] exited with status 0 Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.355.0 2 Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.356.0 (null) Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=356 Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources: Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.356.0) Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create resources Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.356.0 Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node2_an-pdu03' to device list (8 active devices) Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-51.raw Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.356.0 of the CIB to disk (digest: 1875c02acd266077dc4a8a91e160ad0f) Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.QTcw3z (digest: /var/lib/pacemaker/cib/cib.yRJDCz) Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4163923] exited with status 0 Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.356.0 2 Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.357.0 (null) Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=357 Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu04']: Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.357.0) Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create primitive[@id='apc_snmp_node2_an-pdu04'] Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.357.0 Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-52.raw Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.357.0 of the CIB to disk (digest: 3f945bd70d452313c0393344f22ab803) Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.TiMmFK (digest: /var/lib/pacemaker/cib/cib.ujNomI) Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4163925] exited with status 0 Jul 23 16:24:48 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4163927] exited with status 0 Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4163929] exited with status 0 Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4163930] exited with status 0 Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4163931] exited with status 0 Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 626) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/626) Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/627) Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 628) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.357.0 2 Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.357.1 (null) Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/626, version=0.357.1) Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.357.1 2 Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.357.2 (null) Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/627, version=0.357.2) Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/628) Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.357.2 2 Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.357.3 (null) Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/628, version=0.357.3) Jul 23 16:24:49 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4163932] exited with status 0 Jul 23 16:24:50 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4163933] exited with status 0 Jul 23 16:24:50 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4163934] exited with status 0 Jul 23 16:24:50 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4163941] exited with status 0 Jul 23 16:24:50 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4163942] exited with status 0 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4163944] exited with status 0 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4163945] exited with status 0 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4163946] exited with status 0 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/629) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 122 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.357.3 2 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.357.4 (null) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/629, version=0.357.4) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/632) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/633) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/634) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 122 aborted: History refresh | source=process_resource_updates:268 complete=true Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/632, version=0.357.4) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.357.4 2 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.357.5 (null) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/633, version=0.357.5) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/634, version=0.357.5) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 16:23:49 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:23:09 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:23:36 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:23:22 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for delay_node2 on an-a02n02 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for delay_node1 on an-a02n01 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu03 on an-a02n02 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node2_an-pdu04 ( an-a02n02 ) due to node availability Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Move delay_node2 ( an-a02n01 -> an-a02n02 ) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Move delay_node1 ( an-a02n02 -> an-a02n01 ) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node2_an-pdu03 ( an-a02n02 ) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 123, saving inputs in /var/lib/pacemaker/pengine/pe-input-144.bz2 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 123 (ref=pe_calc-dc-1627082691-411) derived from /var/lib/pacemaker/pengine/pe-input-144.bz2 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node2_an-pdu04_stop_0 on an-a02n02 | action 12 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation delay_node2_stop_0 locally on an-a02n01 | action 15 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for delay_node2 on an-a02n01 | transition_key=15:123:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=delay_node2_stop_0 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:delay_node2 action:stop call_id:163 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation delay_node1_stop_0 on an-a02n02 | action 20 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node2_an-pdu03_start_0 on an-a02n02 | action 35 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: delay_node2 stop (call 163) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) info: Result of monitor operation for delay_node2 on an-a02n01: Cancelled | call=154 key=delay_node2_monitor_60000 confirmed=true Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for delay_node2 on an-a02n01: ok | rc=0 call=163 key=delay_node2_stop_0 confirmed=true cib-update=638 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/637) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/638) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.357.5 2 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.357.6 (null) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='delay_node2']/lrm_rsc_op[@id='delay_node2_last_0']: @operation_key=delay_node2_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=15:123:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;15:123:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change= Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/637, version=0.357.6) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.357.6 2 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.357.7 (null) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='delay_node2']/lrm_rsc_op[@id='delay_node2_last_0']: @transition-magic=0:0;15:123:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=163, @rc-code=0, @op-status=0 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/638, version=0.357.7) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 123 action 15 (delay_node2_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=163 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation delay_node2_start_0 on an-a02n02 | action 16 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.357.7 2 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.357.8 (null) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @operation_key=apc_snmp_node2_an-pdu04_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=12:123:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;12:123:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code= Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/164, version=0.357.8) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.357.8 2 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.357.9 (null) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='delay_node1']/lrm_rsc_op[@id='delay_node1_last_0']: @operation_key=delay_node1_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=20:123:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;20:123:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change= Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/165, version=0.357.9) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.357.9 2 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.357.10 (null) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:0;12:123:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=140, @rc-code=0, @op-status=0 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/166, version=0.357.10) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 123 action 12 (apc_snmp_node2_an-pdu04_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=140 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.357.10 2 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.357.11 (null) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='delay_node1']/lrm_rsc_op[@id='delay_node1_last_0']: @transition-magic=0:0;20:123:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=142, @rc-code=0, @op-status=0 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/167, version=0.357.11) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 123 action 20 (delay_node1_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=142 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation delay_node1_start_0 locally on an-a02n01 | action 21 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for delay_node1 on an-a02n01 | transition_key=21:123:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=delay_node1_start_0 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:delay_node1 action:start call_id:164 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.357.11 2 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.357.12 (null) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @operation_key=apc_snmp_node2_an-pdu03_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=35:123:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;35:123:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/168, version=0.357.12) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/639) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.357.12 2 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.357.13 (null) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='delay_node1']/lrm_rsc_op[@id='delay_node1_last_0']: @operation_key=delay_node1_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=21:123:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;21:123:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-chang Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/639, version=0.357.13) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.357.13 2 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.357.14 (null) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='delay_node2']/lrm_rsc_op[@id='delay_node2_last_0']: @operation_key=delay_node2_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=16:123:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;16:123:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-chang Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/169, version=0.357.14) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: delay_node1 start (call 164) exited with status 0 (execution time 9ms, queue time 0ms) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4163948] exited with status 0 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for delay_node1 on an-a02n01: ok | rc=0 call=164 key=delay_node1_start_0 confirmed=true cib-update=640 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/640) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.357.14 2 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.357.15 (null) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=15 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='delay_node1']/lrm_rsc_op[@id='delay_node1_last_0']: @transition-magic=0:0;21:123:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=164, @rc-code=0, @op-status=0, @exec-time=9 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/640, version=0.357.15) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 123 action 21 (delay_node1_start_0 on an-a02n01) confirmed: ok | rc=0 call-id=164 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation delay_node1_monitor_60000 locally on an-a02n01 | action 22 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of monitor operation for delay_node1 on an-a02n01 | transition_key=22:123:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=delay_node1_monitor_60000 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/641) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.357.15 2 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.357.16 (null) Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=16 Jul 23 16:24:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='delay_node1']: Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/122, version=0.357.22) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 122 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.357.22 2 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.357.23 (null) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=23 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/123, version=0.357.23) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 123 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=1627082705 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 123 aborted by status-2-fail-count-apc_snmp_node2_an-pdu03.start_0 doing create fail-count-apc_snmp_node2_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.357.22 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 123 aborted by status-2-last-failure-apc_snmp_node2_an-pdu03.start_0 doing create last-failure-apc_snmp_node2_an-pdu03#start_0=1627082705: Transient attribute change | cib=0.357.23 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 16:23:49 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:23:09 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:23:36 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:23:22 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:24:51 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:24:51 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_0 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu03 on an-a02n02 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node2_an-pdu03 ( an-a02n02 ) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 124, saving inputs in /var/lib/pacemaker/pengine/pe-input-145.bz2 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627082705-420 is obsolete Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 16:23:49 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:23:09 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:23:36 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:23:22 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:24:51 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:24:51 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_0 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu03 on an-a02n01 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node2_an-pdu03 ( an-a02n02 -> an-a02n01 ) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 125, saving inputs in /var/lib/pacemaker/pengine/pe-input-146.bz2 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 125 (ref=pe_calc-dc-1627082705-421) derived from /var/lib/pacemaker/pengine/pe-input-146.bz2 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node2_an-pdu03_stop_0 on an-a02n02 | action 11 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.357.23 2 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.357.24 (null) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=24 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @operation_key=apc_snmp_node2_an-pdu03_stop_0, @operation=stop, @transition-key=11:125:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;11:125:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/174, version=0.357.24) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.357.24 2 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.357.25 (null) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=25 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:0;11:125:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=146, @rc-code=0, @op-status=0 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/175, version=0.357.25) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 125 action 11 (apc_snmp_node2_an-pdu03_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=146 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node2_an-pdu03_start_0 locally on an-a02n01 | action 32 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node2_an-pdu03 on an-a02n01 | transition_key=32:125:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu03_start_0 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/646) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node2_an-pdu03 action:start call_id:166 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.357.25 2 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.357.26 (null) Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=26 Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @operation_key=apc_snmp_node2_an-pdu03_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=32:125:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;32:125:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:25:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/646, version=0.357.26) Jul 23 16:25:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: a41f2bd77db91a7ee32191e48bd10191 for 0.357.26 (0x56197608b400 0) Jul 23 16:25:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4163973] error output [ 2021-07-23 16:25:11,309 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:25:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4163973] error output [ ] Jul 23 16:25:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4163973] error output [ ] Jul 23 16:25:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4163973] error output [ 2021-07-23 16:25:11,309 ERROR: Please use '-h' for usage ] Jul 23 16:25:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4163973] error output [ ] Jul 23 16:25:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4163973] stderr: [ 2021-07-23 16:25:11,309 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:25:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4163973] stderr: [ ] Jul 23 16:25:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4163973] stderr: [ ] Jul 23 16:25:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4163973] stderr: [ 2021-07-23 16:25:11,309 ERROR: Please use '-h' for usage ] Jul 23 16:25:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4163973] stderr: [ ] Jul 23 16:25:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (internal_stonith_action_execute) info: Attempt 2 to execute fence_apc_snmp (monitor). remaining timeout is 14 Jul 23 16:25:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:25:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.357.26 2 Jul 23 16:25:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.358.0 (null) Jul 23 16:25:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=358, @num_updates=0 Jul 23 16:25:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu03']: Jul 23 16:25:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.358.0) Jul 23 16:25:13 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 125 aborted by meta_attributes.apc_snmp_node2_an-pdu03-meta_attributes 'create': Configuration change | cib=0.358.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu03'] complete=false Jul 23 16:25:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create primitive[@id='apc_snmp_node2_an-pdu03'] Jul 23 16:25:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.358.0 Jul 23 16:25:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 16:25:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 16:25:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-53.raw Jul 23 16:25:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.358.0 of the CIB to disk (digest: 0d3b0fe26c7c07196330c649ae925dcd) Jul 23 16:25:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.StcKKu (digest: /var/lib/pacemaker/cib/cib.mKXL5y) Jul 23 16:25:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:25:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.358.0 2 Jul 23 16:25:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.359.0 (null) Jul 23 16:25:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=359 Jul 23 16:25:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03']: Jul 23 16:25:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.359.0) Jul 23 16:25:16 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 125 aborted by meta_attributes.apc_snmp_node1_an-pdu03-meta_attributes 'create': Configuration change | cib=0.359.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03'] complete=false Jul 23 16:25:16 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create primitive[@id='apc_snmp_node1_an-pdu03'] Jul 23 16:25:16 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.359.0 Jul 23 16:25:16 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 16:25:16 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu03 has been disabled Jul 23 16:25:16 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 16:25:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-54.raw Jul 23 16:25:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.359.0 of the CIB to disk (digest: 6c342a432a28daa6236dfc89d0946887) Jul 23 16:25:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.5tBGdf (digest: /var/lib/pacemaker/cib/cib.OuYqbf) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4164894] error output [ 2021-07-23 16:25:18,468 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4164894] error output [ ] Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4164894] error output [ ] Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4164894] error output [ 2021-07-23 16:25:18,468 ERROR: Please use '-h' for usage ] Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4164894] error output [ ] Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4164894] stderr: [ 2021-07-23 16:25:18,468 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4164894] stderr: [ ] Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4164894] stderr: [ ] Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4164894] stderr: [ 2021-07-23 16:25:18,468 ERROR: Please use '-h' for usage ] Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4164894] stderr: [ ] Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_remaining_timeout) info: Attempted to execute agent fence_apc_snmp (monitor) the maximum number of times (2) allowed Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_operation) notice: Operation 'monitor' [4164894] using apc_snmp_node2_an-pdu03 returned -201 (Generic Pacemaker error) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node2_an-pdu03 start (call 166) exited with status 1 (execution time 13317ms, queue time 0ms) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for apc_snmp_node2_an-pdu03 on an-a02n01: error | rc=1 call=166 key=apc_snmp_node2_an-pdu03_start_0 confirmed=true cib-update=647 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/647) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.359.0 2 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.359.1 (null) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:1;32:125:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=166, @rc-code=1, @op-status=0, @exec-time=13317 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/124, version=0.359.2) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 124 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.359.2 2 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.359.3 (null) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/125, version=0.359.3) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 125 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=1627082718 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=1627082705 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 125 aborted by status-1-fail-count-apc_snmp_node2_an-pdu03.start_0 doing create fail-count-apc_snmp_node2_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.359.2 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 125 aborted by status-1-last-failure-apc_snmp_node2_an-pdu03.start_0 doing create last-failure-apc_snmp_node2_an-pdu03#start_0=1627082718: Transient attribute change | cib=0.359.3 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 16:23:49 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:23:09 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:25:05 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:25:05 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_0 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:23:36 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:23:22 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:24:51 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node2_an-pdu03 ( an-a02n01 ) due to node availability Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 126, saving inputs in /var/lib/pacemaker/pengine/pe-input-147.bz2 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627082718-424 is obsolete Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 16:23:49 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:23:09 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:25:05 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:25:05 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_0 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:23:36 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:23:22 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:24:51 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node2_an-pdu03 ( an-a02n01 ) due to node availability Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 127, saving inputs in /var/lib/pacemaker/pengine/pe-input-148.bz2 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 127 (ref=pe_calc-dc-1627082718-425) derived from /var/lib/pacemaker/pengine/pe-input-148.bz2 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node2_an-pdu03_stop_0 locally on an-a02n01 | action 7 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node2_an-pdu03 on an-a02n01 | transition_key=7:127:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu03_stop_0 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node2_an-pdu03 action:stop call_id:167 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/651) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Device 'apc_snmp_node2_an-pdu03' not found (5 active devices) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node2_an-pdu03 stop (call 167) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node2_an-pdu03 on an-a02n01: ok | rc=0 call=167 key=apc_snmp_node2_an-pdu03_stop_0 confirmed=true cib-update=652 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/652) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.359.3 2 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.359.4 (null) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @operation_key=apc_snmp_node2_an-pdu03_stop_0, @operation=stop, @transition-key=7:127:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;7:127:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1627 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/651, version=0.359.4) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.359.4 2 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.359.5 (null) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:0;7:127:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=167, @rc-code=0, @op-status=0 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/652, version=0.359.5) Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 127 action 7 (apc_snmp_node2_an-pdu03_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=167 Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 127 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-148.bz2): Complete Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 16:25:18 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.359.5 2 Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.360.0 (null) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=360, @num_updates=0 Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu04']: Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.360.0) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 127 aborted by meta_attributes.apc_snmp_node1_an-pdu04-meta_attributes 'create': Configuration change | cib=0.360.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu04'] complete=true Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create primitive[@id='apc_snmp_node1_an-pdu04'] Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.360.0 Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu03 has been disabled Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu04 has been disabled Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-55.raw Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.360.0 of the CIB to disk (digest: 255d8bfc3795a18586b5402d9410921d) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.JPtsHX (digest: /var/lib/pacemaker/cib/cib.bsmyVY) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 16:23:49 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:23:09 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:25:05 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:23:36 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:23:22 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:24:51 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 128, saving inputs in /var/lib/pacemaker/pengine/pe-input-149.bz2 Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 128 (ref=pe_calc-dc-1627082719-427) derived from /var/lib/pacemaker/pengine/pe-input-149.bz2 Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 128 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-149.bz2): Complete Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 16:25:19 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:25:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 11f0408af05c0b696f05a4d7d89d5992 for 0.360.0 (0x5619762b8940 0) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.360.0 2 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.361.0 (null) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-2'] Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=361 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Local-only Replace: 0.361.0 from an-a02n01 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.361.0) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 128 aborted by deletion of fencing-level[@id='fl-an-a02n02-2']: Configuration change | cib=0.361.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-2'] complete=true Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.361.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-2'] Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 126 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 127 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 128 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/656) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 129 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 130 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 131 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/657) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 132 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 133 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/656, version=0.361.0) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.361.0 2 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.361.1 (null) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/657, version=0.361.1) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/126) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/127) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/128) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/129) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/130) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/126, version=0.361.1) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 126 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=1627082718 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=1627082705 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/127, version=0.361.1) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 127 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/128, version=0.361.1) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 128 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/129, version=0.361.1) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 129 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=1627082602 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627082616 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.361.1 2 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.361.2 (null) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/130, version=0.361.2) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 130 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/131) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/132) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/133) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.361.2 2 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.361.3 (null) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/131, version=0.361.3) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 131 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/132, version=0.361.3) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 132 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/133, version=0.361.3) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 133 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=1627082642 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627082629 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-56.raw Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/659, version=0.361.3) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/660) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.361.0 of the CIB to disk (digest: a690e69841fdb490dd95a945f27b127b) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.ZkxyXp (digest: /var/lib/pacemaker/cib/cib.4i0N5r) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/660, version=0.361.3) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/662) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/662, version=0.361.3) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/664) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-13 offer to an-a02n02 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-13 offer to an-a02n01 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-13 requests from 2 outstanding nodes Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/664, version=0.361.3) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/666) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/666, version=0.361.3) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: 7af670fb730ab0a14f2c3b2b0e220d45 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.361.3 with 0.361.3 from an-a02n01 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/669, version=0.361.3) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/670) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/671) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/670, version=0.361.3) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/671, version=0.361.3) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4165749] exited with status 0 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-57.raw Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.361.0 of the CIB to disk (digest: a690e69841fdb490dd95a945f27b127b) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.EO1y4E (digest: /var/lib/pacemaker/cib/cib.Hrui5D) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //primitive[@id='apc_snmp_node2_an-pdu04'] to all (origin=local/cibadmin/2) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.361.3 2 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.362.0 (null) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu04'] Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=362, @num_updates=0 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //primitive[@id='apc_snmp_node2_an-pdu04']: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.362.0) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Device 'apc_snmp_node2_an-pdu04' not found (4 active devices) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-58.raw Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.362.0 of the CIB to disk (digest: 7bb4b0a2e2aeb9bde5ec2b76de33bf40) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.FGLHMZ (digest: /var/lib/pacemaker/cib/cib.1ogsIX) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4165751] exited with status 0 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.362.0 2 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.363.0 (null) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=363 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources: Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.363.0) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create resources Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.363.0 Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-59.raw Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu03 has been disabled Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu04 has been disabled Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node2_an-pdu04' to device list (5 active devices) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.363.0 of the CIB to disk (digest: bc6d7be2e2b64d64bdf50232ebf3ddf7) Jul 23 16:25:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.hpTubT (digest: /var/lib/pacemaker/cib/cib.J03OGS) Jul 23 16:25:30 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4165758] exited with status 0 Jul 23 16:25:30 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4165770] exited with status 0 Jul 23 16:25:30 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4165791] exited with status 0 Jul 23 16:25:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:25:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.363.0 2 Jul 23 16:25:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.364.0 (null) Jul 23 16:25:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-1'] Jul 23 16:25:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=364 Jul 23 16:25:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.364.0) Jul 23 16:25:30 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.364.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-1'] Jul 23 16:25:30 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 16:25:30 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:25:30 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 16:25:30 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:25:30 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 16:25:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-60.raw Jul 23 16:25:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.364.0 of the CIB to disk (digest: 1be782f8929b398bbbb5621fdb585681) Jul 23 16:25:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.H7NIom (digest: /var/lib/pacemaker/cib/cib.vPgqvm) Jul 23 16:25:30 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4165795] exited with status 0 Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4165800] exited with status 0 Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.364.0 2 Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.365.0 (null) Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-3'] Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=365 Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.365.0) Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.365.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-3'] Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-61.raw Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.365.0 of the CIB to disk (digest: a79e205b244174519428e3d93a825782) Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.r174z4 (digest: /var/lib/pacemaker/cib/cib.lx3hc4) Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4165804] exited with status 0 Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 672) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/672) Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/673) Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 674) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.365.0 2 Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.365.1 (null) Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/672, version=0.365.1) Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.365.1 2 Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.365.2 (null) Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/673, version=0.365.2) Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/674) Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.365.2 2 Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.365.3 (null) Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/674, version=0.365.3) Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4165814] exited with status 0 Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.365.3 2 Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.366.0 (null) Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=366, @num_updates=0 Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.366.0) Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.366.0 for /cib/configuration/fencing-topology Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (1 active entry) Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-62.raw Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.366.0 of the CIB to disk (digest: 16efb0f57bad73b7576809104ba157b8) Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.GLIUBa (digest: /var/lib/pacemaker/cib/cib.wiDJyd) Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4165815] exited with status 0 Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.366.0 2 Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.367.0 (null) Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=367 Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.367.0) Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.367.0 for /cib/configuration/fencing-topology Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (1 active entry) Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-63.raw Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.367.0 of the CIB to disk (digest: bfa0453441437913459443ffc1dd3936) Jul 23 16:25:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.PHWJiO (digest: /var/lib/pacemaker/cib/cib.gWyMEM) Jul 23 16:25:32 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4165875] exited with status 0 Jul 23 16:25:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:25:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.367.0 2 Jul 23 16:25:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.368.0 (null) Jul 23 16:25:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=368 Jul 23 16:25:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 16:25:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.368.0) Jul 23 16:25:32 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.368.0 for /cib/configuration/fencing-topology Jul 23 16:25:32 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 16:25:32 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:25:32 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 16:25:32 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (1 active entry) Jul 23 16:25:32 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:25:32 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:25:32 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 16:25:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-64.raw Jul 23 16:25:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.368.0 of the CIB to disk (digest: 136556b9ec0856db7bd52e464fdacd92) Jul 23 16:25:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.eayeNw (digest: /var/lib/pacemaker/cib/cib.dip4Ex) Jul 23 16:25:32 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4165888] exited with status 0 Jul 23 16:25:32 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4165898] exited with status 0 Jul 23 16:25:32 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4165901] exited with status 0 Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4165907] exited with status 0 Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4165910] exited with status 0 Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/675) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 128 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.368.0 2 Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.368.1 (null) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/675, version=0.368.1) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/678) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/679) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/680) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 128 aborted: History refresh | source=process_resource_updates:268 complete=true Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/678, version=0.368.1) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.368.1 2 Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.368.2 (null) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/679, version=0.368.2) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/680, version=0.368.2) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 16:23:49 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:23:09 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:25:05 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:23:36 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:23:22 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:24:51 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu04 on an-a02n02 Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node2_an-pdu04 ( an-a02n02 ) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 129, saving inputs in /var/lib/pacemaker/pengine/pe-input-150.bz2 Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 129 (ref=pe_calc-dc-1627082733-435) derived from /var/lib/pacemaker/pengine/pe-input-150.bz2 Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node2_an-pdu04_start_0 on an-a02n02 | action 31 Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.368.2 2 Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.368.3 (null) Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @operation_key=apc_snmp_node2_an-pdu04_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=31:129:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;31:129:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:25:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/178, version=0.368.3) Jul 23 16:25:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 781bf2bae0cff154adff55ef21c922fb for 0.368.3 (0x5619764432b0 0) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.368.3 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.368.4 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:1;31:129:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=147, @rc-code=1, @op-status=0, @exec-time=13314 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/135, version=0.368.5) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 135 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.368.5 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.368.6 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/136, version=0.368.6) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 136 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=1627082746 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 129 aborted by status-2-fail-count-apc_snmp_node2_an-pdu04.start_0 doing create fail-count-apc_snmp_node2_an-pdu04#start_0=INFINITY: Transient attribute change | cib=0.368.5 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 129 aborted by status-2-last-failure-apc_snmp_node2_an-pdu04.start_0 doing create last-failure-apc_snmp_node2_an-pdu04#start_0=1627082746: Transient attribute change | cib=0.368.6 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 16:23:49 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:23:09 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:25:05 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:23:36 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:23:22 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:24:51 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:25:33 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:25:33 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu04 on an-a02n02 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node2_an-pdu04 ( an-a02n02 ) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 130, saving inputs in /var/lib/pacemaker/pengine/pe-input-151.bz2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627082746-437 is obsolete Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 16:23:49 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:23:09 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:25:05 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:23:36 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:23:22 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:24:51 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:25:33 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:25:33 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu04 on an-a02n01 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node2_an-pdu04 ( an-a02n02 -> an-a02n01 ) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 131, saving inputs in /var/lib/pacemaker/pengine/pe-input-152.bz2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 131 (ref=pe_calc-dc-1627082746-438) derived from /var/lib/pacemaker/pengine/pe-input-152.bz2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node2_an-pdu04_stop_0 on an-a02n02 | action 11 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.368.6 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.368.7 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @operation_key=apc_snmp_node2_an-pdu04_stop_0, @operation=stop, @transition-key=11:131:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;11:131:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/180, version=0.368.7) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.368.7 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.368.8 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:0;11:131:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=148, @rc-code=0, @op-status=0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/181, version=0.368.8) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 131 action 11 (apc_snmp_node2_an-pdu04_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=148 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node2_an-pdu04_start_0 locally on an-a02n01 | action 32 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node2_an-pdu04 on an-a02n01 | transition_key=32:131:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu04_start_0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/686) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node2_an-pdu04 action:start call_id:168 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.368.8 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.368.9 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @operation_key=apc_snmp_node2_an-pdu04_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=32:131:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;32:131:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/686, version=0.368.9) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/crmd/687) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]: 1627082718 -> (unset) | from an-a02n01 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 137 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]: 1627082705 -> (unset) | from an-a02n01 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node2_an-pdu03#start_0' delayed: update 137 in progress Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 138 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n01 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node2_an-pdu03#start_0' delayed: update 138 in progress Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 139 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n01 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node1_an-pdu03#start_0' delayed: update 139 in progress Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: 1627082602 -> (unset) | from an-a02n01 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 140 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: 1627082616 -> (unset) | from an-a02n01 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node1_an-pdu03#start_0' delayed: update 140 in progress Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n01 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 141 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]: 1627082746 -> (unset) | from an-a02n01 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 142 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 143 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n01 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node1_an-pdu04#start_0' delayed: update 143 in progress Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]: 1627082642 -> (unset) | from an-a02n01 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 144 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]: 1627082629 -> (unset) | from an-a02n01 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node1_an-pdu04#start_0' delayed: update 144 in progress Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.368.9 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.368.10 2092ea2bb472e2fdcfd702f7a3c4e287 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/687, version=0.368.9) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (delete_resource) info: Removing resource apc_snmp_node1_an-pdu03 from executor for 96c85666-bbd9-496e-87ad-9fcb2ae4d229 as root Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/137) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_resource_history) info: Clearing resource history for apc_snmp_node1_an-pdu03 on an-a02n01 (via CIB call 688) | xpath=//node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (notify_deleted) info: Notifying 96c85666-bbd9-496e-87ad-9fcb2ae4d229 on an-a02n01 that apc_snmp_node1_an-pdu03 was deleted Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/138) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/139) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/140) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/141) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.368.9 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.368.10 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu03.start_0'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/137, version=0.368.10) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 137 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=1627082705 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 145 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.368.10 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.368.11 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu03.start_0'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/138, version=0.368.11) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 138 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 146 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.368.11 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.368.12 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/139, version=0.368.12) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 139 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 147 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.368.12 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.368.13 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/140, version=0.368.13) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 140 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627082616 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 148 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.368.13 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.368.14 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu04.start_0'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/141, version=0.368.14) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 141 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/crmd/688) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/142) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/143) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/144) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 131 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu03.start_0']: Transient attribute change | cib=0.368.10 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu03.start_0'] complete=false Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/145) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/146) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 131 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu03.start_0']: Transient attribute change | cib=0.368.11 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu03.start_0'] complete=false Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 131 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.368.12 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0'] complete=false Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 131 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.368.13 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0'] complete=false Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 131 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu04.start_0']: Transient attribute change | cib=0.368.14 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu04.start_0'] complete=false Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.368.14 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.368.15 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=15 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/688, version=0.368.15) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.368.15 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.368.16 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu04.start_0'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=16 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/142, version=0.368.16) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 142 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.368.16 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.368.17 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu04.start_0'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=17 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/143, version=0.368.17) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 143 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 149 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.368.17 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.368.18 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu04.start_0'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=18 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/144, version=0.368.18) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 144 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627082629 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 150 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.368.18 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.368.19 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu03.start_0'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=19 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/145, version=0.368.19) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 145 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.368.19 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.368.20 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu03.start_0'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=20 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/146, version=0.368.20) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 146 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/690) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] to all (origin=local/crmd/691) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/147) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/148) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/149) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/150) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.368.20 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.0 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=369, @num_updates=0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']: @value=1627082746 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/690, version=0.369.0) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.0 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.1 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n01/crmd/691, version=0.369.0) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (delete_resource) info: Removing resource apc_snmp_node1_an-pdu04 from executor for 96c85666-bbd9-496e-87ad-9fcb2ae4d229 as root Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_resource_history) info: Clearing resource history for apc_snmp_node1_an-pdu04 on an-a02n01 (via CIB call 692) | xpath=//node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (notify_deleted) info: Notifying 96c85666-bbd9-496e-87ad-9fcb2ae4d229 on an-a02n01 that apc_snmp_node1_an-pdu04 was deleted Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.0 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.1 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/147, version=0.369.1) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 147 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.1 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.2 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/148, version=0.369.2) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 148 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.2 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.3 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: OK (rc=0, origin=an-a02n02/crmd/182, version=0.369.2) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.2 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.3 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu04.start_0'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/149, version=0.369.3) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 149 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.3 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.4 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu04.start_0'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/150, version=0.369.4) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 150 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] to all (origin=local/crmd/692) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.4 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.5 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: OK (rc=0, origin=an-a02n02/crmd/183, version=0.369.5) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.5 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.6 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n01/crmd/692, version=0.369.6) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/694) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] to all (origin=local/crmd/695) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-65.raw Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/694, version=0.369.6) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.369.0 of the CIB to disk (digest: e2c78b3faa1e6029bd1ae9ceedab7a33) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.6 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.7 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.IbaCU8 (digest: /var/lib/pacemaker/cib/cib.rFOczb) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/695, version=0.369.6) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (delete_resource) info: Removing resource apc_snmp_node2_an-pdu03 from executor for 96c85666-bbd9-496e-87ad-9fcb2ae4d229 as root Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_resource_history) info: Clearing resource history for apc_snmp_node2_an-pdu03 on an-a02n01 (via CIB call 696) | xpath=//node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (notify_deleted) info: Notifying 96c85666-bbd9-496e-87ad-9fcb2ae4d229 on an-a02n01 that apc_snmp_node2_an-pdu03 was deleted Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] to all (origin=local/crmd/696) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n02/crmd/185, version=0.369.6) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.6 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.7 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/187, version=0.369.6) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.6 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.7 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/696, version=0.369.7) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.7 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.8 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/188, version=0.369.8) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n02/crmd/190, version=0.369.8) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 131 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu03']: Resource state removal | cib=0.368.15 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] complete=false Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 131 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu04.start_0']: Transient attribute change | cib=0.368.16 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu04.start_0'] complete=false Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 131 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.368.17 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu04.start_0'] complete=false Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 131 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.368.18 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu04.start_0'] complete=false Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 131 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu03.start_0']: Transient attribute change | cib=0.368.19 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu03.start_0'] complete=false Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 131 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu03.start_0']: Transient attribute change | cib=0.368.20 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu03.start_0'] complete=false Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 131 aborted by cib-bootstrap-options-last-lrm-refresh doing modify last-lrm-refresh=1627082746: Configuration change | cib=0.369.0 source=te_update_diff_v2:474 path=/cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh'] complete=false Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 131 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.369.1 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] complete=false Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 131 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.369.2 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] complete=false Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.8 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.9 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 131 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.369.3 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu04.start_0'] complete=false Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 131 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.369.4 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu04.start_0'] complete=false Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 131 aborted by deletion of lrm_resource[@id='apc_snmp_node2_an-pdu04']: Resource state removal | cib=0.369.5 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] complete=false Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 131 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu04']: Resource state removal | cib=0.369.6 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] complete=false Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 131 aborted by deletion of lrm_resource[@id='apc_snmp_node2_an-pdu03']: Resource state removal | cib=0.369.7 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] complete=false Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 131 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu03']: Resource state removal | cib=0.369.8 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] complete=false Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n02/crmd/191, version=0.369.8) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/698) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/698, version=0.369.8) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.8 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.9 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n02/crmd/192, version=0.369.9) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 131 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu04']: Resource state removal | cib=0.369.9 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] complete=false Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n02/crmd/194, version=0.369.9) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.9 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.10 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/195, version=0.369.9) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.9 2 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.10 (null) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/196, version=0.369.10) Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 131 aborted by deletion of lrm_resource[@id='apc_snmp_node2_an-pdu03']: Resource state removal | cib=0.369.10 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] complete=false Jul 23 16:25:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n02/crmd/198, version=0.369.10) Jul 23 16:25:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 8aa6d729ad95a641b51b4c0c0fb71e09 for 0.369.10 (0x561976302500 0) Jul 23 16:25:53 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4166837] error output [ 2021-07-23 16:25:53,016 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 16:25:53 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4166837] error output [ ] Jul 23 16:25:53 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4166837] error output [ ] Jul 23 16:25:53 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4166837] error output [ 2021-07-23 16:25:53,017 ERROR: Please use '-h' for usage ] Jul 23 16:25:53 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4166837] error output [ ] Jul 23 16:25:53 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4166837] stderr: [ 2021-07-23 16:25:53,016 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 16:25:53 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4166837] stderr: [ ] Jul 23 16:25:53 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4166837] stderr: [ ] Jul 23 16:25:53 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4166837] stderr: [ 2021-07-23 16:25:53,017 ERROR: Please use '-h' for usage ] Jul 23 16:25:53 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4166837] stderr: [ ] Jul 23 16:25:53 an-a02n01.alteeve.com pacemaker-fenced [121629] (internal_stonith_action_execute) info: Attempt 2 to execute fence_apc_snmp (monitor). remaining timeout is 13 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4167779] error output [ 2021-07-23 16:26:00,175 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4167779] error output [ ] Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4167779] error output [ ] Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4167779] error output [ 2021-07-23 16:26:00,175 ERROR: Please use '-h' for usage ] Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4167779] error output [ ] Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4167779] stderr: [ 2021-07-23 16:26:00,175 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4167779] stderr: [ ] Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4167779] stderr: [ ] Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4167779] stderr: [ 2021-07-23 16:26:00,175 ERROR: Please use '-h' for usage ] Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4167779] stderr: [ ] Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_remaining_timeout) info: Attempted to execute agent fence_apc_snmp (monitor) the maximum number of times (2) allowed Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_operation) notice: Operation 'monitor' [4167779] using apc_snmp_node2_an-pdu04 returned -201 (Generic Pacemaker error) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node2_an-pdu04 start (call 168) exited with status 1 (execution time 13316ms, queue time 0ms) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for apc_snmp_node2_an-pdu04 on an-a02n01: error | rc=1 call=168 key=apc_snmp_node2_an-pdu04_start_0 confirmed=true cib-update=700 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/700) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.10 2 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.11 (null) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:1;32:131:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=168, @rc-code=1, @op-status=0, @exec-time=13316 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/151, version=0.369.12) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 151 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.12 2 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.13 (null) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/152, version=0.369.13) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 152 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=1627082760 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 131 aborted by status-1-fail-count-apc_snmp_node2_an-pdu04.start_0 doing create fail-count-apc_snmp_node2_an-pdu04#start_0=INFINITY: Transient attribute change | cib=0.369.12 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 131 aborted by status-1-last-failure-apc_snmp_node2_an-pdu04.start_0 doing create last-failure-apc_snmp_node2_an-pdu04#start_0=1627082760: Transient attribute change | cib=0.369.13 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:25:46 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:25:46 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_0 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu04 on an-a02n02 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node2_an-pdu04 ( an-a02n01 -> an-a02n02 ) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 132, saving inputs in /var/lib/pacemaker/pengine/pe-input-153.bz2 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627082760-444 is obsolete Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:25:46 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:25:46 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_0 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu04 on an-a02n02 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node2_an-pdu04 ( an-a02n01 -> an-a02n02 ) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 133, saving inputs in /var/lib/pacemaker/pengine/pe-input-154.bz2 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 133 (ref=pe_calc-dc-1627082760-445) derived from /var/lib/pacemaker/pengine/pe-input-154.bz2 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu03_monitor_0 on an-a02n02 | action 15 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu03_monitor_0 locally on an-a02n01 | action 12 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_get_rsc_info) info: Agent information for 'apc_snmp_node1_an-pdu03' not in cache Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_rsc_register) info: Cached agent information for 'apc_snmp_node1_an-pdu03' Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=12:133:7:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_monitor_0 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu04_monitor_0 on an-a02n02 | action 16 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu04_monitor_0 locally on an-a02n01 | action 13 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_get_rsc_info) info: Agent information for 'apc_snmp_node1_an-pdu04' not in cache Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/704) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_rsc_register) info: Cached agent information for 'apc_snmp_node1_an-pdu04' Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for apc_snmp_node1_an-pdu04 on an-a02n01 | transition_key=13:133:7:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu04_monitor_0 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/705) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node2_an-pdu03_monitor_0 on an-a02n02 | action 17 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node2_an-pdu03_monitor_0 locally on an-a02n01 | action 14 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_get_rsc_info) info: Agent information for 'apc_snmp_node2_an-pdu03' not in cache Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_rsc_register) info: Cached agent information for 'apc_snmp_node2_an-pdu03' Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for apc_snmp_node2_an-pdu03 on an-a02n01 | transition_key=14:133:7:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu03_monitor_0 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.13 2 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.14 (null) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node2_an-pdu04_monitor_0 on an-a02n02 | action 18 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node2_an-pdu04_stop_0 locally on an-a02n01 | action 7 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node2_an-pdu04 on an-a02n01 | transition_key=7:133:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu04_stop_0 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/704, version=0.369.14) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node2_an-pdu04 action:stop call_id:184 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of probe operation for apc_snmp_node1_an-pdu03 on an-a02n01: not running | rc=7 call=175 key=apc_snmp_node1_an-pdu03_monitor_0 confirmed=true cib-update=708 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.14 2 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.15 (null) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=15 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of probe operation for apc_snmp_node1_an-pdu04 on an-a02n01: not running | rc=7 call=179 key=apc_snmp_node1_an-pdu04_monitor_0 confirmed=true cib-update=709 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of probe operation for apc_snmp_node2_an-pdu03 on an-a02n01: not running | rc=7 call=183 key=apc_snmp_node2_an-pdu03_monitor_0 confirmed=true cib-update=710 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node2_an-pdu04 stop (call 184) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/705, version=0.369.15) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/706) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node2_an-pdu04 on an-a02n01: ok | rc=0 call=184 key=apc_snmp_node2_an-pdu04_stop_0 confirmed=true cib-update=711 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/707) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/708) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/709) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/710) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.15 2 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.16 (null) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=16 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/706, version=0.369.16) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.16 2 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.17 (null) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=17 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @operation_key=apc_snmp_node2_an-pdu04_stop_0, @operation=stop, @transition-key=7:133:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;7:133:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1627 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/707, version=0.369.17) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.17 2 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.18 (null) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=18 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/199, version=0.369.18) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.18 2 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.19 (null) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=19 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:7;15:133:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=156, @rc-code=7, @op-status=0 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/200, version=0.369.19) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 133 action 15 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=156 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.19 2 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.20 (null) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=20 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/201, version=0.369.20) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.20 2 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.21 (null) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=21 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:7;12:133:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=175, @rc-code=7, @op-status=0 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/708, version=0.369.21) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 133 action 12 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n01) confirmed: not running | rc=7 call-id=175 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.21 2 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.22 (null) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=22 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:7;13:133:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=179, @rc-code=7, @op-status=0 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/709, version=0.369.22) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 133 action 13 (apc_snmp_node1_an-pdu04_monitor_0 on an-a02n01) confirmed: not running | rc=7 call-id=179 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.22 2 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.23 (null) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=23 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:7;14:133:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=183, @rc-code=7, @op-status=0 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/710, version=0.369.23) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 133 action 14 (apc_snmp_node2_an-pdu03_monitor_0 on an-a02n01) confirmed: not running | rc=7 call-id=183 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.23 2 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.24 (null) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=24 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:7;16:133:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=160, @rc-code=7, @op-status=0 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/202, version=0.369.24) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 133 action 16 (apc_snmp_node1_an-pdu04_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=160 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.24 2 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.25 (null) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=25 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/203, version=0.369.25) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.25 2 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.26 (null) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=26 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:7;17:133:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=164, @rc-code=7, @op-status=0 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/204, version=0.369.26) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 133 action 17 (apc_snmp_node2_an-pdu03_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=164 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.26 2 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.27 (null) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=27 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/205, version=0.369.27) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.27 2 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.28 (null) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=28 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:7;18:133:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=168, @rc-code=7, @op-status=0 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/206, version=0.369.28) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 133 action 18 (apc_snmp_node2_an-pdu04_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=168 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/711) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.28 2 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.29 (null) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=29 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:0;7:133:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=184, @rc-code=0, @op-status=0 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/711, version=0.369.29) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 133 action 7 (apc_snmp_node2_an-pdu04_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=184 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node2_an-pdu04_start_0 on an-a02n02 | action 39 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.29 2 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.30 (null) Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=30 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @operation_key=apc_snmp_node2_an-pdu04_start_0, @operation=start, @transition-key=39:133:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;39:133:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1 Jul 23 16:26:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/207, version=0.369.30) Jul 23 16:26:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: d9c727aa52e1777320c66e24aad9552c for 0.369.30 (0x561976302500 0) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.30 2 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.31 (null) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=31 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:1;39:133:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=169, @rc-code=1, @op-status=0, @exec-time=13322 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/153, version=0.369.32) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 153 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.32 2 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.33 (null) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=33 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/154, version=0.369.33) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 154 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=1627082760 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=1627082773 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 133 aborted by status-2-fail-count-apc_snmp_node2_an-pdu04.start_0 doing create fail-count-apc_snmp_node2_an-pdu04#start_0=INFINITY: Transient attribute change | cib=0.369.32 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 133 aborted by status-2-last-failure-apc_snmp_node2_an-pdu04.start_0 doing create last-failure-apc_snmp_node2_an-pdu04#start_0=1627082773: Transient attribute change | cib=0.369.33 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:25:46 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:26:00 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:26:00 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_0 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu04 on an-a02n02 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node2_an-pdu04 ( an-a02n02 ) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 134, saving inputs in /var/lib/pacemaker/pengine/pe-input-155.bz2 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627082773-455 is obsolete Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:25:46 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:26:00 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:26:00 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_0 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node2_an-pdu04 ( an-a02n02 ) due to node availability Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 135, saving inputs in /var/lib/pacemaker/pengine/pe-input-156.bz2 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 135 (ref=pe_calc-dc-1627082773-456) derived from /var/lib/pacemaker/pengine/pe-input-156.bz2 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node2_an-pdu04_stop_0 on an-a02n02 | action 11 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.33 2 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.34 (null) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=34 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @operation_key=apc_snmp_node2_an-pdu04_stop_0, @operation=stop, @transition-key=11:135:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;11:135:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/209, version=0.369.34) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.34 2 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.369.35 (null) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=35 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:0;11:135:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=170, @rc-code=0, @op-status=0 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/210, version=0.369.35) Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 135 action 11 (apc_snmp_node2_an-pdu04_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=170 Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 135 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-156.bz2): Complete Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 16:26:13 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.369.35 2 Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.370.0 (null) Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=370, @num_updates=0 Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu04']: Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.370.0) Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 135 aborted by meta_attributes.apc_snmp_node2_an-pdu04-meta_attributes 'create': Configuration change | cib=0.370.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu04'] complete=true Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create primitive[@id='apc_snmp_node2_an-pdu04'] Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.370.0 Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu03 has been disabled Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu04 has been disabled Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-66.raw Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.370.0 of the CIB to disk (digest: 542aa129a82672846031074075689b1f) Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.HpE0gf (digest: /var/lib/pacemaker/cib/cib.Mj0Xwf) Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:25:46 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:26:00 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 136, saving inputs in /var/lib/pacemaker/pengine/pe-input-157.bz2 Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 136 (ref=pe_calc-dc-1627082776-458) derived from /var/lib/pacemaker/pengine/pe-input-157.bz2 Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 136 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-157.bz2): Complete Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 16:26:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:26:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 05627c9880b531d64a542f55bbf599cf for 0.370.0 (0x561975c6a5c0 0) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] to all (origin=local/crmd/716) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 155 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n01 Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node2_an-pdu04#start_0' delayed: update 155 in progress Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]: 1627082760 -> (unset) | from an-a02n01 Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 156 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]: 1627082773 -> (unset) | from an-a02n01 Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node2_an-pdu04#start_0' delayed: update 156 in progress Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.370.0 2 Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.370.1 (null) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: OK (rc=0, origin=an-a02n01/crmd/716, version=0.370.0) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-controld [121633] (delete_resource) info: Removing resource apc_snmp_node2_an-pdu04 from executor for c523d877-e940-4517-938d-28616cc71eb5 as root Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/155) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_resource_history) info: Clearing resource history for apc_snmp_node2_an-pdu04 on an-a02n01 (via CIB call 717) | xpath=//node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-controld [121633] (notify_deleted) info: Notifying c523d877-e940-4517-938d-28616cc71eb5 on an-a02n01 that apc_snmp_node2_an-pdu04 was deleted Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/156) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] to all (origin=local/crmd/717) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.370.0 2 Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.370.1 (null) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu04.start_0'] Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/155, version=0.370.1) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 136 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu04.start_0']: Transient attribute change | cib=0.370.1 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu04.start_0'] complete=true Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 155 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 157 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.370.1 2 Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.370.2 (null) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu04.start_0'] Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/156, version=0.370.2) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 136 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu04.start_0']: Transient attribute change | cib=0.370.2 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu04.start_0'] complete=true Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 156 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=1627082773 Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 158 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.370.2 2 Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.370.3 (null) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: OK (rc=0, origin=an-a02n01/crmd/717, version=0.370.3) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 136 aborted by deletion of lrm_resource[@id='apc_snmp_node2_an-pdu04']: Resource state removal | cib=0.370.3 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] complete=true Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/719) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/157) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/158) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.370.3 2 Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.371.0 (null) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=371, @num_updates=0 Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']: @value=1627082796 Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/719, version=0.371.0) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.371.0 2 Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.371.1 (null) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu04.start_0'] Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/157, version=0.371.1) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 157 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.371.1 2 Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.371.2 (null) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu04.start_0'] Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/158, version=0.371.2) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 158 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.371.2 2 Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.371.3 (null) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: OK (rc=0, origin=an-a02n02/crmd/211, version=0.371.2) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.371.2 2 Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.371.3 (null) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: OK (rc=0, origin=an-a02n02/crmd/212, version=0.371.3) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n02/crmd/214, version=0.371.3) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-67.raw Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 136 aborted by cib-bootstrap-options-last-lrm-refresh doing modify last-lrm-refresh=1627082796: Configuration change | cib=0.371.0 source=te_update_diff_v2:474 path=/cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh'] complete=true Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 136 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu04.start_0']: Transient attribute change | cib=0.371.1 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu04.start_0'] complete=true Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 136 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu04.start_0']: Transient attribute change | cib=0.371.2 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu04.start_0'] complete=true Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 136 aborted by deletion of lrm_resource[@id='apc_snmp_node2_an-pdu04']: Resource state removal | cib=0.371.3 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] complete=true Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.371.0 of the CIB to disk (digest: 9b21006e0b6036623d2c6b007db3385a) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.S4RdnG (digest: /var/lib/pacemaker/cib/cib.4sc3JG) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:26:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 137, saving inputs in /var/lib/pacemaker/pengine/pe-input-158.bz2 Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627082796-460 is obsolete Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 138, saving inputs in /var/lib/pacemaker/pengine/pe-input-158.bz2 Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (handle_pecalc_op) info: Input has not changed since last time, not saving to disk Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 138 (ref=pe_calc-dc-1627082797-461) derived from /var/lib/pacemaker/pengine/pe-input-158.bz2 Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node2_an-pdu04_monitor_0 on an-a02n02 | action 12 Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node2_an-pdu04_monitor_0 locally on an-a02n01 | action 11 Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_get_rsc_info) info: Agent information for 'apc_snmp_node2_an-pdu04' not in cache Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_rsc_register) info: Cached agent information for 'apc_snmp_node2_an-pdu04' Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for apc_snmp_node2_an-pdu04 on an-a02n01 | transition_key=11:138:7:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu04_monitor_0 Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of probe operation for apc_snmp_node2_an-pdu04 on an-a02n01: not running | rc=7 call=189 key=apc_snmp_node2_an-pdu04_monitor_0 confirmed=true cib-update=730 Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/729) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/730) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.371.3 2 Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.371.4 (null) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/729, version=0.371.4) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.371.4 2 Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.371.5 (null) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:7;11:138:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=189, @rc-code=7, @op-status=0 Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/730, version=0.371.5) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 138 action 11 (apc_snmp_node2_an-pdu04_monitor_0 on an-a02n01) confirmed: not running | rc=7 call-id=189 Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.371.5 2 Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.371.6 (null) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/216, version=0.371.6) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.371.6 2 Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.371.7 (null) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:7;12:138:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=175, @rc-code=7, @op-status=0 Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/217, version=0.371.7) Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 138 action 12 (apc_snmp_node2_an-pdu04_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=175 Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 138 (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-158.bz2): Complete Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 16:26:37 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:26:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 6d631b56a17c918394e90d929db5f15a for 0.371.7 (0x5619761a3270 0) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.371.7 2 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.372.0 4efc879f479eede1c5e374cec765efb7 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=372, @num_updates=0 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu04']/instance_attributes[@id='apc_snmp_node1_an-pdu04-instance_attributes']/nvpair[@id='apc_snmp_node1_an-pdu04-instance_attributes-ip']: @value=10.201.2.4 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu04']/instance_attributes[@id='apc_snmp_node1_an-pdu04-instance_attributes']/nvpair[@id='apc_snmp_node1_an-pdu04-instance_attributes-port']: @value=3 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-1']: @id=fl-an-a02n02-1, @index=1 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-2']: @id=fl-an-a02n02-2, @index=2 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-3']: @id=fl-an-a02n02-3, @index=3 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.371.7 -> 0.372.0 from an-a02n01 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.372.0) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 138 aborted by apc_snmp_node1_an-pdu04-instance_attributes-ip doing modify ip=10.201.2.4: Configuration change | cib=0.372.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu04']/instance_attributes[@id='apc_snmp_node1_an-pdu04-instance_attributes']/nvpair[@id='apc_snmp_node1_an-pdu04-instance_attributes-ip'] complete=true Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 159 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 160 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 161 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 162 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 163 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 164 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 165 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 166 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/733) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/734) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 1 from fencing topology for an-a02n02 (2 active levels remaining) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 2 from fencing topology for an-a02n02 (2 active levels remaining) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 3 from fencing topology for an-a02n02 (2 active levels remaining) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: modify nvpair[@id='apc_snmp_node1_an-pdu04-instance_attributes-ip'] Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.372.0 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/159) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/160) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/161) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-68.raw Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/162) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/163) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu03 has been disabled Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu04 has been disabled Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.372.0 of the CIB to disk (digest: 75fa4660437e76ea0075ffdd53407476) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.MQMZON (digest: /var/lib/pacemaker/cib/cib.WwX8fN) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/733, version=0.372.0) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.372.0 2 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.372.1 (null) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/734, version=0.372.1) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.372.1 2 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.372.2 (null) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/159, version=0.372.2) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 159 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.372.2 2 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.372.3 (null) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/160, version=0.372.3) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 160 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.372.3 2 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.372.4 (null) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/161, version=0.372.4) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 161 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.372.4 2 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.372.5 (null) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/162, version=0.372.5) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 162 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.372.5 2 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.372.6 (null) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/163, version=0.372.6) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/164) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 163 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/165) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/166) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.372.6 2 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.372.7 (null) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/164, version=0.372.7) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 164 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.372.7 2 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.372.8 (null) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/165, version=0.372.8) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 165 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.372.8 2 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.372.9 (null) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/166, version=0.372.9) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 166 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/736, version=0.372.9) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/737) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/739) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/737, version=0.372.9) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/739, version=0.372.9) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/741) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-14 offer to an-a02n02 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/743) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-14 offer to an-a02n01 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-14 requests from 2 outstanding nodes Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/741, version=0.372.9) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/743, version=0.372.9) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: 14118dcd288e3c57acb787d38f2512e1 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.372.9 with 0.372.9 from an-a02n01 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/746, version=0.372.9) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/747) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/748) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/747, version=0.372.9) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/748, version=0.372.9) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4182112] exited with status 0 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-69.raw Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.372.0 of the CIB to disk (digest: 75fa4660437e76ea0075ffdd53407476) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.zP5L4W (digest: /var/lib/pacemaker/cib/cib.VcAQFW) Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4182114] exited with status 0 Jul 23 16:30:56 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4182115] exited with status 0 Jul 23 16:30:57 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4182116] exited with status 0 Jul 23 16:30:57 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4182129] exited with status 0 Jul 23 16:30:57 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4182131] exited with status 0 Jul 23 16:30:57 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4182132] exited with status 0 Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4182134] exited with status 0 Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 749) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 751) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/749) Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/750) Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/751) Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.372.9 2 Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.372.10 (null) Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/749, version=0.372.10) Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.372.10 2 Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.372.11 (null) Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/750, version=0.372.11) Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.372.11 2 Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.372.12 (null) Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/751, version=0.372.12) Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4182136] exited with status 0 Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4182137] exited with status 0 Jul 23 16:30:58 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4182138] exited with status 0 Jul 23 16:30:59 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4182139] exited with status 0 Jul 23 16:30:59 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4182140] exited with status 0 Jul 23 16:30:59 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4182142] exited with status 0 Jul 23 16:30:59 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4182157] exited with status 0 Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4182159] exited with status 0 Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/752) Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 138 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/755) Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/756) Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/757) Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.372.12 2 Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.372.13 (null) Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/752, version=0.372.13) Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/755, version=0.372.13) Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.372.13 2 Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.372.14 (null) Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/756, version=0.372.14) Jul 23 16:31:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/757, version=0.372.14) Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 138 aborted: History refresh | source=process_resource_updates:268 complete=true Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 139, saving inputs in /var/lib/pacemaker/pengine/pe-input-159.bz2 Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 139 (ref=pe_calc-dc-1627083061-471) derived from /var/lib/pacemaker/pengine/pe-input-159.bz2 Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 139 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-159.bz2): Complete Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 16:31:01 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:31:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 06978b2602d064c45971f51be8969809 for 0.372.14 (0x561975c23b70 0) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.372.14 2 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.373.0 (null) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=373, @num_updates=0 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03']/instance_attributes[@id='apc_snmp_node1_an-pdu03-instance_attributes']/nvpair[@id='apc_snmp_node1_an-pdu03-instance_attributes-ip']: @value=10.201.2.3 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03']/instance_attributes[@id='apc_snmp_node1_an-pdu03-instance_attributes']/nvpair[@id='apc_snmp_node1_an-pdu03-instance_attributes-port']: @value=3 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.372.14 -> 0.373.0 from an-a02n01 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.373.0) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 139 aborted by apc_snmp_node1_an-pdu03-instance_attributes-ip doing modify ip=10.201.2.3: Configuration change | cib=0.373.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03']/instance_attributes[@id='apc_snmp_node1_an-pdu03-instance_attributes']/nvpair[@id='apc_snmp_node1_an-pdu03-instance_attributes-ip'] complete=true Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: modify nvpair[@id='apc_snmp_node1_an-pdu03-instance_attributes-ip'] Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 168 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.373.0 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 169 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 170 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 171 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 172 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 173 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 174 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 175 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/762) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/763) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/168) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu03 has been disabled Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu04 has been disabled Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/169) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/170) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/171) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/172) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/762, version=0.373.0) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.373.0 2 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.373.1 (null) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/763, version=0.373.1) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.373.1 2 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.373.2 (null) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/168, version=0.373.2) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 168 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.373.2 2 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.373.3 (null) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/169, version=0.373.3) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 169 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.373.3 2 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.373.4 (null) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/170, version=0.373.4) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 170 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.373.4 2 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.373.5 (null) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/171, version=0.373.5) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 171 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.373.5 2 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.373.6 (null) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/172, version=0.373.6) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/173) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 172 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/174) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/175) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.373.6 2 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.373.7 (null) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/173, version=0.373.7) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 173 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.373.7 2 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.373.8 (null) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/174, version=0.373.8) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 174 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.373.8 2 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.373.9 (null) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/175, version=0.373.9) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 175 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-70.raw Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.373.0 of the CIB to disk (digest: 97acb14597656ba02cb99daaa0ac0e65) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/765, version=0.373.9) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.uWEqkP (digest: /var/lib/pacemaker/cib/cib.jHH6nN) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/766) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/768) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/770) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-15 offer to an-a02n02 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-15 offer to an-a02n01 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-15 requests from 2 outstanding nodes Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/766, version=0.373.9) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/768, version=0.373.9) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/770, version=0.373.9) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/772) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/772, version=0.373.9) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: 06687bf74aa446f749509f911bf827af Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.373.9 with 0.373.9 from an-a02n01 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/775, version=0.373.9) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/776) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/777) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-71.raw Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/776, version=0.373.9) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.373.0 of the CIB to disk (digest: 97acb14597656ba02cb99daaa0ac0e65) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.34kCBX (digest: /var/lib/pacemaker/cib/cib.5Xg0tV) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/777, version=0.373.9) Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4182231] exited with status 0 Jul 23 16:31:07 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4182233] exited with status 0 Jul 23 16:31:08 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4182234] exited with status 0 Jul 23 16:31:08 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4182252] exited with status 0 Jul 23 16:31:08 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4182276] exited with status 0 Jul 23 16:31:08 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4182295] exited with status 0 Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4182441] exited with status 0 Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4182590] exited with status 0 Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 778) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/778) Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 780) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/779) Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/780) Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.373.9 2 Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.373.10 (null) Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/778, version=0.373.10) Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.373.10 2 Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.373.11 (null) Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/779, version=0.373.11) Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.373.11 2 Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.373.12 (null) Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/780, version=0.373.12) Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4182734] exited with status 0 Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.373.12) Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-72.raw Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.373.0 of the CIB to disk (digest: e6405735a0612811e0606983451b0efd) Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.C3f7f8 (digest: /var/lib/pacemaker/cib/cib.Mk0SNa) Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.373.12 2 Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.374.0 (null) Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03'] Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=374, @num_updates=0 Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.374.0) Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Device 'apc_snmp_node1_an-pdu03' not found (4 active devices) Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-73.raw Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.374.0 of the CIB to disk (digest: 723326fe2fa93b91e8ffddb30e031ad2) Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.eP4T6f (digest: /var/lib/pacemaker/cib/cib.BXPM7c) Jul 23 16:31:09 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4182736] exited with status 0 Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.374.0 2 Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.375.0 (null) Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=375 Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources: Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.375.0) Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create resources Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.375.0 Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu04 has been disabled Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node1_an-pdu03' to device list (5 active devices) Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-74.raw Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.375.0 of the CIB to disk (digest: e772def26322aa39f7bf5b6796284ce1) Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.SIC5gz (digest: /var/lib/pacemaker/cib/cib.n8lg8B) Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4182741] exited with status 0 Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4182744] exited with status 0 Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4182896] exited with status 0 Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.375.0) Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-75.raw Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.375.0 of the CIB to disk (digest: e772def26322aa39f7bf5b6796284ce1) Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.rFpCyr (digest: /var/lib/pacemaker/cib/cib.T2yD4u) Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.375.0 2 Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.376.0 (null) Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu04'] Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=376 Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.376.0) Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Device 'apc_snmp_node1_an-pdu04' not found (5 active devices) Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-76.raw Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.376.0 of the CIB to disk (digest: 79c30b6692048285184e09848579e99f) Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.ohhtky (digest: /var/lib/pacemaker/cib/cib.6hbuiw) Jul 23 16:31:10 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4182899] exited with status 0 Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.376.0 2 Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.377.0 (null) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=377 Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources: Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.377.0) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create resources Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.377.0 Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node1_an-pdu04' to device list (6 active devices) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-77.raw Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.377.0 of the CIB to disk (digest: e1c602e24a488d071b94d94c8999c2cc) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.xPSSBC (digest: /var/lib/pacemaker/cib/cib.eGokZB) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4182911] exited with status 0 Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4183015] exited with status 0 Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/781) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 139 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.377.0 2 Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.377.1 (null) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/781, version=0.377.1) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 139 aborted: History refresh | source=process_resource_updates:268 complete=true Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/784) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/785) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/786) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/784, version=0.377.1) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.377.1 2 Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.377.2 (null) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/785, version=0.377.2) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/786, version=0.377.2) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n02 Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu04 ( an-a02n02 ) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 140, saving inputs in /var/lib/pacemaker/pengine/pe-input-160.bz2 Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 140 (ref=pe_calc-dc-1627083071-479) derived from /var/lib/pacemaker/pengine/pe-input-160.bz2 Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 on an-a02n02 | action 31 Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu04_start_0 on an-a02n02 | action 33 Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.377.2 2 Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.377.3 (null) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=31:140:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;31:140:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/222, version=0.377.3) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.377.3 2 Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.377.4 (null) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=33:140:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;33:140:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/223, version=0.377.4) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.377.4 2 Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.378.0 (null) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=378, @num_updates=0 Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.378.0) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 140 aborted by deletion of fencing-level[@id='fl-an-a02n01-1']: Configuration change | cib=0.378.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] complete=false Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.378.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (1 active entry) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-78.raw Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.378.0 of the CIB to disk (digest: 5d44f084df5a094c03cad150543cd706) Jul 23 16:31:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.GWNPoG (digest: /var/lib/pacemaker/cib/cib.LnCjtG) Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.378.0 2 Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.379.0 (null) Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3'] Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=379 Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.379.0) Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by deletion of fencing-level[@id='fl-an-a02n01-3']: Configuration change | cib=0.379.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3'] complete=false Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.379.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3'] Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-79.raw Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.379.0 of the CIB to disk (digest: 2db36135c4940a8d2d0ae6e225284c36) Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.RXO8tN (digest: /var/lib/pacemaker/cib/cib.h8prDL) Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.379.0 2 Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.380.0 (null) Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=380 Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.380.0) Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by fencing-level.fl-an-a02n01-1 'create': Configuration change | cib=0.380.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology complete=false Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.380.0 for /cib/configuration/fencing-topology Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-80.raw Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.380.0 of the CIB to disk (digest: 7d96536300e6b9a929a8e54c050eb7af) Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.cpHvyS (digest: /var/lib/pacemaker/cib/cib.g0J9pR) Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.380.0 2 Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.381.0 (null) Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=381 Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.381.0) Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by fencing-level.fl-an-a02n01-2 'create': Configuration change | cib=0.381.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology complete=false Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.381.0 for /cib/configuration/fencing-topology Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-81.raw Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.381.0 of the CIB to disk (digest: c8734d3ef267b8fa7f839e8890ec52fb) Jul 23 16:31:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.4jYUPV (digest: /var/lib/pacemaker/cib/cib.y2I2LW) Jul 23 16:31:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.381.0 2 Jul 23 16:31:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.382.0 (null) Jul 23 16:31:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=382 Jul 23 16:31:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 16:31:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.382.0) Jul 23 16:31:13 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by fencing-level.fl-an-a02n01-3 'create': Configuration change | cib=0.382.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology complete=false Jul 23 16:31:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.382.0 for /cib/configuration/fencing-topology Jul 23 16:31:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 16:31:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:31:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:31:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 16:31:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 16:31:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:31:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 16:31:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 16:31:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-82.raw Jul 23 16:31:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.382.0 of the CIB to disk (digest: b9236522de9225c07355d74be7fd15a7) Jul 23 16:31:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.ATOuu5 (digest: /var/lib/pacemaker/cib/cib.CqW9M6) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.382.0 2 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.383.0 (null) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=383 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu03']/instance_attributes[@id='apc_snmp_node2_an-pdu03-instance_attributes']/nvpair[@id='apc_snmp_node2_an-pdu03-instance_attributes-ip']: @value=10.201.2.3 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu03']/instance_attributes[@id='apc_snmp_node2_an-pdu03-instance_attributes']/nvpair[@id='apc_snmp_node2_an-pdu03-instance_attributes-pcmk_host_list']: @value=an-a02n01 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu03']/instance_attributes[@id='apc_snmp_node2_an-pdu03-instance_attributes']/nvpair[@id='apc_snmp_node2_an-pdu03-instance_attributes-port']: @value=4 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1']: @id=fl-an-a02n01-1, @index=1 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2']: @id=fl-an-a02n01-2, @index=2 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3']: @id=fl-an-a02n01-3, @index=3 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Local-only Replace: 0.383.0 from an-a02n01 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.383.0) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by apc_snmp_node2_an-pdu03-instance_attributes-ip doing modify ip=10.201.2.3: Configuration change | cib=0.383.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu03']/instance_attributes[@id='apc_snmp_node2_an-pdu03-instance_attributes']/nvpair[@id='apc_snmp_node2_an-pdu03-instance_attributes-ip'] complete=false Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 177 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 178 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 179 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 180 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 181 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 182 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 183 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 184 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 1 from fencing topology for an-a02n01 (2 active levels remaining) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 2 from fencing topology for an-a02n01 (2 active levels remaining) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/177) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 3 from fencing topology for an-a02n01 (2 active levels remaining) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/178) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: modify nvpair[@id='apc_snmp_node2_an-pdu03-instance_attributes-ip'] Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.383.0 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/179) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/180) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/181) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.383.0 2 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.383.1 (null) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/177, version=0.383.1) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 177 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.383.1 2 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.383.2 (null) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/178, version=0.383.2) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 178 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.383.2 2 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.383.3 (null) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/179, version=0.383.3) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 179 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.383.3 2 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.383.4 (null) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/180, version=0.383.4) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 180 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.383.4 2 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.383.5 (null) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/181, version=0.383.5) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 181 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/791) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/792) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/182) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/183) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/184) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/791, version=0.383.5) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.383.5 2 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.383.6 (null) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/792, version=0.383.6) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.383.6 2 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.383.7 (null) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/182, version=0.383.7) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 182 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.383.7 2 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.383.8 (null) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/183, version=0.383.8) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 183 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.383.8 2 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.383.9 (null) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/184, version=0.383.9) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 184 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-83.raw Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.383.0 of the CIB to disk (digest: 70b0f7053e29ae14dfdcc9d1078ced87) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.eF2RbG (digest: /var/lib/pacemaker/cib/cib.JyJ0zE) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/793, version=0.383.9) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/794) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/796) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/798) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/800) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-16 offer to an-a02n02 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-16 offer to an-a02n01 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-16 requests from 2 outstanding nodes Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/794, version=0.383.9) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/796, version=0.383.9) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/798, version=0.383.9) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/800, version=0.383.9) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: 048be56023e41a33187633cbb71c3c5b Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.383.9 with 0.383.9 from an-a02n01 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/803, version=0.383.9) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/804) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/805) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-84.raw Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.383.0 of the CIB to disk (digest: 70b0f7053e29ae14dfdcc9d1078ced87) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.Xjcw8R (digest: /var/lib/pacemaker/cib/cib.cTAbhR) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/804, version=0.383.9) Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4183208] exited with status 0 Jul 23 16:31:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/805, version=0.383.9) Jul 23 16:31:18 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4183210] exited with status 0 Jul 23 16:31:18 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4183212] exited with status 0 Jul 23 16:31:18 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4183213] exited with status 0 Jul 23 16:31:18 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4183214] exited with status 0 Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4183216] exited with status 0 Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4183229] exited with status 0 Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4183230] exited with status 0 Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 806) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 808) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/806) Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/807) Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/808) Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.383.9 2 Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.383.10 (null) Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/806, version=0.383.10) Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.383.10 2 Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.383.11 (null) Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/807, version=0.383.11) Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.383.11 2 Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.383.12 (null) Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/808, version=0.383.12) Jul 23 16:31:19 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4183231] exited with status 0 Jul 23 16:31:20 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4183232] exited with status 0 Jul 23 16:31:20 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4183233] exited with status 0 Jul 23 16:31:20 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4183234] exited with status 0 Jul 23 16:31:20 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4183235] exited with status 0 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4183237] exited with status 0 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4183238] exited with status 0 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4183254] exited with status 0 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/809) Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by deletion of lrm[@id='2']: Resource state removal | cib=0.383.10 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2'] complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv14-file-processor_monitor_0 'create' on an-a02n02: Old event | magic=0:7;20:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.11 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 20 (srv14-file-processor_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=41 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation delay_node1_stop_0 'create' on an-a02n02: Old event | magic=0:0;20:123:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.11 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 123 action 20 (delay_node1_stop_0 on an-a02n02): arrived really late | rc=0 target-rc=0 call-id=142 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation delay_node2_start_0 'create' on an-a02n02: Old event | magic=0:0;16:123:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.11 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 123 action 16 (delay_node2_start_0 on an-a02n02): arrived really late | rc=0 target-rc=0 call-id=144 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation delay_node2_monitor_60000 'create' on an-a02n02: Change in recurring result | magic=0:0;17:123:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.11 source=process_graph_event:408 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 123 action 17 (delay_node2_monitor_60000 on an-a02n02): arrived after initial scheduling | rc=0 target-rc=0 call-id=145 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation apc_snmp_node1_an-pdu04_monitor_0 'create' on an-a02n02: Old event | magic=0:7;16:133:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.11 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 133 action 16 (apc_snmp_node1_an-pdu04_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=160 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation apc_snmp_node1_an-pdu03_monitor_0 'create' on an-a02n02: Old event | magic=0:7;15:133:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.11 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 133 action 15 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=156 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv22-machineinterface_monitor_0 'create' on an-a02n02: Old event | magic=0:7;28:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.11 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 28 (srv22-machineinterface_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=73 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv13-psql_monitor_0 'create' on an-a02n02: Old event | magic=0:7;19:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.11 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 19 (srv13-psql_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=37 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv24-inventory_monitor_0 'create' on an-a02n02: Old event | magic=0:7;30:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.11 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 30 (srv24-inventory_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=81 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv15-tickets_monitor_0 'create' on an-a02n02: Old event | magic=0:7;21:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.11 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 21 (srv15-tickets_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=45 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv18-inventory_monitor_0 'create' on an-a02n02: Old event | magic=0:7;24:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.11 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 24 (srv18-inventory_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=57 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv21-tickets_monitor_0 'create' on an-a02n02: Old event | magic=0:7;27:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.11 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 27 (srv21-tickets_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=69 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv17-order-processing_monitor_0 'create' on an-a02n02: Old event | magic=0:7;23:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.11 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 23 (srv17-order-processing_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=53 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation ipmilan_node1_start_0 'create' on an-a02n02: Old event | magic=0:0;22:122:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.11 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 122 action 22 (ipmilan_node1_start_0 on an-a02n02): arrived really late | rc=0 target-rc=0 call-id=137 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation ipmilan_node1_monitor_60000 'create' on an-a02n02: Change in recurring result | magic=0:0;23:122:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.11 source=process_graph_event:408 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 122 action 23 (ipmilan_node1_monitor_60000 on an-a02n02): arrived after initial scheduling | rc=0 target-rc=0 call-id=138 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation ipmilan_node2_stop_0 'create' on an-a02n02: Old event | magic=0:0;16:122:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.11 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 122 action 16 (ipmilan_node2_stop_0 on an-a02n02): arrived really late | rc=0 target-rc=0 call-id=136 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv19-uls_monitor_0 'create' on an-a02n02: Old event | magic=0:7;25:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.11 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 25 (srv19-uls_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=61 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation apc_snmp_node2_an-pdu03_monitor_0 'create' on an-a02n02: Old event | magic=0:7;17:133:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.11 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 133 action 17 (apc_snmp_node2_an-pdu03_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=164 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation apc_snmp_node2_an-pdu04_monitor_0 'create' on an-a02n02: Old event | magic=0:7;12:138:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.11 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 138 action 12 (apc_snmp_node2_an-pdu04_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=175 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv20-fileprocessor_monitor_0 'create' on an-a02n02: Old event | magic=0:7;26:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.11 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 26 (srv20-fileprocessor_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=65 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv16-machine-interface_monitor_0 'create' on an-a02n02: Old event | magic=0:7;22:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.11 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 22 (srv16-machine-interface_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=49 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv23-orderprocessing_monitor_0 'create' on an-a02n02: Old event | magic=0:7;29:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.11 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 29 (srv23-orderprocessing_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=77 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by deletion of lrm[@id='1']: Resource state removal | cib=0.383.12 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1'] complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted: Peer Cancelled | source=do_te_invoke:129 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.383.12 2 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.383.13 (null) Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/809, version=0.383.13) Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/812) Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/813) Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/814) Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv14-file-processor_monitor_0 'create' on an-a02n01: Old event | magic=0:7;10:0:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.13 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 0 action 10 (srv14-file-processor_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=41 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation delay_node1_start_0 'create' on an-a02n01: Old event | magic=0:0;21:123:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.13 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 123 action 21 (delay_node1_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=164 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation delay_node1_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;22:123:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.13 source=process_graph_event:408 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 123 action 22 (delay_node1_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=165 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation delay_node2_stop_0 'create' on an-a02n01: Old event | magic=0:0;15:123:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.13 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 123 action 15 (delay_node2_stop_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=163 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation apc_snmp_node1_an-pdu04_monitor_0 'create' on an-a02n01: Old event | magic=0:7;13:133:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.13 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 133 action 13 (apc_snmp_node1_an-pdu04_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=179 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation apc_snmp_node1_an-pdu03_monitor_0 'create' on an-a02n01: Old event | magic=0:7;12:133:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.13 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 133 action 12 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=175 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv13-psql_monitor_0 'create' on an-a02n01: Old event | magic=0:7;9:0:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.13 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 0 action 9 (srv13-psql_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=37 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv22-machineinterface_start_0 'create' on an-a02n01: Old event | magic=0:0;22:8:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.13 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 8 action 22 (srv22-machineinterface_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=100 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv22-machineinterface_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;24:9:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.13 source=process_graph_event:408 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 9 action 24 (srv22-machineinterface_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=101 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv24-inventory_start_0 'create' on an-a02n01: Old event | magic=0:0;28:10:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.13 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 10 action 28 (srv24-inventory_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=104 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv24-inventory_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;29:10:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.13 source=process_graph_event:408 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 10 action 29 (srv24-inventory_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=105 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv15-tickets_monitor_0 'create' on an-a02n01: Old event | magic=0:7;11:0:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.13 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 0 action 11 (srv15-tickets_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=45 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv18-inventory_monitor_0 'create' on an-a02n01: Old event | magic=0:7;14:0:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.13 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 0 action 14 (srv18-inventory_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=57 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv21-tickets_start_0 'create' on an-a02n01: Old event | magic=0:0;19:7:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.13 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 7 action 19 (srv21-tickets_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=98 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv21-tickets_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;21:8:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.13 source=process_graph_event:408 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 8 action 21 (srv21-tickets_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=99 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv17-order-processing_monitor_0 'create' on an-a02n01: Old event | magic=0:7;13:0:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.13 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 0 action 13 (srv17-order-processing_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=53 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation ipmilan_node1_stop_0 'create' on an-a02n01: Old event | magic=0:0;21:122:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.13 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 122 action 21 (ipmilan_node1_stop_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=159 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation ipmilan_node2_start_0 'create' on an-a02n01: Old event | magic=0:0;17:122:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.13 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 122 action 17 (ipmilan_node2_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=160 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation ipmilan_node2_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;18:122:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.13 source=process_graph_event:408 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 122 action 18 (ipmilan_node2_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=161 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv19-uls_start_0 'create' on an-a02n01: Old event | magic=0:0;13:5:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.13 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 5 action 13 (srv19-uls_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=94 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv19-uls_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;15:6:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.13 source=process_graph_event:408 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 6 action 15 (srv19-uls_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=95 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation apc_snmp_node2_an-pdu03_monitor_0 'create' on an-a02n01: Old event | magic=0:7;14:133:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.13 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 133 action 14 (apc_snmp_node2_an-pdu03_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=183 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation apc_snmp_node2_an-pdu04_monitor_0 'create' on an-a02n01: Old event | magic=0:7;11:138:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.13 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 138 action 11 (apc_snmp_node2_an-pdu04_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=189 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv20-fileprocessor_start_0 'create' on an-a02n01: Old event | magic=0:0;16:6:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.13 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 6 action 16 (srv20-fileprocessor_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=96 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv20-fileprocessor_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;18:7:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.13 source=process_graph_event:408 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 7 action 18 (srv20-fileprocessor_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=97 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv23-orderprocessing_start_0 'create' on an-a02n01: Old event | magic=0:0;25:9:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.13 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 9 action 25 (srv23-orderprocessing_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=102 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv23-orderprocessing_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;27:10:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.13 source=process_graph_event:408 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 10 action 27 (srv23-orderprocessing_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=103 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by operation srv16-machine-interface_monitor_0 'create' on an-a02n01: Old event | magic=0:7;12:0:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.383.13 source=process_graph_event:412 complete=false Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 0 action 12 (srv16-machine-interface_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=49 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/812, version=0.383.13) Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.383.13 2 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.383.14 (null) Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/813, version=0.383.14) Jul 23 16:31:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/814, version=0.383.14) Jul 23 16:31:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.383.14 2 Jul 23 16:31:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.383.15 (null) Jul 23 16:31:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=15 Jul 23 16:31:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource Jul 23 16:31:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=31:140:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=0:1;31:140:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=176, @rc-code= Jul 23 16:31:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: Jul 23 16:31:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/186, version=0.383.16) Jul 23 16:31:24 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 186 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:31:24 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:31:24 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:31:24 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by status-2-fail-count-apc_snmp_node1_an-pdu03.start_0 doing create fail-count-apc_snmp_node1_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.383.16 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=false Jul 23 16:31:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.383.16 2 Jul 23 16:31:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.383.17 (null) Jul 23 16:31:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=17 Jul 23 16:31:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 16:31:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/187, version=0.383.17) Jul 23 16:31:24 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by status-2-last-failure-apc_snmp_node1_an-pdu03.start_0 doing create last-failure-apc_snmp_node1_an-pdu03#start_0=1627083084: Transient attribute change | cib=0.383.17 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=false Jul 23 16:31:24 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 187 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:31:24 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:31:24 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627083084 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.383.17 2 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.383.18 (null) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=18 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=33:140:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=0:1;33:140:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=177, @rc-code= Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/188, version=0.383.19) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 188 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.383.19 2 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.383.20 (null) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=20 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/189, version=0.383.20) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 189 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627083085 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by status-2-fail-count-apc_snmp_node1_an-pdu04.start_0 doing create fail-count-apc_snmp_node1_an-pdu04#start_0=INFINITY: Transient attribute change | cib=0.383.19 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 140 aborted by status-2-last-failure-apc_snmp_node1_an-pdu04.start_0 doing create last-failure-apc_snmp_node1_an-pdu04#start_0=1627083085: Transient attribute change | cib=0.383.20 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:31:11 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:31:11 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:31:11 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:31:11 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n01 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n02 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n02 -> an-a02n01 ) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu04 ( an-a02n02 ) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 141, saving inputs in /var/lib/pacemaker/pengine/pe-input-161.bz2 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627083085-489 is obsolete Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:31:11 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:31:11 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:31:11 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:31:11 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n01 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n01 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n02 -> an-a02n01 ) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu04 ( an-a02n02 -> an-a02n01 ) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 142, saving inputs in /var/lib/pacemaker/pengine/pe-input-162.bz2 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 142 (ref=pe_calc-dc-1627083085-490) derived from /var/lib/pacemaker/pengine/pe-input-162.bz2 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 on an-a02n02 | action 11 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu04_stop_0 on an-a02n02 | action 10 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.383.20 2 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.383.21 (null) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=21 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @transition-key=11:142:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;11:142:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/228, version=0.383.21) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.383.21 2 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.383.22 (null) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=22 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_stop_0, @operation=stop, @transition-key=10:142:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;10:142:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/229, version=0.383.22) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.383.22 2 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.383.23 (null) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=23 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;11:142:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=178, @rc-code=0, @op-status=0 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/230, version=0.383.23) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 142 action 11 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=178 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 locally on an-a02n01 | action 33 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=33:142:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_start_0 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.383.23 2 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.383.24 (null) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=24 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:0;10:142:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=179, @rc-code=0, @op-status=0 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/231, version=0.383.24) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/819) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu03 action:start call_id:190 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 142 action 10 (apc_snmp_node1_an-pdu04_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=179 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu04_start_0 locally on an-a02n01 | action 35 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node1_an-pdu04 on an-a02n01 | transition_key=35:142:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu04_start_0 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.383.24 2 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.383.25 (null) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=25 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=33:142:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;33:142:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/819, version=0.383.25) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/820) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu04 action:start call_id:191 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.383.25 2 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.383.26 (null) Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=26 Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=35:142:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;35:142:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:31:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/820, version=0.383.26) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.383.26 2 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.0 (null) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=384, @num_updates=0 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu04']/instance_attributes[@id='apc_snmp_node2_an-pdu04-instance_attributes']/nvpair[@id='apc_snmp_node2_an-pdu04-instance_attributes-ip']: @value=10.201.2.4 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu04']/instance_attributes[@id='apc_snmp_node2_an-pdu04-instance_attributes']/nvpair[@id='apc_snmp_node2_an-pdu04-instance_attributes-pcmk_host_list']: @value=an-a02n01 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu04']/instance_attributes[@id='apc_snmp_node2_an-pdu04-instance_attributes']/nvpair[@id='apc_snmp_node2_an-pdu04-instance_attributes-port']: @value=4 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.383.26 -> 0.384.0 from an-a02n01 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.384.0) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 142 aborted by apc_snmp_node2_an-pdu04-instance_attributes-ip doing modify ip=10.201.2.4: Configuration change | cib=0.384.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu04']/instance_attributes[@id='apc_snmp_node2_an-pdu04-instance_attributes']/nvpair[@id='apc_snmp_node2_an-pdu04-instance_attributes-ip'] complete=false Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: modify nvpair[@id='apc_snmp_node2_an-pdu04-instance_attributes-ip'] Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.384.0 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 190 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 191 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 192 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 193 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 194 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 195 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 196 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 197 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/823) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/824) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/190) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/191) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/192) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/193) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/194) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-85.raw Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/823, version=0.384.0) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.384.0 of the CIB to disk (digest: bdf8d21f829d9c2d6e140659a7aa73d4) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.3Q1fvP (digest: /var/lib/pacemaker/cib/cib.BR2ZLR) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.0 2 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.1 (null) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/824, version=0.384.1) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.1 2 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.2 (null) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/190, version=0.384.2) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 190 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.2 2 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.3 (null) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/191, version=0.384.3) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 191 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.3 2 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.4 (null) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/192, version=0.384.4) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 192 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.4 2 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.5 (null) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/193, version=0.384.5) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 193 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627083084 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.5 2 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.6 (null) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/194, version=0.384.6) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 194 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/195) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/196) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/197) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.6 2 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.7 (null) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/195, version=0.384.7) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 195 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.7 2 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.8 (null) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/196, version=0.384.8) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 196 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.8 2 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.9 (null) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/197, version=0.384.9) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 197 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627083085 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/825, version=0.384.9) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/826) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/828) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/826, version=0.384.9) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/828, version=0.384.9) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/830) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-17 offer to an-a02n02 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-17 offer to an-a02n01 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-17 requests from 2 outstanding nodes Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/830, version=0.384.9) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/832) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/832, version=0.384.9) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: 1fe4d0ea47b463baf762b85a5fd80287 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.384.9 with 0.384.9 from an-a02n01 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/835, version=0.384.9) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/836) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/837) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/836, version=0.384.9) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/837, version=0.384.9) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-86.raw Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4183316] exited with status 0 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.384.0 of the CIB to disk (digest: bdf8d21f829d9c2d6e140659a7aa73d4) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.fg264X (digest: /var/lib/pacemaker/cib/cib.GHePxY) Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4183318] exited with status 0 Jul 23 16:31:27 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4183319] exited with status 0 Jul 23 16:31:28 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4183320] exited with status 0 Jul 23 16:31:28 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4183321] exited with status 0 Jul 23 16:31:28 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4183323] exited with status 0 Jul 23 16:31:28 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4183324] exited with status 0 Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4183335] exited with status 0 Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 838) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 840) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/838) Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/839) Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/840) Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.9 2 Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.10 (null) Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/838, version=0.384.10) Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.10 2 Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.11 (null) Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/839, version=0.384.11) Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.11 2 Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.12 (null) Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/840, version=0.384.12) Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4183349] exited with status 0 Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4183350] exited with status 0 Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4183351] exited with status 0 Jul 23 16:31:29 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4183352] exited with status 0 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4183359] exited with status 0 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4183367] exited with status 0 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4183368] exited with status 0 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4183369] exited with status 0 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/841) Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by deletion of lrm[@id='2']: Resource state removal | cib=0.384.10 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2'] complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv14-file-processor_monitor_0 'create' on an-a02n02: Old event | magic=0:7;20:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.11 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 20 (srv14-file-processor_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=41 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation delay_node1_stop_0 'create' on an-a02n02: Old event | magic=0:0;20:123:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.11 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 123 action 20 (delay_node1_stop_0 on an-a02n02): arrived really late | rc=0 target-rc=0 call-id=142 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation delay_node2_start_0 'create' on an-a02n02: Old event | magic=0:0;16:123:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.11 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 123 action 16 (delay_node2_start_0 on an-a02n02): arrived really late | rc=0 target-rc=0 call-id=144 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation delay_node2_monitor_60000 'create' on an-a02n02: Change in recurring result | magic=0:0;17:123:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.11 source=process_graph_event:408 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 123 action 17 (delay_node2_monitor_60000 on an-a02n02): arrived after initial scheduling | rc=0 target-rc=0 call-id=145 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation apc_snmp_node1_an-pdu04_start_0 'create' on an-a02n02: Old event | magic=0:1;33:140:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.11 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 140 action 33 (apc_snmp_node1_an-pdu04_start_0 on an-a02n02): arrived really late | rc=1 target-rc=0 call-id=177 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation apc_snmp_node1_an-pdu03_start_0 'create' on an-a02n02: Old event | magic=0:1;31:140:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.11 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 140 action 31 (apc_snmp_node1_an-pdu03_start_0 on an-a02n02): arrived really late | rc=1 target-rc=0 call-id=176 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv22-machineinterface_monitor_0 'create' on an-a02n02: Old event | magic=0:7;28:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.11 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 28 (srv22-machineinterface_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=73 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv13-psql_monitor_0 'create' on an-a02n02: Old event | magic=0:7;19:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.11 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 19 (srv13-psql_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=37 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv24-inventory_monitor_0 'create' on an-a02n02: Old event | magic=0:7;30:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.11 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 30 (srv24-inventory_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=81 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv15-tickets_monitor_0 'create' on an-a02n02: Old event | magic=0:7;21:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.11 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 21 (srv15-tickets_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=45 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv18-inventory_monitor_0 'create' on an-a02n02: Old event | magic=0:7;24:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.11 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 24 (srv18-inventory_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=57 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv21-tickets_monitor_0 'create' on an-a02n02: Old event | magic=0:7;27:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.11 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 27 (srv21-tickets_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=69 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv17-order-processing_monitor_0 'create' on an-a02n02: Old event | magic=0:7;23:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.11 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 23 (srv17-order-processing_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=53 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation ipmilan_node1_start_0 'create' on an-a02n02: Old event | magic=0:0;22:122:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.11 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 122 action 22 (ipmilan_node1_start_0 on an-a02n02): arrived really late | rc=0 target-rc=0 call-id=137 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation ipmilan_node1_monitor_60000 'create' on an-a02n02: Change in recurring result | magic=0:0;23:122:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.11 source=process_graph_event:408 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 122 action 23 (ipmilan_node1_monitor_60000 on an-a02n02): arrived after initial scheduling | rc=0 target-rc=0 call-id=138 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation ipmilan_node2_stop_0 'create' on an-a02n02: Old event | magic=0:0;16:122:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.11 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 122 action 16 (ipmilan_node2_stop_0 on an-a02n02): arrived really late | rc=0 target-rc=0 call-id=136 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv19-uls_monitor_0 'create' on an-a02n02: Old event | magic=0:7;25:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.11 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 25 (srv19-uls_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=61 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation apc_snmp_node2_an-pdu03_monitor_0 'create' on an-a02n02: Old event | magic=0:7;17:133:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.11 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 133 action 17 (apc_snmp_node2_an-pdu03_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=164 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation apc_snmp_node2_an-pdu04_monitor_0 'create' on an-a02n02: Old event | magic=0:7;12:138:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.11 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 138 action 12 (apc_snmp_node2_an-pdu04_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=175 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv20-fileprocessor_monitor_0 'create' on an-a02n02: Old event | magic=0:7;26:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.11 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 26 (srv20-fileprocessor_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=65 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv16-machine-interface_monitor_0 'create' on an-a02n02: Old event | magic=0:7;22:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.11 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 22 (srv16-machine-interface_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=49 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv23-orderprocessing_monitor_0 'create' on an-a02n02: Old event | magic=0:7;29:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.11 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 29 (srv23-orderprocessing_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=77 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by deletion of lrm[@id='1']: Resource state removal | cib=0.384.12 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1'] complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted: Peer Cancelled | source=do_te_invoke:129 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.12 2 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.13 (null) Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/841, version=0.384.13) Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv14-file-processor_monitor_0 'create' on an-a02n01: Old event | magic=0:7;10:0:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.13 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 0 action 10 (srv14-file-processor_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=41 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation delay_node1_start_0 'create' on an-a02n01: Old event | magic=0:0;21:123:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.13 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 123 action 21 (delay_node1_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=164 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation delay_node1_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;22:123:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.13 source=process_graph_event:408 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 123 action 22 (delay_node1_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=165 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/844) Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation delay_node2_stop_0 'create' on an-a02n01: Old event | magic=0:0;15:123:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.13 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 123 action 15 (delay_node2_stop_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=163 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation apc_snmp_node1_an-pdu04_monitor_0 'create' on an-a02n01: Old event | magic=0:7;13:133:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.13 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 133 action 13 (apc_snmp_node1_an-pdu04_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=179 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation apc_snmp_node1_an-pdu03_monitor_0 'create' on an-a02n01: Old event | magic=0:7;12:133:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.13 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 133 action 12 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=175 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv13-psql_monitor_0 'create' on an-a02n01: Old event | magic=0:7;9:0:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.13 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 0 action 9 (srv13-psql_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=37 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv22-machineinterface_start_0 'create' on an-a02n01: Old event | magic=0:0;22:8:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.13 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 8 action 22 (srv22-machineinterface_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=100 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv22-machineinterface_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;24:9:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.13 source=process_graph_event:408 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/845) Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 9 action 24 (srv22-machineinterface_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=101 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv24-inventory_start_0 'create' on an-a02n01: Old event | magic=0:0;28:10:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.13 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 10 action 28 (srv24-inventory_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=104 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv24-inventory_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;29:10:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.13 source=process_graph_event:408 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 10 action 29 (srv24-inventory_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=105 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv15-tickets_monitor_0 'create' on an-a02n01: Old event | magic=0:7;11:0:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.13 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 0 action 11 (srv15-tickets_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=45 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv18-inventory_monitor_0 'create' on an-a02n01: Old event | magic=0:7;14:0:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.13 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/846) Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 0 action 14 (srv18-inventory_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=57 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv21-tickets_start_0 'create' on an-a02n01: Old event | magic=0:0;19:7:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.13 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 7 action 19 (srv21-tickets_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=98 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv21-tickets_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;21:8:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.13 source=process_graph_event:408 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 8 action 21 (srv21-tickets_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=99 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv17-order-processing_monitor_0 'create' on an-a02n01: Old event | magic=0:7;13:0:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.13 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 0 action 13 (srv17-order-processing_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=53 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation ipmilan_node1_stop_0 'create' on an-a02n01: Old event | magic=0:0;21:122:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.13 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 122 action 21 (ipmilan_node1_stop_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=159 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation ipmilan_node2_start_0 'create' on an-a02n01: Old event | magic=0:0;17:122:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.13 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 122 action 17 (ipmilan_node2_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=160 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation ipmilan_node2_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;18:122:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.13 source=process_graph_event:408 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 122 action 18 (ipmilan_node2_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=161 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv19-uls_start_0 'create' on an-a02n01: Old event | magic=0:0;13:5:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.13 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 5 action 13 (srv19-uls_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=94 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv19-uls_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;15:6:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.13 source=process_graph_event:408 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 6 action 15 (srv19-uls_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=95 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation apc_snmp_node2_an-pdu03_monitor_0 'create' on an-a02n01: Old event | magic=0:7;14:133:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.13 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 133 action 14 (apc_snmp_node2_an-pdu03_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=183 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation apc_snmp_node2_an-pdu04_monitor_0 'create' on an-a02n01: Old event | magic=0:7;11:138:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.13 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 138 action 11 (apc_snmp_node2_an-pdu04_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=189 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv20-fileprocessor_start_0 'create' on an-a02n01: Old event | magic=0:0;16:6:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.13 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 6 action 16 (srv20-fileprocessor_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=96 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv20-fileprocessor_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;18:7:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.13 source=process_graph_event:408 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 7 action 18 (srv20-fileprocessor_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=97 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv23-orderprocessing_start_0 'create' on an-a02n01: Old event | magic=0:0;25:9:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.13 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 9 action 25 (srv23-orderprocessing_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=102 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv23-orderprocessing_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;27:10:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.13 source=process_graph_event:408 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 10 action 27 (srv23-orderprocessing_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=103 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by operation srv16-machine-interface_monitor_0 'create' on an-a02n01: Old event | magic=0:7;12:0:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.384.13 source=process_graph_event:412 complete=false Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 0 action 12 (srv16-machine-interface_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=49 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/844, version=0.384.13) Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.13 2 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.14 (null) Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/845, version=0.384.14) Jul 23 16:31:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/846, version=0.384.14) Jul 23 16:31:32 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4183290] error output [ 2021-07-23 16:31:32,114 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:31:32 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4183290] error output [ ] Jul 23 16:31:32 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4183290] error output [ ] Jul 23 16:31:32 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4183290] error output [ 2021-07-23 16:31:32,114 ERROR: Please use '-h' for usage ] Jul 23 16:31:32 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4183290] error output [ ] Jul 23 16:31:32 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4183290] stderr: [ 2021-07-23 16:31:32,114 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:31:32 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4183290] stderr: [ ] Jul 23 16:31:32 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4183290] stderr: [ ] Jul 23 16:31:32 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4183290] stderr: [ 2021-07-23 16:31:32,114 ERROR: Please use '-h' for usage ] Jul 23 16:31:32 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4183290] stderr: [ ] Jul 23 16:31:32 an-a02n01.alteeve.com pacemaker-fenced [121629] (internal_stonith_action_execute) info: Attempt 2 to execute fence_apc_snmp (monitor). remaining timeout is 13 Jul 23 16:31:33 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4183291] error output [ 2021-07-23 16:31:32,114 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 16:31:33 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4183291] error output [ ] Jul 23 16:31:33 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4183291] error output [ ] Jul 23 16:31:33 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4183291] error output [ 2021-07-23 16:31:32,114 ERROR: Please use '-h' for usage ] Jul 23 16:31:33 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4183291] error output [ ] Jul 23 16:31:33 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4183291] stderr: [ 2021-07-23 16:31:32,114 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 16:31:33 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4183291] stderr: [ ] Jul 23 16:31:33 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4183291] stderr: [ ] Jul 23 16:31:33 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4183291] stderr: [ 2021-07-23 16:31:32,114 ERROR: Please use '-h' for usage ] Jul 23 16:31:33 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4183291] stderr: [ ] Jul 23 16:31:33 an-a02n01.alteeve.com pacemaker-fenced [121629] (internal_stonith_action_execute) info: Attempt 2 to execute fence_apc_snmp (monitor). remaining timeout is 12 Jul 23 16:31:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 56ec96a1f458c00e3991a689ee4a5f8d for 0.384.14 (0x561976307d10 0) Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4183385] error output [ 2021-07-23 16:31:39,274 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4183385] error output [ ] Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4183385] error output [ ] Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4183385] error output [ 2021-07-23 16:31:39,275 ERROR: Please use '-h' for usage ] Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4183385] error output [ ] Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4183385] stderr: [ 2021-07-23 16:31:39,274 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4183385] stderr: [ ] Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4183385] stderr: [ ] Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4183385] stderr: [ 2021-07-23 16:31:39,275 ERROR: Please use '-h' for usage ] Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4183385] stderr: [ ] Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_remaining_timeout) info: Attempted to execute agent fence_apc_snmp (monitor) the maximum number of times (2) allowed Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_operation) notice: Operation 'monitor' [4183385] using apc_snmp_node1_an-pdu03 returned -201 (Generic Pacemaker error) Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu03 start (call 190) exited with status 1 (execution time 13321ms, queue time 0ms) Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for apc_snmp_node1_an-pdu03 on an-a02n01: error | rc=1 call=190 key=apc_snmp_node1_an-pdu03_start_0 confirmed=true cib-update=847 Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted: Peer Cancelled | source=do_te_invoke:129 complete=false Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/847) Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.14 2 Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.15 (null) Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=15 Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=33:142:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=0:1;33:142:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=190, @rc-code= Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/199, version=0.384.16) Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by status-1-fail-count-apc_snmp_node1_an-pdu03.start_0 doing create fail-count-apc_snmp_node1_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.384.16 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=false Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 199 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/200) Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.16 2 Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.17 (null) Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=17 Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/200, version=0.384.17) Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by status-1-last-failure-apc_snmp_node1_an-pdu03.start_0 doing create last-failure-apc_snmp_node1_an-pdu03#start_0=1627083099: Transient attribute change | cib=0.384.17 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=false Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 200 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=1627083099 Jul 23 16:31:39 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627083084 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4183615] error output [ 2021-07-23 16:31:40,276 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4183615] error output [ ] Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4183615] error output [ ] Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4183615] error output [ 2021-07-23 16:31:40,276 ERROR: Please use '-h' for usage ] Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4183615] error output [ ] Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4183615] stderr: [ 2021-07-23 16:31:40,276 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4183615] stderr: [ ] Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4183615] stderr: [ ] Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4183615] stderr: [ 2021-07-23 16:31:40,276 ERROR: Please use '-h' for usage ] Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4183615] stderr: [ ] Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_remaining_timeout) info: Attempted to execute agent fence_apc_snmp (monitor) the maximum number of times (2) allowed Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_operation) notice: Operation 'monitor' [4183615] using apc_snmp_node1_an-pdu04 returned -201 (Generic Pacemaker error) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu04 start (call 191) exited with status 1 (execution time 14321ms, queue time 0ms) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for apc_snmp_node1_an-pdu04 on an-a02n01: error | rc=1 call=191 key=apc_snmp_node1_an-pdu04_start_0 confirmed=true cib-update=848 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted: Peer Cancelled | source=do_te_invoke:129 complete=false Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/848) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.17 2 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.18 (null) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=18 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=35:142:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=0:1;35:142:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=191, @rc-code= Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/201, version=0.384.19) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 201 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.19 2 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.20 (null) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=20 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/202, version=0.384.20) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 202 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=1627083100 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627083085 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by status-1-fail-count-apc_snmp_node1_an-pdu04.start_0 doing create fail-count-apc_snmp_node1_an-pdu04#start_0=INFINITY: Transient attribute change | cib=0.384.19 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 142 aborted by status-1-last-failure-apc_snmp_node1_an-pdu04.start_0 doing create last-failure-apc_snmp_node1_an-pdu04#start_0=1627083100: Transient attribute change | cib=0.384.20 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 16:31:25 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 16:31:25 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:31:25 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:31:25 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:31:11 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:31:11 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n01 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu03 ( an-a02n01 ) due to node availability Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu04 ( an-a02n01 ) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 143, saving inputs in /var/lib/pacemaker/pengine/pe-input-163.bz2 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627083100-502 is obsolete Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 16:31:25 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 16:31:25 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:31:25 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:31:25 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:31:11 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:31:11 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu03 ( an-a02n01 ) due to node availability Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu04 ( an-a02n01 ) due to node availability Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 144, saving inputs in /var/lib/pacemaker/pengine/pe-input-164.bz2 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 144 (ref=pe_calc-dc-1627083100-503) derived from /var/lib/pacemaker/pengine/pe-input-164.bz2 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 locally on an-a02n01 | action 3 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=3:144:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_stop_0 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu03 action:stop call_id:192 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu04_stop_0 locally on an-a02n01 | action 2 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node1_an-pdu04 on an-a02n01 | transition_key=2:144:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu04_stop_0 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/853) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/854) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu03 stop (call 192) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu04 action:stop call_id:193 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu04 stop (call 193) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01: ok | rc=0 call=192 key=apc_snmp_node1_an-pdu03_stop_0 confirmed=true cib-update=855 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/855) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node1_an-pdu04 on an-a02n01: ok | rc=0 call=193 key=apc_snmp_node1_an-pdu04_stop_0 confirmed=true cib-update=856 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/856) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.20 2 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.21 (null) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=21 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @transition-key=3:144:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;3:144:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1627 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/853, version=0.384.21) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.21 2 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.22 (null) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=22 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_stop_0, @operation=stop, @transition-key=2:144:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;2:144:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1627 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/854, version=0.384.22) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.22 2 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.23 (null) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=23 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;3:144:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=192, @rc-code=0, @op-status=0 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/855, version=0.384.23) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 144 action 3 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=192 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.23 2 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.24 (null) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=24 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:0;2:144:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=193, @rc-code=0, @op-status=0 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/856, version=0.384.24) Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 144 action 2 (apc_snmp_node1_an-pdu04_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=193 Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 144 (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-164.bz2): Complete Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 16:31:40 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/crmd/857) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 203 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n01 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node1_an-pdu03#start_0' delayed: update 203 in progress Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: 1627083099 -> (unset) | from an-a02n01 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 204 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: 1627083084 -> (unset) | from an-a02n01 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node1_an-pdu03#start_0' delayed: update 204 in progress Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/203) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 205 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n01 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node1_an-pdu04#start_0' delayed: update 205 in progress Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]: 1627083100 -> (unset) | from an-a02n01 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/204) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 206 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]: 1627083085 -> (unset) | from an-a02n01 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node1_an-pdu04#start_0' delayed: update 206 in progress Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/205) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.24 2 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.25 (null) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=25 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/857, version=0.384.24) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-controld [121633] (delete_resource) info: Removing resource apc_snmp_node1_an-pdu03 from executor for afeb41b8-0453-4295-a7bc-4e284d3cdcd7 as root Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_resource_history) info: Clearing resource history for apc_snmp_node1_an-pdu03 on an-a02n01 (via CIB call 858) | xpath=//node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-controld [121633] (notify_deleted) info: Notifying afeb41b8-0453-4295-a7bc-4e284d3cdcd7 on an-a02n01 that apc_snmp_node1_an-pdu03 was deleted Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.24 2 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.25 (null) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0'] Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=25 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/203, version=0.384.25) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 203 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 207 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.25 2 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.26 (null) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0'] Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=26 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/204, version=0.384.26) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 204 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627083084 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 208 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.26 2 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.27 (null) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu04.start_0'] Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=27 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/205, version=0.384.27) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 205 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/crmd/858) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 209 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/206) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/207) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/208) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/209) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 144 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.384.25 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 144 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.384.26 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 144 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.384.27 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu04.start_0'] complete=true Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.27 2 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.28 (null) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=28 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/858, version=0.384.28) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.28 2 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.29 (null) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu04.start_0'] Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=29 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/206, version=0.384.29) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 206 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627083085 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 210 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.29 2 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.30 (null) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=30 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/207, version=0.384.30) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 207 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.30 2 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.31 (null) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=31 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/208, version=0.384.31) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 208 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.31 2 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.384.32 (null) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu04.start_0'] Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=32 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/209, version=0.384.32) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 209 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/860) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] to all (origin=local/crmd/864) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/210) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.384.32 2 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.0 (null) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=385, @num_updates=0 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']: @value=1627083104 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/860, version=0.385.0) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.0 2 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.1 (null) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/234, version=0.385.0) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.0 2 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.1 (null) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n01/crmd/864, version=0.385.0) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-controld [121633] (delete_resource) info: Removing resource apc_snmp_node1_an-pdu04 from executor for afeb41b8-0453-4295-a7bc-4e284d3cdcd7 as root Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_resource_history) info: Clearing resource history for apc_snmp_node1_an-pdu04 on an-a02n01 (via CIB call 865) | xpath=//node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-controld [121633] (notify_deleted) info: Notifying afeb41b8-0453-4295-a7bc-4e284d3cdcd7 on an-a02n01 that apc_snmp_node1_an-pdu04 was deleted Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.0 2 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.1 (null) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu04.start_0'] Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/210, version=0.385.1) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 210 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.1 2 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.2 (null) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/235, version=0.385.2) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n02/crmd/237, version=0.385.2) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.2 2 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.3 (null) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n02/crmd/239, version=0.385.2) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] to all (origin=local/crmd/865) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 144 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu03']: Resource state removal | cib=0.384.28 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] complete=true Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 144 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.384.29 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu04.start_0'] complete=true Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 144 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.384.30 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 144 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.384.31 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 144 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.384.32 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu04.start_0'] complete=true Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.2 2 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.3 (null) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n02/crmd/240, version=0.385.3) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n02/crmd/242, version=0.385.3) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.3 2 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.4 (null) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n01/crmd/865, version=0.385.4) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/867) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/867, version=0.385.4) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-87.raw Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.385.0 of the CIB to disk (digest: 0dcc54c059b4db0b749e49000463063a) Jul 23 16:31:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.ikrL1s (digest: /var/lib/pacemaker/cib/cib.9sliur) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 144 aborted by cib-bootstrap-options-last-lrm-refresh doing modify last-lrm-refresh=1627083104: Configuration change | cib=0.385.0 source=te_update_diff_v2:474 path=/cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh'] complete=true Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 144 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.385.1 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu04.start_0'] complete=true Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 144 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu03']: Resource state removal | cib=0.385.2 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] complete=true Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 144 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu04']: Resource state removal | cib=0.385.3 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] complete=true Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 144 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu04']: Resource state removal | cib=0.385.4 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] complete=true Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n02 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu04 ( an-a02n02 ) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 145, saving inputs in /var/lib/pacemaker/pengine/pe-input-165.bz2 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 145 (ref=pe_calc-dc-1627083105-508) derived from /var/lib/pacemaker/pengine/pe-input-165.bz2 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu03_monitor_0 on an-a02n02 | action 13 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu03_monitor_0 locally on an-a02n01 | action 11 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_get_rsc_info) info: Agent information for 'apc_snmp_node1_an-pdu03' not in cache Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_rsc_register) info: Cached agent information for 'apc_snmp_node1_an-pdu03' Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=11:145:7:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_monitor_0 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu04_monitor_0 on an-a02n02 | action 14 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu04_monitor_0 locally on an-a02n01 | action 12 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_get_rsc_info) info: Agent information for 'apc_snmp_node1_an-pdu04' not in cache Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_rsc_register) info: Cached agent information for 'apc_snmp_node1_an-pdu04' Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for apc_snmp_node1_an-pdu04 on an-a02n01 | transition_key=12:145:7:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu04_monitor_0 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of probe operation for apc_snmp_node1_an-pdu03 on an-a02n01: not running | rc=7 call=199 key=apc_snmp_node1_an-pdu03_monitor_0 confirmed=true cib-update=881 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/879) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of probe operation for apc_snmp_node1_an-pdu04 on an-a02n01: not running | rc=7 call=203 key=apc_snmp_node1_an-pdu04_monitor_0 confirmed=true cib-update=882 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/880) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/881) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/882) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.4 2 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.5 (null) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/879, version=0.385.5) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.5 2 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.6 (null) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/880, version=0.385.6) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.6 2 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.7 (null) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:7;11:145:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=199, @rc-code=7, @op-status=0 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/881, version=0.385.7) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 145 action 11 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n01) confirmed: not running | rc=7 call-id=199 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.7 2 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.8 (null) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:7;12:145:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=203, @rc-code=7, @op-status=0 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/882, version=0.385.8) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 145 action 12 (apc_snmp_node1_an-pdu04_monitor_0 on an-a02n01) confirmed: not running | rc=7 call-id=203 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.8 2 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.9 (null) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/243, version=0.385.9) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.9 2 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.10 (null) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:7;13:145:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=185, @rc-code=7, @op-status=0 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/244, version=0.385.10) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 145 action 13 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=185 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 on an-a02n02 | action 35 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.10 2 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.11 (null) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/245, version=0.385.11) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.11 2 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.12 (null) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:7;14:145:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=189, @rc-code=7, @op-status=0 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/246, version=0.385.12) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 145 action 14 (apc_snmp_node1_an-pdu04_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=189 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu04_start_0 on an-a02n02 | action 37 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.12 2 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.13 (null) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @transition-key=35:145:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;35:145:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/247, version=0.385.13) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.13 2 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.14 (null) Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_start_0, @operation=start, @transition-key=37:145:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;37:145:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1 Jul 23 16:31:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/248, version=0.385.14) Jul 23 16:31:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 300aaf55837b4aee0996097c3e7b165b for 0.385.14 (0x561975c4a6f0 0) Jul 23 16:31:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.14 2 Jul 23 16:31:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.15 (null) Jul 23 16:31:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=15 Jul 23 16:31:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:1;35:145:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=190, @rc-code=1, @op-status=0, @exec-time=13321 Jul 23 16:31:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: Jul 23 16:31:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/211, version=0.385.16) Jul 23 16:31:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 211 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:31:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:31:58 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 145 aborted by status-2-fail-count-apc_snmp_node1_an-pdu03.start_0 doing create fail-count-apc_snmp_node1_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.385.16 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=false Jul 23 16:31:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:31:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.16 2 Jul 23 16:31:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.17 (null) Jul 23 16:31:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=17 Jul 23 16:31:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 16:31:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/212, version=0.385.17) Jul 23 16:31:58 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 145 aborted by status-2-last-failure-apc_snmp_node1_an-pdu03.start_0 doing create last-failure-apc_snmp_node1_an-pdu03#start_0=1627083118: Transient attribute change | cib=0.385.17 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=false Jul 23 16:31:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 212 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:31:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:31:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627083118 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.17 2 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.18 (null) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=18 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:1;37:145:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=191, @rc-code=1, @op-status=0, @exec-time=14322 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/213, version=0.385.19) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 213 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.19 2 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.20 (null) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=20 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/214, version=0.385.20) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 214 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627083119 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 145 aborted by status-2-fail-count-apc_snmp_node1_an-pdu04.start_0 doing create fail-count-apc_snmp_node1_an-pdu04#start_0=INFINITY: Transient attribute change | cib=0.385.19 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 145 aborted by status-2-last-failure-apc_snmp_node1_an-pdu04.start_0 doing create last-failure-apc_snmp_node1_an-pdu04#start_0=1627083119: Transient attribute change | cib=0.385.20 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:31:45 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:31:45 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:31:45 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:31:45 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n01 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n02 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n02 -> an-a02n01 ) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu04 ( an-a02n02 ) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 146, saving inputs in /var/lib/pacemaker/pengine/pe-input-166.bz2 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627083119-515 is obsolete Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:31:45 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:31:45 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:31:45 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:31:45 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n01 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n01 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n02 -> an-a02n01 ) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu04 ( an-a02n02 -> an-a02n01 ) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 147, saving inputs in /var/lib/pacemaker/pengine/pe-input-167.bz2 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 147 (ref=pe_calc-dc-1627083119-516) derived from /var/lib/pacemaker/pengine/pe-input-167.bz2 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 on an-a02n02 | action 11 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu04_stop_0 on an-a02n02 | action 12 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.20 2 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.21 (null) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=21 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @transition-key=11:147:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;11:147:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/251, version=0.385.21) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.21 2 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.22 (null) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=22 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_stop_0, @operation=stop, @transition-key=12:147:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;12:147:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/252, version=0.385.22) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.22 2 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.23 (null) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=23 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;11:147:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=192, @rc-code=0, @op-status=0 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/253, version=0.385.23) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 147 action 11 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=192 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 locally on an-a02n01 | action 33 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=33:147:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_start_0 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.23 2 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.24 (null) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=24 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:0;12:147:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=193, @rc-code=0, @op-status=0 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/254, version=0.385.24) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/886) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu03 action:start call_id:204 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 147 action 12 (apc_snmp_node1_an-pdu04_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=193 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu04_start_0 locally on an-a02n01 | action 35 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node1_an-pdu04 on an-a02n01 | transition_key=35:147:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu04_start_0 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.24 2 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.25 (null) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=25 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @transition-key=33:147:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;33:147:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change= Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/886, version=0.385.25) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/887) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu04 action:start call_id:205 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.25 2 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.26 (null) Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=26 Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_start_0, @operation=start, @transition-key=35:147:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;35:147:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change= Jul 23 16:31:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/887, version=0.385.26) Jul 23 16:32:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 61f7419ff72011b759cbada739b2bc08 for 0.385.26 (0x561975c4a6f0 0) Jul 23 16:32:06 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4185795] error output [ 2021-07-23 16:32:06,056 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 16:32:06 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4185795] error output [ ] Jul 23 16:32:06 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4185795] error output [ ] Jul 23 16:32:06 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4185795] error output [ 2021-07-23 16:32:06,057 ERROR: Please use '-h' for usage ] Jul 23 16:32:06 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4185795] error output [ ] Jul 23 16:32:06 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4185795] stderr: [ 2021-07-23 16:32:06,056 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 16:32:06 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4185795] stderr: [ ] Jul 23 16:32:06 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4185795] stderr: [ ] Jul 23 16:32:06 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4185795] stderr: [ 2021-07-23 16:32:06,057 ERROR: Please use '-h' for usage ] Jul 23 16:32:06 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4185795] stderr: [ ] Jul 23 16:32:06 an-a02n01.alteeve.com pacemaker-fenced [121629] (internal_stonith_action_execute) info: Attempt 2 to execute fence_apc_snmp (monitor). remaining timeout is 13 Jul 23 16:32:06 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4185794] error output [ 2021-07-23 16:32:06,058 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:32:06 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4185794] error output [ ] Jul 23 16:32:06 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4185794] error output [ ] Jul 23 16:32:06 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4185794] error output [ 2021-07-23 16:32:06,058 ERROR: Please use '-h' for usage ] Jul 23 16:32:06 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4185794] error output [ ] Jul 23 16:32:06 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4185794] stderr: [ 2021-07-23 16:32:06,058 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:32:06 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4185794] stderr: [ ] Jul 23 16:32:06 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4185794] stderr: [ ] Jul 23 16:32:06 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4185794] stderr: [ 2021-07-23 16:32:06,058 ERROR: Please use '-h' for usage ] Jul 23 16:32:06 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4185794] stderr: [ ] Jul 23 16:32:06 an-a02n01.alteeve.com pacemaker-fenced [121629] (internal_stonith_action_execute) info: Attempt 2 to execute fence_apc_snmp (monitor). remaining timeout is 13 Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4186752] error output [ 2021-07-23 16:32:12,218 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4186752] error output [ ] Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4186752] error output [ ] Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4186752] error output [ 2021-07-23 16:32:12,218 ERROR: Please use '-h' for usage ] Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4186752] error output [ ] Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4186752] stderr: [ 2021-07-23 16:32:12,218 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4186752] stderr: [ ] Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4186752] stderr: [ ] Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4186752] stderr: [ 2021-07-23 16:32:12,218 ERROR: Please use '-h' for usage ] Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4186752] stderr: [ ] Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_remaining_timeout) info: Attempted to execute agent fence_apc_snmp (monitor) the maximum number of times (2) allowed Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_operation) notice: Operation 'monitor' [4186752] using apc_snmp_node1_an-pdu04 returned -201 (Generic Pacemaker error) Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu04 start (call 205) exited with status 1 (execution time 12323ms, queue time 0ms) Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for apc_snmp_node1_an-pdu04 on an-a02n01: error | rc=1 call=205 key=apc_snmp_node1_an-pdu04_start_0 confirmed=true cib-update=888 Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/888) Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.26 2 Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.27 (null) Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=27 Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:1;35:147:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=205, @rc-code=1, @op-status=0, @exec-time=12323 Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/215, version=0.385.28) Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 215 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 147 aborted by status-1-fail-count-apc_snmp_node1_an-pdu04.start_0 doing create fail-count-apc_snmp_node1_an-pdu04#start_0=INFINITY: Transient attribute change | cib=0.385.28 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=false Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.28 2 Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.29 (null) Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=29 Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/216, version=0.385.29) Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 147 aborted by status-1-last-failure-apc_snmp_node1_an-pdu04.start_0 doing create last-failure-apc_snmp_node1_an-pdu04#start_0=1627083132: Transient attribute change | cib=0.385.29 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=false Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 216 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=1627083132 Jul 23 16:32:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627083119 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4186853] error output [ 2021-07-23 16:32:13,217 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4186853] error output [ ] Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4186853] error output [ ] Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4186853] error output [ 2021-07-23 16:32:13,217 ERROR: Please use '-h' for usage ] Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4186853] error output [ ] Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4186853] stderr: [ 2021-07-23 16:32:13,217 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4186853] stderr: [ ] Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4186853] stderr: [ ] Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4186853] stderr: [ 2021-07-23 16:32:13,217 ERROR: Please use '-h' for usage ] Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4186853] stderr: [ ] Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_remaining_timeout) info: Attempted to execute agent fence_apc_snmp (monitor) the maximum number of times (2) allowed Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_operation) notice: Operation 'monitor' [4186853] using apc_snmp_node1_an-pdu03 returned -201 (Generic Pacemaker error) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu03 start (call 204) exited with status 1 (execution time 13321ms, queue time 0ms) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for apc_snmp_node1_an-pdu03 on an-a02n01: error | rc=1 call=204 key=apc_snmp_node1_an-pdu03_start_0 confirmed=true cib-update=889 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/889) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.29 2 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.30 (null) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=30 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:1;33:147:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=204, @rc-code=1, @op-status=0, @exec-time=13321 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/217, version=0.385.31) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 217 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.31 2 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.32 (null) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=32 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/218, version=0.385.32) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 218 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=1627083133 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627083118 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 147 aborted by status-1-fail-count-apc_snmp_node1_an-pdu03.start_0 doing create fail-count-apc_snmp_node1_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.385.31 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 147 aborted by status-1-last-failure-apc_snmp_node1_an-pdu03.start_0 doing create last-failure-apc_snmp_node1_an-pdu03#start_0=1627083133: Transient attribute change | cib=0.385.32 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:31:59 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:31:59 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 16:31:59 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 16:31:59 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:31:45 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:31:45 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n01 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n01 ) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu04 ( an-a02n01 ) due to node availability Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 148, saving inputs in /var/lib/pacemaker/pengine/pe-input-168.bz2 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627083133-521 is obsolete Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:31:59 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:31:59 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 16:31:59 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 16:31:59 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:31:45 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:31:45 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu03 ( an-a02n01 ) due to node availability Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu04 ( an-a02n01 ) due to node availability Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 149, saving inputs in /var/lib/pacemaker/pengine/pe-input-169.bz2 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 149 (ref=pe_calc-dc-1627083133-522) derived from /var/lib/pacemaker/pengine/pe-input-169.bz2 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 locally on an-a02n01 | action 9 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=9:149:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_stop_0 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu03 action:stop call_id:206 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu04_stop_0 locally on an-a02n01 | action 10 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node1_an-pdu04 on an-a02n01 | transition_key=10:149:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu04_stop_0 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/893) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu03 stop (call 206) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/894) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu04 action:stop call_id:207 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu04 stop (call 207) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01: ok | rc=0 call=206 key=apc_snmp_node1_an-pdu03_stop_0 confirmed=true cib-update=895 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/895) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node1_an-pdu04 on an-a02n01: ok | rc=0 call=207 key=apc_snmp_node1_an-pdu04_stop_0 confirmed=true cib-update=896 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/896) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.32 2 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.33 (null) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=33 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @transition-key=9:149:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;9:149:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1627 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/893, version=0.385.33) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.33 2 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.34 (null) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=34 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_stop_0, @operation=stop, @transition-key=10:149:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;10:149:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/894, version=0.385.34) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.34 2 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.35 (null) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=35 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;9:149:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=206, @rc-code=0, @op-status=0 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/895, version=0.385.35) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 149 action 9 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=206 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.35 2 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.385.36 (null) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=36 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:0;10:149:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=207, @rc-code=0, @op-status=0 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/896, version=0.385.36) Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 149 action 10 (apc_snmp_node1_an-pdu04_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=207 Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 149 (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-169.bz2): Complete Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 16:32:13 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:32:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 877f247b38f1b4aca1270effcb6f25a9 for 0.385.36 (0x561975c4a6f0 0) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.385.36 2 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.386.0 9c63a9fdb8cbfdb38f32ee48c38426d5 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=386, @num_updates=0 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03']/instance_attributes[@id='apc_snmp_node1_an-pdu03-instance_attributes']/nvpair[@id='apc_snmp_node1_an-pdu03-instance_attributes-ip']: @value=10.201.2.3 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03']/instance_attributes[@id='apc_snmp_node1_an-pdu03-instance_attributes']/nvpair[@id='apc_snmp_node1_an-pdu03-instance_attributes-port']: @value=3 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.385.36 -> 0.386.0 from an-a02n01 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.386.0) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 149 aborted by apc_snmp_node1_an-pdu03-instance_attributes-ip doing modify ip=10.201.2.3: Configuration change | cib=0.386.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03']/instance_attributes[@id='apc_snmp_node1_an-pdu03-instance_attributes']/nvpair[@id='apc_snmp_node1_an-pdu03-instance_attributes-ip'] complete=true Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 219 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 220 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 221 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 222 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 223 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 224 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 225 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 226 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/899) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/900) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: modify nvpair[@id='apc_snmp_node1_an-pdu03-instance_attributes-ip'] Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.386.0 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/219) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/220) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/221) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/222) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/223) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-88.raw Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.386.0 of the CIB to disk (digest: 9944a97b63370dd9845ebd07ae64f7d2) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.bfl9cx (digest: /var/lib/pacemaker/cib/cib.gxpDxx) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node1_an-pdu03' to device list (6 active devices) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/899, version=0.386.0) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.386.0 2 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.386.1 (null) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/900, version=0.386.1) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.386.1 2 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.386.2 (null) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/219, version=0.386.2) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 219 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.386.2 2 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.386.3 (null) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/220, version=0.386.3) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 220 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/221, version=0.386.3) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 221 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/222, version=0.386.3) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 222 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=1627083133 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627083118 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.386.3 2 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.386.4 (null) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/223, version=0.386.4) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 223 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/224) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/225) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/226) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.386.4 2 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.386.5 (null) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/224, version=0.386.5) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 224 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/225, version=0.386.5) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 225 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/226, version=0.386.5) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 226 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=1627083132 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627083119 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/902, version=0.386.5) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/903) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/905) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/903, version=0.386.5) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/905, version=0.386.5) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/907) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/909) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-18 offer to an-a02n02 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-18 offer to an-a02n01 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-18 requests from 2 outstanding nodes Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/907, version=0.386.5) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/909, version=0.386.5) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: 7616381fd4005227b8f07098664e2e91 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.386.5 with 0.386.5 from an-a02n01 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/912, version=0.386.5) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/913) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/914) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-89.raw Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/913, version=0.386.5) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.386.0 of the CIB to disk (digest: 9944a97b63370dd9845ebd07ae64f7d2) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.eF1bBG (digest: /var/lib/pacemaker/cib/cib.1j8bVF) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/914, version=0.386.5) Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4189554] exited with status 0 Jul 23 16:33:26 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4189556] exited with status 0 Jul 23 16:33:27 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4189557] exited with status 0 Jul 23 16:33:27 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4189558] exited with status 0 Jul 23 16:33:27 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4189559] exited with status 0 Jul 23 16:33:27 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4189561] exited with status 0 Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4189563] exited with status 0 Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4189570] exited with status 0 Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 915) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 917) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/915) Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/916) Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/917) Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.386.5 2 Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.386.6 (null) Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/915, version=0.386.6) Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.386.6 2 Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.386.7 (null) Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/916, version=0.386.7) Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.386.7 2 Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.386.8 (null) Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/917, version=0.386.8) Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4189720] exited with status 0 Jul 23 16:33:28 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4189721] exited with status 0 Jul 23 16:33:29 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4189732] exited with status 0 Jul 23 16:33:29 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4189741] exited with status 0 Jul 23 16:33:29 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4189769] exited with status 0 Jul 23 16:33:29 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4189779] exited with status 0 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4189780] exited with status 0 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4189933] exited with status 0 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/918) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 149 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.386.8 2 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.386.9 (null) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/918, version=0.386.9) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 149 aborted: History refresh | source=process_resource_updates:268 complete=true Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/921) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/922) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/923) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/921, version=0.386.9) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.386.9 2 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.386.10 (null) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/922, version=0.386.10) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/923, version=0.386.10) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 16:31:59 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n01 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now 5e7d5a6e316938194ebb385cfee4a6f2 (reload:3.10.2) 0:1;33:147:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n01 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:31:59 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:31:45 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n02 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now 5e7d5a6e316938194ebb385cfee4a6f2 (reload:3.10.2) 0:1;35:145:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:31:45 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 150, saving inputs in /var/lib/pacemaker/pengine/pe-input-170.bz2 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 150 (ref=pe_calc-dc-1627083210-532) derived from /var/lib/pacemaker/pengine/pe-input-170.bz2 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 on an-a02n02 | action 33 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-controld [121633] (te_crm_command) info: Executing crm-event (11) without waiting: clear_failcount on an-a02n02 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-controld [121633] (te_crm_command) info: Executing crm-event (2) locally without waiting: clear_failcount on an-a02n01 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-controld [121633] (update_attrd_clear_failures) info: Asking pacemaker-attrd to clear failure of all operations for apc_snmp_node1_an-pdu03 on cluster node an-a02n01 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 228 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: 1627083133 -> (unset) | from an-a02n01 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 229 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section /cib/status/node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0'] to all (origin=local/crmd/926) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n02 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node1_an-pdu03#start_0' delayed: update 228 in progress Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: 1627083118 -> (unset) | from an-a02n02 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node1_an-pdu03#start_0' delayed: update 229 in progress Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.386.10 2 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.386.11 (null) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=33:150:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;33:150:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/257, version=0.386.11) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.386.11 2 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.386.12 (null) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0'] Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section /cib/status/node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0']: OK (rc=0, origin=an-a02n02/crmd/258, version=0.386.12) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 150 aborted by deletion of lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0']: Resource operation removal | cib=0.386.12 source=process_op_deletion:358 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0'] complete=false Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.386.12 2 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.386.13 (null) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0'] Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section /cib/status/node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0']: OK (rc=0, origin=an-a02n01/crmd/926, version=0.386.13) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 150 aborted by deletion of lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0']: Resource operation removal | cib=0.386.13 source=process_op_deletion:358 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0'] complete=false Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/228) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/229) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.386.13 2 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.386.14 (null) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0'] Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/228, version=0.386.14) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 228 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 150 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.386.14 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0'] complete=false Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 230 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.386.14 2 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.386.15 (null) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0'] Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=15 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/229, version=0.386.15) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 229 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 150 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.386.15 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0'] complete=false Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627083118 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/230) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 231 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/231) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.386.15 2 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.386.16 (null) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=16 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/230, version=0.386.16) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 230 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 150 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.386.16 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] complete=false Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.386.16 2 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.386.17 (null) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=17 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/231, version=0.386.17) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 231 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 150 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.386.17 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] complete=false Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:33:30 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.386.17 2 Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.386.18 (null) Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=18 Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;33:150:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=194, @rc-code=0, @op-status=0, @exec-time=333 Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/259, version=0.386.18) Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 150 action 33 (apc_snmp_node1_an-pdu03_start_0 on an-a02n02) confirmed: ok | rc=0 call-id=194 Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 150 (Complete=3, Pending=0, Fired=0, Skipped=1, Incomplete=1, Source=/var/lib/pacemaker/pengine/pe-input-170.bz2): Stopped Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 16:31:59 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:31:45 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Started an-a02n02) Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 151, saving inputs in /var/lib/pacemaker/pengine/pe-input-171.bz2 Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 151 (ref=pe_calc-dc-1627083211-536) derived from /var/lib/pacemaker/pengine/pe-input-171.bz2 Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu03_monitor_60000 on an-a02n02 | action 33 Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.386.18 2 Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.386.19 (null) Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=19 Jul 23 16:33:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/947, version=0.387.9) Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/948) Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.387.9 2 Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.387.10 (null) Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/948, version=0.387.10) Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4191226] exited with status 0 Jul 23 16:33:45 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191227] exited with status 0 Jul 23 16:33:46 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191228] exited with status 0 Jul 23 16:33:46 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191238] exited with status 0 Jul 23 16:33:46 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4191239] exited with status 0 Jul 23 16:33:46 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191241] exited with status 0 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191242] exited with status 0 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191243] exited with status 0 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/949) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 151 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.387.10 2 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.387.11 (null) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/949, version=0.387.11) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 151 aborted: History refresh | source=process_resource_updates:268 complete=true Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/952) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/953) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/954) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/952, version=0.387.11) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.387.11 2 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.387.12 (null) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/953, version=0.387.12) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/954, version=0.387.12) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu04 on an-a02n01 changed: hash was dcfe282eb9c3d7f7189c538ed5e16421 vs. now f0c09ec1d269ad244b958c44ffdfd5e6 (reload:3.10.2) 0:1;35:147:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu04 on an-a02n01 because resource parameters have changed | apc_snmp_node1_an-pdu04_clear_failcount_0 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 16:31:59 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu04 on an-a02n02 changed: hash was dcfe282eb9c3d7f7189c538ed5e16421 vs. now f0c09ec1d269ad244b958c44ffdfd5e6 (reload:3.10.2) 0:1;37:145:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu04 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu04_clear_failcount_0 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:31:45 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n02 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Started an-a02n02) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu04 ( an-a02n02 ) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 152, saving inputs in /var/lib/pacemaker/pengine/pe-input-172.bz2 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 152 (ref=pe_calc-dc-1627083227-545) derived from /var/lib/pacemaker/pengine/pe-input-172.bz2 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu04_start_0 on an-a02n02 | action 36 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-controld [121633] (te_crm_command) info: Executing crm-event (11) without waiting: clear_failcount on an-a02n02 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-controld [121633] (te_crm_command) info: Executing crm-event (2) locally without waiting: clear_failcount on an-a02n01 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-controld [121633] (update_attrd_clear_failures) info: Asking pacemaker-attrd to clear failure of all operations for apc_snmp_node1_an-pdu04 on cluster node an-a02n01 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section /cib/status/node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_failure_0'] to all (origin=local/crmd/957) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 241 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/241) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]: 1627083132 -> (unset) | from an-a02n01 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 242 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n02 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node1_an-pdu04#start_0' delayed: update 241 in progress Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]: 1627083119 -> (unset) | from an-a02n02 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node1_an-pdu04#start_0' delayed: update 242 in progress Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.387.12 2 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.387.13 (null) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_failure_0'] Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section /cib/status/node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_failure_0']: OK (rc=0, origin=an-a02n01/crmd/957, version=0.387.13) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 152 aborted by deletion of lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_failure_0']: Resource operation removal | cib=0.387.13 source=process_op_deletion:358 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_failure_0'] complete=false Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.387.13 2 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.387.14 (null) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu04.start_0'] Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/241, version=0.387.14) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 241 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 152 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.387.14 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu04.start_0'] complete=false Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 243 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.387.14 2 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.387.15 (null) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=15 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=36:152:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;36:152:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/264, version=0.387.15) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.387.15 2 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.387.16 (null) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_failure_0'] Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=16 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section /cib/status/node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_failure_0']: OK (rc=0, origin=an-a02n02/crmd/265, version=0.387.16) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/242) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 152 aborted by deletion of lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_failure_0']: Resource operation removal | cib=0.387.16 source=process_op_deletion:358 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_failure_0'] complete=false Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/243) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.387.16 2 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.387.17 (null) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu04.start_0'] Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=17 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/242, version=0.387.17) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 242 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627083119 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 152 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.387.17 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu04.start_0'] complete=false Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 244 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.387.17 2 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.387.18 (null) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu04.start_0'] Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=18 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/243, version=0.387.18) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 243 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 152 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.387.18 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu04.start_0'] complete=false Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/244) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.387.18 2 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.387.19 (null) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu04.start_0'] Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=19 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/244, version=0.387.19) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 244 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 152 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.387.19 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu04.start_0'] complete=false Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.387.19 2 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.387.20 (null) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=20 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:0;36:152:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=196, @rc-code=0, @op-status=0, @exec-time=328 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/266, version=0.387.20) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 152 action 36 (apc_snmp_node1_an-pdu04_start_0 on an-a02n02) confirmed: ok | rc=0 call-id=196 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 152 (Complete=3, Pending=0, Fired=0, Skipped=1, Incomplete=1, Source=/var/lib/pacemaker/pengine/pe-input-172.bz2): Stopped Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n02 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Started an-a02n02) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Started an-a02n02) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 153, saving inputs in /var/lib/pacemaker/pengine/pe-input-173.bz2 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 153 (ref=pe_calc-dc-1627083227-549) derived from /var/lib/pacemaker/pengine/pe-input-173.bz2 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu04_monitor_60000 on an-a02n02 | action 36 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.387.20 2 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.387.21 (null) Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=21 Jul 23 16:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/978, version=0.388.11) Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.388.11 2 Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.388.12 (null) Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/979, version=0.388.12) Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4191279] exited with status 0 Jul 23 16:33:57 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191280] exited with status 0 Jul 23 16:33:58 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191281] exited with status 0 Jul 23 16:33:58 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191282] exited with status 0 Jul 23 16:33:58 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4191283] exited with status 0 Jul 23 16:33:58 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191285] exited with status 0 Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191286] exited with status 0 Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191287] exited with status 0 Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/980) Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 153 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/983) Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/984) Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/985) Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.388.12 2 Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.388.13 (null) Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/980, version=0.388.13) Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/983, version=0.388.13) Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.388.13 2 Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.388.14 (null) Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/984, version=0.388.14) Jul 23 16:33:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/985, version=0.388.14) Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 153 aborted: History refresh | source=process_resource_updates:268 complete=true Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Started an-a02n02) Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Started an-a02n02) Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 154, saving inputs in /var/lib/pacemaker/pengine/pe-input-174.bz2 Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 154 (ref=pe_calc-dc-1627083240-558) derived from /var/lib/pacemaker/pengine/pe-input-174.bz2 Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 154 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-174.bz2): Complete Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 16:34:00 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:34:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 242584edf843eae55c651adf1ead3538 for 0.388.14 (0x5619760acb80 0) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.388.14 2 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.389.0 (null) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=389, @num_updates=0 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu04']/instance_attributes[@id='apc_snmp_node2_an-pdu04-instance_attributes']/nvpair[@id='apc_snmp_node2_an-pdu04-instance_attributes-pcmk_host_list']: @value=an-a02n02 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.388.14 -> 0.389.0 from an-a02n01 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.389.0) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 154 aborted by apc_snmp_node2_an-pdu04-instance_attributes-pcmk_host_list doing modify pcmk_host_list=an-a02n02: Configuration change | cib=0.389.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu04']/instance_attributes[@id='apc_snmp_node2_an-pdu04-instance_attributes']/nvpair[@id='apc_snmp_node2_an-pdu04-instance_attributes-pcmk_host_list'] c Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 254 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 255 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 256 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 257 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 258 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 259 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 260 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 261 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/254) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: modify nvpair[@id='apc_snmp_node2_an-pdu04-instance_attributes-pcmk_host_list'] Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.389.0 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/255) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/256) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/257) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/258) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.389.0 2 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.389.1 (null) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/254, version=0.389.1) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 254 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.389.1 2 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.389.2 (null) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/255, version=0.389.2) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 255 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.389.2 2 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.389.3 (null) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/256, version=0.389.3) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 256 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.389.3 2 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.389.4 (null) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/257, version=0.389.4) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 257 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.389.4 2 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.389.5 (null) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/258, version=0.389.5) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 258 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/990) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/991) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/259) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/260) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/261) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/990, version=0.389.5) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.389.5 2 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.389.6 (null) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/991, version=0.389.6) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.389.6 2 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.389.7 (null) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/259, version=0.389.7) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 259 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.389.7 2 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.389.8 (null) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/260, version=0.389.8) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 260 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.389.8 2 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.389.9 (null) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/261, version=0.389.9) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 261 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-94.raw Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.389.0 of the CIB to disk (digest: 313d66cb7391d8b71c42470181bc65ce) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.XE5IWD (digest: /var/lib/pacemaker/cib/cib.NE8GYA) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/993, version=0.389.9) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/994) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/996) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/998) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-21 offer to an-a02n02 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1000) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-21 offer to an-a02n01 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-21 requests from 2 outstanding nodes Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/994, version=0.389.9) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/996, version=0.389.9) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/998, version=0.389.9) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1000, version=0.389.9) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: 78e3bc81916b99f151f320f216b83790 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.389.9 with 0.389.9 from an-a02n01 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/1003, version=0.389.9) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1004) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1005) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-95.raw Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1004, version=0.389.9) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.389.0 of the CIB to disk (digest: 313d66cb7391d8b71c42470181bc65ce) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.djXhhL (digest: /var/lib/pacemaker/cib/cib.OnHTSJ) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1005, version=0.389.9) Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4191314] exited with status 0 Jul 23 16:34:07 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191316] exited with status 0 Jul 23 16:34:08 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191317] exited with status 0 Jul 23 16:34:08 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191318] exited with status 0 Jul 23 16:34:08 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4191319] exited with status 0 Jul 23 16:34:08 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191321] exited with status 0 Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191322] exited with status 0 Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191323] exited with status 0 Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 1006) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 1008) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/1006) Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1007) Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/1008) Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.389.9 2 Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.389.10 (null) Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/1006, version=0.389.10) Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.389.10 2 Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.389.11 (null) Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1007, version=0.389.11) Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.389.11 2 Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.389.12 (null) Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/1008, version=0.389.12) Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4191324] exited with status 0 Jul 23 16:34:09 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191325] exited with status 0 Jul 23 16:34:10 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191326] exited with status 0 Jul 23 16:34:10 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191327] exited with status 0 Jul 23 16:34:10 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4191328] exited with status 0 Jul 23 16:34:10 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191330] exited with status 0 Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191331] exited with status 0 Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191332] exited with status 0 Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1009) Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 154 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.389.12 2 Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.389.13 (null) Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1009, version=0.389.13) Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 154 aborted: History refresh | source=process_resource_updates:268 complete=true Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1012) Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1013) Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1014) Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1012, version=0.389.13) Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.389.13 2 Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.389.14 (null) Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1013, version=0.389.14) Jul 23 16:34:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1014, version=0.389.14) Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Started an-a02n02) Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Started an-a02n02) Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 155, saving inputs in /var/lib/pacemaker/pengine/pe-input-175.bz2 Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 155 (ref=pe_calc-dc-1627083252-566) derived from /var/lib/pacemaker/pengine/pe-input-175.bz2 Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 155 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-175.bz2): Complete Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 16:34:12 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.389.14 2 Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.390.0 (null) Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=390, @num_updates=0 Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03']: Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.390.0) Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 155 aborted by meta_attributes.apc_snmp_node1_an-pdu03-meta_attributes 'create': Configuration change | cib=0.390.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03'] complete=true Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create primitive[@id='apc_snmp_node1_an-pdu03'] Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.390.0 Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu03 has been disabled Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-96.raw Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.390.0 of the CIB to disk (digest: 15f8d550a6c01e9fc4ee2458a27cf318) Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.JNxt6K (digest: /var/lib/pacemaker/cib/cib.GPMCcM) Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu03 ( an-a02n02 ) due to node availability Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Started an-a02n02) Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 156, saving inputs in /var/lib/pacemaker/pengine/pe-input-176.bz2 Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 156 (ref=pe_calc-dc-1627083253-567) derived from /var/lib/pacemaker/pengine/pe-input-176.bz2 Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 on an-a02n02 | action 33 Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.390.0 2 Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.390.1 (null) Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=33:156:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;33:156:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code= Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/273, version=0.390.1) Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.390.1 2 Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.390.2 (null) Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;33:156:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=199, @rc-code=0, @op-status=0 Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/274, version=0.390.2) Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 156 action 33 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=199 Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 156 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-176.bz2): Complete Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 16:34:13 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.390.2 2 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.391.0 (null) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=391, @num_updates=0 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2']: @devices=apc_snmp_node1_an-pdu04 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.390.2 -> 0.391.0 from an-a02n02 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.391.0) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 263 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 156 aborted by fencing-level.fl-an-a02n01-2 'modify': Configuration change | cib=0.391.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] complete=true Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 264 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 265 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 266 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 267 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 268 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 269 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 270 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 2 from fencing topology for an-a02n01 (2 active levels remaining) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1021) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1022) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/263) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/264) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/265) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/266) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/267) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-97.raw Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.391.0 of the CIB to disk (digest: c0457557d28552c3d5ef049c447be1aa) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.dwM4Is (digest: /var/lib/pacemaker/cib/cib.ZFm9iv) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1021, version=0.391.0) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.391.0 2 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.391.1 (null) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1022, version=0.391.1) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.391.1 2 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.391.2 (null) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/263, version=0.391.2) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 263 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.391.2 2 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.391.3 (null) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/264, version=0.391.3) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 264 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.391.3 2 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.391.4 (null) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/265, version=0.391.4) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 265 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.391.4 2 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.391.5 (null) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/266, version=0.391.5) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 266 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.391.5 2 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.391.6 (null) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/267, version=0.391.6) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 267 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/268) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/269) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/270) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.391.6 2 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.391.7 (null) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/268, version=0.391.7) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 268 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.391.7 2 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.391.8 (null) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/269, version=0.391.8) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 269 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.391.8 2 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.391.9 (null) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/270, version=0.391.9) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 270 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/1024, version=0.391.9) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1025) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1025, version=0.391.9) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1027) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1029) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-22 offer to an-a02n02 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-22 offer to an-a02n01 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-22 requests from 2 outstanding nodes Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1027, version=0.391.9) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1029, version=0.391.9) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1031) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1031, version=0.391.9) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: 30f17dec39b640296bea96eeed168290 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.391.9 with 0.391.9 from an-a02n01 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/1034, version=0.391.9) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1035) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1036) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-98.raw Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1035, version=0.391.9) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.391.0 of the CIB to disk (digest: c0457557d28552c3d5ef049c447be1aa) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.ddMtIC (digest: /var/lib/pacemaker/cib/cib.fcKVzD) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1036, version=0.391.9) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4191352] exited with status 0 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.391.9 2 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.392.0 (null) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03'] Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=392, @num_updates=0 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.392.0) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Device 'apc_snmp_node1_an-pdu03' not found (5 active devices) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-99.raw Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.392.0 of the CIB to disk (digest: 41727f940cc8443ad2cee6537bdd209e) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.Ja9jSN (digest: /var/lib/pacemaker/cib/cib.33P0tM) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191354] exited with status 0 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.392.0 2 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.393.0 (null) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=393 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources: Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.393.0) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create resources Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.393.0 Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node1_an-pdu03' to device list (6 active devices) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-0.raw Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.393.0 of the CIB to disk (digest: 9e048ca2840ec729f97e5e09e327f76c) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.INRORZ (digest: /var/lib/pacemaker/cib/cib.HA3Fw0) Jul 23 16:34:15 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191357] exited with status 0 Jul 23 16:34:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.393.0 2 Jul 23 16:34:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.394.0 (null) Jul 23 16:34:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=394 Jul 23 16:34:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu04']: Jul 23 16:34:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.394.0) Jul 23 16:34:16 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create primitive[@id='apc_snmp_node1_an-pdu04'] Jul 23 16:34:16 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.394.0 Jul 23 16:34:16 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 16:34:16 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 16:34:16 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu04 has been disabled Jul 23 16:34:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-1.raw Jul 23 16:34:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.394.0 of the CIB to disk (digest: 07b745dc94200ec52432ade4420a99a4) Jul 23 16:34:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.uV5BW9 (digest: /var/lib/pacemaker/cib/cib.imBst9) Jul 23 16:34:16 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191359] exited with status 0 Jul 23 16:34:16 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4191361] exited with status 0 Jul 23 16:34:16 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191363] exited with status 0 Jul 23 16:34:16 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191364] exited with status 0 Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191365] exited with status 0 Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 1037) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/1037) Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1038) Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 1039) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.394.0 2 Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.394.1 (null) Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/1037, version=0.394.1) Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.394.1 2 Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.394.2 (null) Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1038, version=0.394.2) Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/1039) Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.394.2 2 Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.394.3 (null) Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/1039, version=0.394.3) Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4191366] exited with status 0 Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191367] exited with status 0 Jul 23 16:34:17 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191368] exited with status 0 Jul 23 16:34:18 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191369] exited with status 0 Jul 23 16:34:18 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4191370] exited with status 0 Jul 23 16:34:18 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191372] exited with status 0 Jul 23 16:34:18 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191373] exited with status 0 Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4191374] exited with status 0 Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1040) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 156 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.394.3 2 Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.394.4 (null) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1040, version=0.394.4) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1043) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1044) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1045) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 156 aborted: History refresh | source=process_resource_updates:268 complete=true Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1043, version=0.394.4) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.394.4 2 Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.394.5 (null) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1044, version=0.394.5) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1045, version=0.394.5) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu04 ( an-a02n02 ) due to node availability Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 157, saving inputs in /var/lib/pacemaker/pengine/pe-input-177.bz2 Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 157 (ref=pe_calc-dc-1627083259-576) derived from /var/lib/pacemaker/pengine/pe-input-177.bz2 Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu04_stop_0 on an-a02n02 | action 32 Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 on an-a02n02 | action 33 Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.394.5 2 Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.394.6 (null) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=32:157:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;32:157:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code= Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/277, version=0.394.6) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.394.6 2 Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.394.7 (null) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=33:157:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;33:157:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/278, version=0.394.7) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.394.7 2 Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.394.8 (null) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:0;32:157:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=201, @rc-code=0, @op-status=0 Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/279, version=0.394.8) Jul 23 16:34:19 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 157 action 32 (apc_snmp_node1_an-pdu04_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=201 Jul 23 16:34:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 8b5dc8380fb6f43b301e7cc10b839202 for 0.394.8 (0x561976176560 0) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.394.8 2 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.394.9 (null) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:1;33:157:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=202, @rc-code=1, @op-status=0, @exec-time=13315 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/272, version=0.394.10) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 272 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.394.10 2 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.394.11 (null) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/273, version=0.394.11) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 273 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627083272 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 157 aborted by status-2-fail-count-apc_snmp_node1_an-pdu03.start_0 doing create fail-count-apc_snmp_node1_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.394.10 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 157 aborted by status-2-last-failure-apc_snmp_node1_an-pdu03.start_0 doing create last-failure-apc_snmp_node1_an-pdu03#start_0=1627083272: Transient attribute change | cib=0.394.11 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:34:19 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:34:19 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 158, saving inputs in /var/lib/pacemaker/pengine/pe-input-178.bz2 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627083272-579 is obsolete Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:34:19 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:34:19 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n01 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n02 -> an-a02n01 ) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 159, saving inputs in /var/lib/pacemaker/pengine/pe-input-179.bz2 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 159 (ref=pe_calc-dc-1627083272-580) derived from /var/lib/pacemaker/pengine/pe-input-179.bz2 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 on an-a02n02 | action 10 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.394.11 2 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.394.12 (null) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @transition-key=10:159:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;10:159:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/281, version=0.394.12) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.394.12 2 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.394.13 (null) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;10:159:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=203, @rc-code=0, @op-status=0 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/282, version=0.394.13) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 159 action 10 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=203 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 locally on an-a02n01 | action 32 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=32:159:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_start_0 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1051) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu03 action:start call_id:208 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.394.13 2 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.394.14 (null) Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=32:159:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;32:159:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:34:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1051, version=0.394.14) Jul 23 16:34:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: d0022bc56be580ce062c42d66125522c for 0.394.14 (0x561976176560 0) Jul 23 16:34:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4192328] error output [ 2021-07-23 16:34:38,713 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:34:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4192328] error output [ ] Jul 23 16:34:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4192328] error output [ ] Jul 23 16:34:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4192328] error output [ 2021-07-23 16:34:38,714 ERROR: Please use '-h' for usage ] Jul 23 16:34:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[4192328] error output [ ] Jul 23 16:34:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4192328] stderr: [ 2021-07-23 16:34:38,713 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:34:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4192328] stderr: [ ] Jul 23 16:34:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4192328] stderr: [ ] Jul 23 16:34:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4192328] stderr: [ 2021-07-23 16:34:38,714 ERROR: Please use '-h' for usage ] Jul 23 16:34:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4192328] stderr: [ ] Jul 23 16:34:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (internal_stonith_action_execute) info: Attempt 2 to execute fence_apc_snmp (monitor). remaining timeout is 14 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.394.14 2 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.395.0 a4dc96f2e840cdaa780582c6dc8f372a Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=395, @num_updates=0 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.394.14 -> 0.395.0 from an-a02n01 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.395.0) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 159 aborted by deletion of fencing-level[@id='fl-an-a02n01-2']: Configuration change | cib=0.395.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] complete=false Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 274 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 275 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 276 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 277 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 278 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 279 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 280 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 281 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1054) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1055) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/274) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/275) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/276) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/277) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/278) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.395.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-2.raw Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1054, version=0.395.0) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.395.0 2 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.395.1 (null) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1055, version=0.395.1) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.395.1 2 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.395.2 (null) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/274, version=0.395.2) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 274 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.395.0 of the CIB to disk (digest: 456a7eddb2112387483925c46183a54b) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.395.2 2 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.395.3 (null) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/275, version=0.395.3) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 275 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.GUQkZL (digest: /var/lib/pacemaker/cib/cib.S0bk1I) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.395.3 2 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.395.4 (null) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/276, version=0.395.4) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 276 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.395.4 2 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.395.5 (null) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/277, version=0.395.5) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 277 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627083272 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.395.5 2 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.395.6 (null) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/278, version=0.395.6) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 278 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/279) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/280) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/281) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.395.6 2 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.395.7 (null) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/279, version=0.395.7) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 279 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.395.7 2 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.395.8 (null) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/280, version=0.395.8) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 280 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.395.8 2 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.395.9 (null) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/281, version=0.395.9) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 281 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/1056, version=0.395.9) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1057) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1057, version=0.395.9) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1059) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1059, version=0.395.9) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1061) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-23 offer to an-a02n02 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-23 offer to an-a02n01 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-23 requests from 2 outstanding nodes Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1061, version=0.395.9) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1063) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1063, version=0.395.9) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: e6f57ad4eaf7f87a1728bca828fbd711 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.395.9 with 0.395.9 from an-a02n01 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/1066, version=0.395.9) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1067) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1068) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-3.raw Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1067, version=0.395.9) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.395.0 of the CIB to disk (digest: 456a7eddb2112387483925c46183a54b) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.vSPMi1 (digest: /var/lib/pacemaker/cib/cib.Gcswt1) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1068, version=0.395.9) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4193103] exited with status 0 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu04'] to all (origin=local/cibadmin/2) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.395.9 2 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.396.0 (null) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu04'] Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=396, @num_updates=0 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.396.0) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Device 'apc_snmp_node1_an-pdu04' not found (5 active devices) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-4.raw Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.396.0 of the CIB to disk (digest: 0d2c23d13723f8978cc2f025af829f7e) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.hXb8Al (digest: /var/lib/pacemaker/cib/cib.dvWH6m) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4193105] exited with status 0 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.396.0 2 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.397.0 (null) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=397 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources: Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.397.0) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create resources Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.397.0 Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-5.raw Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node1_an-pdu04' to device list (6 active devices) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.397.0 of the CIB to disk (digest: 5d52fa8ccf890328e24c3153d1d3c63f) Jul 23 16:34:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.PaBaCh (digest: /var/lib/pacemaker/cib/cib.rID9Af) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4193112] exited with status 0 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.397.0 2 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.398.0 (null) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=398 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-2']: @devices=apc_snmp_node2_an-pdu04 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Local-only Replace: 0.398.0 from an-a02n01 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.398.0) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 283 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 284 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 285 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 286 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 287 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 288 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 289 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 2 from fencing topology for an-a02n02 (2 active levels remaining) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 290 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/283) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/284) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/285) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/286) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/287) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.398.0 2 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.398.1 (null) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/283, version=0.398.1) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 283 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.398.1 2 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.398.2 (null) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/284, version=0.398.2) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 284 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.398.2 2 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.398.3 (null) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/285, version=0.398.3) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 285 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.398.3 2 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.398.4 (null) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/286, version=0.398.4) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 286 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627083272 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.398.4 2 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.398.5 (null) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/287, version=0.398.5) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 287 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/288) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/289) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/290) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.398.5 2 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.398.6 (null) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/288, version=0.398.6) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 288 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.398.6 2 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.398.7 (null) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/289, version=0.398.7) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 289 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.398.7 2 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.398.8 (null) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/290, version=0.398.8) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 290 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-6.raw Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.398.0 of the CIB to disk (digest: cab295137764a53b0c7663b43e794dee) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.i6B9Bk (digest: /var/lib/pacemaker/cib/cib.u6DWGl) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //primitive[@id='apc_snmp_node2_an-pdu03'] to all (origin=local/cibadmin/2) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.398.8 2 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.399.0 (null) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu03'] Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=399, @num_updates=0 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //primitive[@id='apc_snmp_node2_an-pdu03']: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.399.0) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Device 'apc_snmp_node2_an-pdu03' not found (6 active devices) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-7.raw Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.399.0 of the CIB to disk (digest: cc256780ae70c328a218029d90233588) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.ZD8SOA (digest: /var/lib/pacemaker/cib/cib.qhHjNE) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4193124] exited with status 0 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4193142] exited with status 0 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4193144] exited with status 0 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.399.0 2 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.400.0 (null) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=400 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources: Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.400.0) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create resources Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.400.0 Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-8.raw Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node2_an-pdu03' to device list (7 active devices) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.400.0 of the CIB to disk (digest: 8c238df30ede790c9f17cb1210a757bd) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.QIciaP (digest: /var/lib/pacemaker/cib/cib.2skUSN) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4192853] error output [ 2021-07-23 16:34:45,874 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4192853] error output [ ] Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4192853] error output [ ] Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4192853] error output [ 2021-07-23 16:34:45,874 ERROR: Please use '-h' for usage ] Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[4192853] error output [ ] Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4192853] stderr: [ 2021-07-23 16:34:45,874 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4192853] stderr: [ ] Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4192853] stderr: [ ] Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4192853] stderr: [ 2021-07-23 16:34:45,874 ERROR: Please use '-h' for usage ] Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[4192853] stderr: [ ] Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_remaining_timeout) info: Attempted to execute agent fence_apc_snmp (monitor) the maximum number of times (2) allowed Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_operation) notice: Operation 'monitor' [4192853] using apc_snmp_node1_an-pdu03 returned -201 (Generic Pacemaker error) Jul 23 16:34:45 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu03 start (call 208) exited with status 1 (execution time 13321ms, queue time 0ms) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4193165] exited with status 0 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.400.0 2 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.401.0 (null) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-2'] Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=401 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Local-only Replace: 0.401.0 from an-a02n01 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.401.0) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.401.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-2'] Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 292 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 293 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 294 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 295 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 296 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 297 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 298 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 299 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/292) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/293) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/294) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/295) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/296) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/297) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/298) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/299) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.401.0 2 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.401.1 (null) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/292, version=0.401.1) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 292 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-9.raw Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.401.1 2 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.401.2 (null) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/293, version=0.401.2) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 293 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.401.2 2 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.401.3 (null) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/294, version=0.401.3) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 294 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.401.3 2 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.401.4 (null) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/295, version=0.401.4) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 295 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627083272 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.401.0 of the CIB to disk (digest: 827dcfce8c10906afe87e383329728b0) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.401.4 2 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.401.5 (null) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/296, version=0.401.5) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.KZfHxz (digest: /var/lib/pacemaker/cib/cib.9AqRzy) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 296 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.401.5 2 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.401.6 (null) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/297, version=0.401.6) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 297 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.401.6 2 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.401.7 (null) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/298, version=0.401.7) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 298 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.401.7 2 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.401.8 (null) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/299, version=0.401.8) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 299 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //primitive[@id='apc_snmp_node2_an-pdu04'] to all (origin=local/cibadmin/2) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.401.8 2 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.402.0 (null) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu04'] Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=402, @num_updates=0 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //primitive[@id='apc_snmp_node2_an-pdu04']: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.402.0) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Device 'apc_snmp_node2_an-pdu04' not found (7 active devices) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-10.raw Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.402.0 of the CIB to disk (digest: aaee519d5efee6870c1ca2703ce96a32) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.JtNntT (digest: /var/lib/pacemaker/cib/cib.sygZ2U) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4193172] exited with status 0 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 159 aborted by deletion of primitive[@id='apc_snmp_node1_an-pdu04']: Configuration change | cib=0.396.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu04'] complete=false Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 159 aborted by primitive.apc_snmp_node1_an-pdu04 'create': Configuration change | cib=0.397.0 source=te_update_diff_v2:474 path=/cib/configuration/resources complete=false Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 159 aborted by fencing-level.fl-an-a02n02-2 'modify': Configuration change | cib=0.398.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-2'] complete=false Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1071) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1072) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for apc_snmp_node1_an-pdu03 on an-a02n01: error | rc=1 call=208 key=apc_snmp_node1_an-pdu03_start_0 confirmed=true cib-update=1073 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 159 aborted by deletion of primitive[@id='apc_snmp_node2_an-pdu03']: Configuration change | cib=0.399.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu03'] complete=false Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 159 aborted by primitive.apc_snmp_node2_an-pdu03 'create': Configuration change | cib=0.400.0 source=te_update_diff_v2:474 path=/cib/configuration/resources complete=false Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 159 aborted by deletion of fencing-level[@id='fl-an-a02n02-2']: Configuration change | cib=0.401.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-2'] complete=false Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 159 aborted by deletion of primitive[@id='apc_snmp_node2_an-pdu04']: Configuration change | cib=0.402.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu04'] complete=false Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_JOIN_RESULT received in state S_ELECTION from route_message Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_JOIN_RESULT received in state S_ELECTION from route_message Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1071, version=0.402.0) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1072, version=0.402.0) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1073) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1076) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1077) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.402.0 2 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.402.1 (null) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:1;32:159:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=208, @rc-code=1, @op-status=0, @exec-time=13321 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/301, version=0.402.3) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 301 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.402.3 2 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.402.4 (null) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/302, version=0.402.4) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 302 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=1627083286 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627083272 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: f01c1bf75afdc5710827537a75141e9f Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.402.4 with 0.402.4 from an-a02n01 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/1088, version=0.402.4) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1089) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1090) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-11.raw Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1089, version=0.402.4) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.402.0 of the CIB to disk (digest: aaee519d5efee6870c1ca2703ce96a32) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.GjEpOh (digest: /var/lib/pacemaker/cib/cib.0NEJ8h) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1090, version=0.402.4) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4193199] exited with status 0 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.402.4 2 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.403.0 (null) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=403, @num_updates=0 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources: Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.403.0) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create resources Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.403.0 Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node2_an-pdu04' to device list (8 active devices) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-12.raw Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.403.0 of the CIB to disk (digest: 0fda71aa7ef4cf9d6ae33f0ec197a9c2) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.4My97Y (digest: /var/lib/pacemaker/cib/cib.7ua1XY) Jul 23 16:34:46 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4193201] exited with status 0 Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4193213] exited with status 0 Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4193228] exited with status 0 Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4193234] exited with status 0 Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.403.0 2 Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.404.0 (null) Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=404 Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.404.0) Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.404.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-13.raw Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.404.0 of the CIB to disk (digest: 34e56e30f13848561297e86f42c769c3) Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.Lq6Vik (digest: /var/lib/pacemaker/cib/cib.xAOj8i) Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4193236] exited with status 0 Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.404.0 2 Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.405.0 (null) Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3'] Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=405 Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.405.0) Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.405.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3'] Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-14.raw Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.405.0 of the CIB to disk (digest: c315785d72f2c8c8533a3c81f188c76e) Jul 23 16:34:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.2GhkgX (digest: /var/lib/pacemaker/cib/cib.rfnMPV) Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4193244] exited with status 0 Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.405.0 2 Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.406.0 (null) Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=406 Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.406.0) Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.406.0 for /cib/configuration/fencing-topology Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-15.raw Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.406.0 of the CIB to disk (digest: da396265269c39c195d64855225107da) Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.6yX4ZP (digest: /var/lib/pacemaker/cib/cib.pH72bO) Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4193253] exited with status 0 Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 1091) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/1091) Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1092) Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 1093) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.406.0 2 Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.406.1 (null) Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/1091, version=0.406.1) Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.406.1 2 Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.406.2 (null) Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1092, version=0.406.2) Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/1093) Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.406.2 2 Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.406.3 (null) Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/1093, version=0.406.3) Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[4193265] exited with status 0 Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.406.3 2 Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.407.0 (null) Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=407, @num_updates=0 Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.407.0) Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.407.0 for /cib/configuration/fencing-topology Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-16.raw Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.407.0 of the CIB to disk (digest: ae84c078a61226f79e7ca3d3aba73339) Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.02juEL (digest: /var/lib/pacemaker/cib/cib.C8oHbK) Jul 23 16:34:48 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4193266] exited with status 0 Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.407.0 2 Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.408.0 (null) Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=408 Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.408.0) Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.408.0 for /cib/configuration/fencing-topology Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-17.raw Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.408.0 of the CIB to disk (digest: f7e002c6ae93042b7af0c3d4997e7691) Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4193276] exited with status 0 Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.axRont (digest: /var/lib/pacemaker/cib/cib.hXemTw) Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4193283] exited with status 0 Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[4193297] exited with status 0 Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4193306] exited with status 0 Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.408.0 2 Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.409.0 (null) Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-1'] Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=409 Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.409.0) Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.409.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-1'] Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-18.raw Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.409.0 of the CIB to disk (digest: 9d566fbaf9025a00aa26357793d8c3a1) Jul 23 16:34:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.zjLIup (digest: /var/lib/pacemaker/cib/cib.YYl7Wm) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4193316] exited with status 0 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.409.0 2 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.410.0 (null) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-3'] Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=410 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.410.0) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.410.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-3'] Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-19.raw Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.410.0 of the CIB to disk (digest: 2089d4006cac94d6a4bcdbb7f361ff2b) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.QDC84Z (digest: /var/lib/pacemaker/cib/cib.xWSaa1) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[4193324] exited with status 0 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1094) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 159 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.410.0 2 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.410.1 (null) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1094, version=0.410.1) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 159 aborted: History refresh | source=process_resource_updates:268 complete=true Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1097) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1098) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1099) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1097, version=0.410.1) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.410.1 2 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.410.2 (null) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1098, version=0.410.2) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1099, version=0.410.2) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:34:32 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:34:19 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n02 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu03 on an-a02n02 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu04 on an-a02n02 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu03 ( an-a02n01 ) due to node availability Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu04 ( an-a02n02 ) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node2_an-pdu03 ( an-a02n02 ) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node2_an-pdu04 ( an-a02n02 ) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 160, saving inputs in /var/lib/pacemaker/pengine/pe-input-180.bz2 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 160 (ref=pe_calc-dc-1627083290-598) derived from /var/lib/pacemaker/pengine/pe-input-180.bz2 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 locally on an-a02n01 | action 2 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=2:160:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_stop_0 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu03 action:stop call_id:209 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu04_start_0 on an-a02n02 | action 32 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node2_an-pdu03_start_0 on an-a02n02 | action 34 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node2_an-pdu04_start_0 on an-a02n02 | action 36 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu03 stop (call 209) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1102) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01: ok | rc=0 call=209 key=apc_snmp_node1_an-pdu03_stop_0 confirmed=true cib-update=1103 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1103) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.410.2 2 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.410.3 (null) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @crm-debug-origin=do_update_resource, @transition-key=2:160:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;2:160:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1627083290 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1102, version=0.410.3) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.410.3 2 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.410.4 (null) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;2:160:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=209, @rc-code=0, @op-status=0 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1103, version=0.410.4) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 160 action 2 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=209 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.410.4 2 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.410.5 (null) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=32:160:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;32:160:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/287, version=0.410.5) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.410.5 2 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.410.6 (null) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @operation_key=apc_snmp_node2_an-pdu03_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=34:160:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;34:160:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/288, version=0.410.6) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.410.6 2 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.410.7 (null) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @operation_key=apc_snmp_node2_an-pdu04_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=36:160:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;36:160:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/289, version=0.410.7) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.410.7 2 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.411.0 (null) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=411, @num_updates=0 Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.411.0) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 160 aborted by fencing-level.fl-an-a02n02-1 'create': Configuration change | cib=0.411.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology complete=false Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.411.0 for /cib/configuration/fencing-topology Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (1 active entry) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-20.raw Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.411.0 of the CIB to disk (digest: 937fc9930cbbab63366fd08847d3f9b3) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.5yKC7F (digest: /var/lib/pacemaker/cib/cib.Jv008D) Jul 23 16:34:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.411.0 2 Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.412.0 (null) Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=412 Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.412.0) Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 160 aborted by fencing-level.fl-an-a02n02-2 'create': Configuration change | cib=0.412.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology complete=false Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.412.0 for /cib/configuration/fencing-topology Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (1 active entry) Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-21.raw Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.412.0 of the CIB to disk (digest: b0b64808a340c9f3e7542664ebbcd52e) Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.KD54ze (digest: /var/lib/pacemaker/cib/cib.P9xfgi) Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.412.0 2 Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.413.0 (null) Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=413 Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.413.0) Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 160 aborted by fencing-level.fl-an-a02n02-3 'create': Configuration change | cib=0.413.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology complete=false Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.413.0 for /cib/configuration/fencing-topology Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (1 active entry) Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-22.raw Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.413.0 of the CIB to disk (digest: cd54e511582baaa179b0be820602012c) Jul 23 16:34:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.cMBFGf (digest: /var/lib/pacemaker/cib/cib.m4sKeg) Jul 23 16:34:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: da4e63de5fa4b0a2a73206808e1fd8b2 for 0.413.0 (0x561976313c60 0) Jul 23 16:34:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:34:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.413.0 2 Jul 23 16:34:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.414.0 (null) Jul 23 16:34:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=414 Jul 23 16:34:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03']: Jul 23 16:34:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:34:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.414.0) Jul 23 16:34:59 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 160 aborted by meta_attributes.apc_snmp_node1_an-pdu03-meta_attributes 'create': Configuration change | cib=0.414.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03'] complete=false Jul 23 16:34:59 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create primitive[@id='apc_snmp_node1_an-pdu03'] Jul 23 16:34:59 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.414.0 Jul 23 16:34:59 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu03 has been disabled Jul 23 16:34:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-23.raw Jul 23 16:34:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.414.0 of the CIB to disk (digest: 66e241a508863624a9a3f98c5db76371) Jul 23 16:34:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.weTQq6 (digest: /var/lib/pacemaker/cib/cib.q87cs9) Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.414.0 2 Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.415.0 (null) Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=415 Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu04']: Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.415.0) Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 160 aborted by meta_attributes.apc_snmp_node1_an-pdu04-meta_attributes 'create': Configuration change | cib=0.415.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu04'] complete=false Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create primitive[@id='apc_snmp_node1_an-pdu04'] Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.415.0 Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu03 has been disabled Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu04 has been disabled Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-24.raw Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.415.0 of the CIB to disk (digest: 7526b3cd078c8525dbd99e9ad6783a7f) Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.moHwJF (digest: /var/lib/pacemaker/cib/cib.gyBNDF) Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.415.0 2 Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.415.1 (null) Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:1;34:160:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=205, @rc-code=1, @op-status=0, @exec-time=12317 Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/303, version=0.415.2) Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 303 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 160 aborted by status-2-fail-count-apc_snmp_node2_an-pdu03.start_0 doing create fail-count-apc_snmp_node2_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.415.2 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=false Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.415.2 2 Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.415.3 (null) Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/304, version=0.415.3) Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 160 aborted by status-2-last-failure-apc_snmp_node2_an-pdu03.start_0 doing create last-failure-apc_snmp_node2_an-pdu03#start_0=1627083302: Transient attribute change | cib=0.415.3 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=false Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 304 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:35:02 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=1627083302 Jul 23 16:35:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.415.3 2 Jul 23 16:35:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.415.4 (null) Jul 23 16:35:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:35:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:1;36:160:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=206, @rc-code=1, @op-status=0, @exec-time=13315 Jul 23 16:35:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: Jul 23 16:35:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/305, version=0.415.5) Jul 23 16:35:03 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 305 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:35:03 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:35:03 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:35:03 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 160 aborted by status-2-fail-count-apc_snmp_node2_an-pdu04.start_0 doing create fail-count-apc_snmp_node2_an-pdu04#start_0=INFINITY: Transient attribute change | cib=0.415.5 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=false Jul 23 16:35:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.415.5 2 Jul 23 16:35:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.415.6 (null) Jul 23 16:35:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:35:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 16:35:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/306, version=0.415.6) Jul 23 16:35:03 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 160 aborted by status-2-last-failure-apc_snmp_node2_an-pdu04.start_0 doing create last-failure-apc_snmp_node2_an-pdu04#start_0=1627083303: Transient attribute change | cib=0.415.6 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=false Jul 23 16:35:03 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 306 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:35:03 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:35:03 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=1627083303 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.415.6 2 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.415.7 (null) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:1;32:160:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=204, @rc-code=1, @op-status=0, @exec-time=14321 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/307, version=0.415.8) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/308) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 307 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.415.8 2 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.415.9 (null) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/308, version=0.415.9) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 308 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627083304 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 160 aborted by status-2-fail-count-apc_snmp_node1_an-pdu04.start_0 doing create fail-count-apc_snmp_node1_an-pdu04#start_0=INFINITY: Transient attribute change | cib=0.415.8 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 160 aborted by status-2-last-failure-apc_snmp_node1_an-pdu04.start_0 doing create last-failure-apc_snmp_node1_an-pdu04#start_0=1627083304: Transient attribute change | cib=0.415.9 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:34:32 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:34:19 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_0 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_0 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu03 on an-a02n01 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu04 on an-a02n01 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu04 ( an-a02n02 ) due to node availability Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node2_an-pdu03 ( an-a02n02 -> an-a02n01 ) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node2_an-pdu04 ( an-a02n02 -> an-a02n01 ) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 161, saving inputs in /var/lib/pacemaker/pengine/pe-input-181.bz2 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627083304-603 is obsolete Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:34:32 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:34:19 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_0 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_0 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu03 on an-a02n01 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu04 on an-a02n01 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu04 ( an-a02n02 ) due to node availability Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node2_an-pdu03 ( an-a02n02 -> an-a02n01 ) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node2_an-pdu04 ( an-a02n02 -> an-a02n01 ) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 162, saving inputs in /var/lib/pacemaker/pengine/pe-input-182.bz2 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 162 (ref=pe_calc-dc-1627083304-604) derived from /var/lib/pacemaker/pengine/pe-input-182.bz2 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu04_stop_0 on an-a02n02 | action 10 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node2_an-pdu03_stop_0 on an-a02n02 | action 12 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node2_an-pdu04_stop_0 on an-a02n02 | action 13 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.415.9 2 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.415.10 (null) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_stop_0, @operation=stop, @transition-key=10:162:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;10:162:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/293, version=0.415.10) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.415.10 2 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.415.11 (null) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @operation_key=apc_snmp_node2_an-pdu03_stop_0, @operation=stop, @transition-key=12:162:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;12:162:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/294, version=0.415.11) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.415.11 2 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.415.12 (null) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:0;10:162:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=207, @rc-code=0, @op-status=0 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/295, version=0.415.12) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 162 action 10 (apc_snmp_node1_an-pdu04_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=207 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.415.12 2 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.415.13 (null) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:0;12:162:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=208, @rc-code=0, @op-status=0 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/296, version=0.415.13) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 162 action 12 (apc_snmp_node2_an-pdu03_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=208 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node2_an-pdu03_start_0 locally on an-a02n01 | action 34 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node2_an-pdu03 on an-a02n01 | transition_key=34:162:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu03_start_0 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node2_an-pdu03 action:start call_id:210 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.415.13 2 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.415.14 (null) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @operation_key=apc_snmp_node2_an-pdu04_stop_0, @operation=stop, @transition-key=13:162:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;13:162:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/297, version=0.415.14) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1107) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.415.14 2 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.415.15 (null) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=15 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:0;13:162:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=209, @rc-code=0, @op-status=0, @exec-time=1 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/298, version=0.415.15) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 162 action 13 (apc_snmp_node2_an-pdu04_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=209 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node2_an-pdu04_start_0 locally on an-a02n01 | action 36 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node2_an-pdu04 on an-a02n01 | transition_key=36:162:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu04_start_0 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.415.15 2 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.415.16 (null) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=16 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node2_an-pdu04 action:start call_id:211 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @operation_key=apc_snmp_node2_an-pdu03_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=34:162:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;34:162:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1107, version=0.415.16) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1108) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.415.16 2 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.415.17 (null) Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=17 Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @operation_key=apc_snmp_node2_an-pdu04_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=36:162:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;36:162:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:35:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1108, version=0.415.17) Jul 23 16:35:06 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:35:06 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.415.17 2 Jul 23 16:35:06 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.416.0 (null) Jul 23 16:35:06 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=416, @num_updates=0 Jul 23 16:35:06 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu03']: Jul 23 16:35:06 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:35:06 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:35:06 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.416.0) Jul 23 16:35:06 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 162 aborted by meta_attributes.apc_snmp_node2_an-pdu03-meta_attributes 'create': Configuration change | cib=0.416.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu03'] complete=false Jul 23 16:35:06 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create primitive[@id='apc_snmp_node2_an-pdu03'] Jul 23 16:35:06 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.416.0 Jul 23 16:35:06 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu03 has been disabled Jul 23 16:35:06 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu04 has been disabled Jul 23 16:35:06 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 16:35:06 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-25.raw Jul 23 16:35:06 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.416.0 of the CIB to disk (digest: 01632c6782eae5548d06d62e76ef2450) Jul 23 16:35:06 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.kn6LNy (digest: /var/lib/pacemaker/cib/cib.Of4zNA) Jul 23 16:35:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 16:35:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.416.0 2 Jul 23 16:35:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.417.0 (null) Jul 23 16:35:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=417 Jul 23 16:35:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu04']: Jul 23 16:35:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:35:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:35:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.417.0) Jul 23 16:35:07 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 162 aborted by meta_attributes.apc_snmp_node2_an-pdu04-meta_attributes 'create': Configuration change | cib=0.417.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu04'] complete=false Jul 23 16:35:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create primitive[@id='apc_snmp_node2_an-pdu04'] Jul 23 16:35:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.417.0 Jul 23 16:35:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu03 has been disabled Jul 23 16:35:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu04 has been disabled Jul 23 16:35:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 16:35:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 16:35:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-26.raw Jul 23 16:35:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.417.0 of the CIB to disk (digest: 5c9bd32fbbb0eff7f6ac621c6a513d38) Jul 23 16:35:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.xGaMFv (digest: /var/lib/pacemaker/cib/cib.P2MnVx) Jul 23 16:35:10 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[364] error output [ 2021-07-23 16:35:10,895 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 16:35:10 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[364] error output [ ] Jul 23 16:35:10 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[364] error output [ ] Jul 23 16:35:10 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[364] error output [ 2021-07-23 16:35:10,895 ERROR: Please use '-h' for usage ] Jul 23 16:35:10 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[364] error output [ ] Jul 23 16:35:10 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[364] stderr: [ 2021-07-23 16:35:10,895 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 16:35:10 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[364] stderr: [ ] Jul 23 16:35:10 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[364] stderr: [ ] Jul 23 16:35:10 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[364] stderr: [ 2021-07-23 16:35:10,895 ERROR: Please use '-h' for usage ] Jul 23 16:35:10 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[364] stderr: [ ] Jul 23 16:35:10 an-a02n01.alteeve.com pacemaker-fenced [121629] (internal_stonith_action_execute) info: Attempt 2 to execute fence_apc_snmp (monitor). remaining timeout is 14 Jul 23 16:35:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[363] error output [ 2021-07-23 16:35:10,895 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:35:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[363] error output [ ] Jul 23 16:35:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[363] error output [ ] Jul 23 16:35:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[363] error output [ 2021-07-23 16:35:10,895 ERROR: Please use '-h' for usage ] Jul 23 16:35:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[363] error output [ ] Jul 23 16:35:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[363] stderr: [ 2021-07-23 16:35:10,895 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:35:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[363] stderr: [ ] Jul 23 16:35:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[363] stderr: [ ] Jul 23 16:35:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[363] stderr: [ 2021-07-23 16:35:10,895 ERROR: Please use '-h' for usage ] Jul 23 16:35:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[363] stderr: [ ] Jul 23 16:35:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (internal_stonith_action_execute) info: Attempt 2 to execute fence_apc_snmp (monitor). remaining timeout is 13 Jul 23 16:35:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 8041a68bbba6f0b981a80f39261a2158 for 0.417.0 (0x561976176210 0) Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[1212] error output [ 2021-07-23 16:35:18,053 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[1212] error output [ ] Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[1212] error output [ ] Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[1212] error output [ 2021-07-23 16:35:18,053 ERROR: Please use '-h' for usage ] Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[1212] error output [ ] Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[1212] stderr: [ 2021-07-23 16:35:18,053 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[1212] stderr: [ ] Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[1212] stderr: [ ] Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[1212] stderr: [ 2021-07-23 16:35:18,053 ERROR: Please use '-h' for usage ] Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[1212] stderr: [ ] Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_remaining_timeout) info: Attempted to execute agent fence_apc_snmp (monitor) the maximum number of times (2) allowed Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_operation) notice: Operation 'monitor' [1212] using apc_snmp_node2_an-pdu04 returned -201 (Generic Pacemaker error) Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node2_an-pdu04 start (call 211) exited with status 1 (execution time 13318ms, queue time 0ms) Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for apc_snmp_node2_an-pdu04 on an-a02n01: error | rc=1 call=211 key=apc_snmp_node2_an-pdu04_start_0 confirmed=true cib-update=1109 Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1109) Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.417.0 2 Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.417.1 (null) Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:1;36:162:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=211, @rc-code=1, @op-status=0, @exec-time=13318 Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/309, version=0.417.2) Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 162 aborted by status-1-fail-count-apc_snmp_node2_an-pdu04.start_0 doing create fail-count-apc_snmp_node2_an-pdu04#start_0=INFINITY: Transient attribute change | cib=0.417.2 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=false Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 309 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/310) Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.417.2 2 Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.417.3 (null) Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/310, version=0.417.3) Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 162 aborted by status-1-last-failure-apc_snmp_node2_an-pdu04.start_0 doing create last-failure-apc_snmp_node2_an-pdu04#start_0=1627083318: Transient attribute change | cib=0.417.3 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=false Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 310 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=1627083318 Jul 23 16:35:18 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=1627083303 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[1374] error output [ 2021-07-23 16:35:19,054 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[1374] error output [ ] Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[1374] error output [ ] Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[1374] error output [ 2021-07-23 16:35:19,054 ERROR: Please use '-h' for usage ] Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[1374] error output [ ] Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[1374] stderr: [ 2021-07-23 16:35:19,054 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[1374] stderr: [ ] Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[1374] stderr: [ ] Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[1374] stderr: [ 2021-07-23 16:35:19,054 ERROR: Please use '-h' for usage ] Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[1374] stderr: [ ] Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_remaining_timeout) info: Attempted to execute agent fence_apc_snmp (monitor) the maximum number of times (2) allowed Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_operation) notice: Operation 'monitor' [1374] using apc_snmp_node2_an-pdu03 returned -201 (Generic Pacemaker error) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node2_an-pdu03 start (call 210) exited with status 1 (execution time 14320ms, queue time 0ms) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for apc_snmp_node2_an-pdu03 on an-a02n01: error | rc=1 call=210 key=apc_snmp_node2_an-pdu03_start_0 confirmed=true cib-update=1110 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1110) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.417.3 2 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.417.4 (null) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:1;34:162:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=210, @rc-code=1, @op-status=0, @exec-time=14320 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/311, version=0.417.5) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 311 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.417.5 2 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.417.6 (null) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/312, version=0.417.6) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 312 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=1627083319 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=1627083302 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 162 aborted by status-1-fail-count-apc_snmp_node2_an-pdu03.start_0 doing create fail-count-apc_snmp_node2_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.417.5 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 162 aborted by status-1-last-failure-apc_snmp_node2_an-pdu03.start_0 doing create last-failure-apc_snmp_node2_an-pdu03#start_0=1627083319: Transient attribute change | cib=0.417.6 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:34:32 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_0 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_0 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:34:19 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node2_an-pdu03 ( an-a02n01 ) due to node availability Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node2_an-pdu04 ( an-a02n01 ) due to node availability Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 163, saving inputs in /var/lib/pacemaker/pengine/pe-input-183.bz2 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627083319-610 is obsolete Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:34:32 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_0 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_0 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:34:19 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node2_an-pdu03 ( an-a02n01 ) due to node availability Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node2_an-pdu04 ( an-a02n01 ) due to node availability Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 164, saving inputs in /var/lib/pacemaker/pengine/pe-input-184.bz2 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 164 (ref=pe_calc-dc-1627083319-611) derived from /var/lib/pacemaker/pengine/pe-input-184.bz2 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node2_an-pdu03_stop_0 locally on an-a02n01 | action 7 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node2_an-pdu03 on an-a02n01 | transition_key=7:164:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu03_stop_0 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node2_an-pdu03 action:stop call_id:212 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node2_an-pdu04_stop_0 locally on an-a02n01 | action 8 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1114) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node2_an-pdu04 on an-a02n01 | transition_key=8:164:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu04_stop_0 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Device 'apc_snmp_node2_an-pdu03' not found (4 active devices) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node2_an-pdu03 stop (call 212) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1115) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node2_an-pdu04 action:stop call_id:213 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Device 'apc_snmp_node2_an-pdu04' not found (4 active devices) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node2_an-pdu04 stop (call 213) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1116) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node2_an-pdu03 on an-a02n01: ok | rc=0 call=212 key=apc_snmp_node2_an-pdu03_stop_0 confirmed=true cib-update=1116 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1117) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node2_an-pdu04 on an-a02n01: ok | rc=0 call=213 key=apc_snmp_node2_an-pdu04_stop_0 confirmed=true cib-update=1117 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.417.6 2 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.417.7 (null) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @operation_key=apc_snmp_node2_an-pdu03_stop_0, @operation=stop, @transition-key=7:164:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;7:164:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1627 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1114, version=0.417.7) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.417.7 2 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.417.8 (null) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @operation_key=apc_snmp_node2_an-pdu04_stop_0, @operation=stop, @transition-key=8:164:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;8:164:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1627 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1115, version=0.417.8) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.417.8 2 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.417.9 (null) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:0;7:164:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=212, @rc-code=0, @op-status=0 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1116, version=0.417.9) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 164 action 7 (apc_snmp_node2_an-pdu03_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=212 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.417.9 2 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.417.10 (null) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:0;8:164:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=213, @rc-code=0, @op-status=0 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1117, version=0.417.10) Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 164 action 8 (apc_snmp_node2_an-pdu04_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=213 Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 164 (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-184.bz2): Complete Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 16:35:19 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:35:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: aa772503b3d74735ff0fbb8dcc66f58f for 0.417.10 (0x561976176210 0) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.417.10 2 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.418.0 733e429b368a0bfefd0406843adafa56 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=418, @num_updates=0 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03']/instance_attributes[@id='apc_snmp_node1_an-pdu03-instance_attributes']/nvpair[@id='apc_snmp_node1_an-pdu03-instance_attributes-ip']: @value=10.201.2.3 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03']/instance_attributes[@id='apc_snmp_node1_an-pdu03-instance_attributes']/nvpair[@id='apc_snmp_node1_an-pdu03-instance_attributes-port']: @value=3 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1']: @id=fl-an-a02n01-1, @index=1 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2']: @id=fl-an-a02n01-2, @index=2 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3']: @id=fl-an-a02n01-3, @index=3 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-1']: @id=fl-an-a02n02-1, @index=1 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-2']: @id=fl-an-a02n02-2, @index=2 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-3']: @id=fl-an-a02n02-3, @index=3 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.417.10 -> 0.418.0 from an-a02n01 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.418.0) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 164 aborted by apc_snmp_node1_an-pdu03-instance_attributes-ip doing modify ip=10.201.2.3: Configuration change | cib=0.418.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03']/instance_attributes[@id='apc_snmp_node1_an-pdu03-instance_attributes']/nvpair[@id='apc_snmp_node1_an-pdu03-instance_attributes-ip'] complete=true Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 313 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 314 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 315 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 316 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 317 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 318 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 319 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 320 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1120) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1121) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 1 from fencing topology for an-a02n01 (2 active levels remaining) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 2 from fencing topology for an-a02n01 (2 active levels remaining) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 3 from fencing topology for an-a02n01 (2 active levels remaining) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 1 from fencing topology for an-a02n02 (2 active levels remaining) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 2 from fencing topology for an-a02n02 (2 active levels remaining) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 3 from fencing topology for an-a02n02 (2 active levels remaining) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: modify nvpair[@id='apc_snmp_node1_an-pdu03-instance_attributes-ip'] Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.418.0 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/313) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/314) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/315) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/316) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/317) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-27.raw Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu03 has been disabled Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu04 has been disabled Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.418.0 of the CIB to disk (digest: 52ce41b40ade94fd4ff9d89be86e0e60) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.3HZMIF (digest: /var/lib/pacemaker/cib/cib.IocH6F) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1120, version=0.418.0) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.418.0 2 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.418.1 (null) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1121, version=0.418.1) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/313, version=0.418.1) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 313 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=1627083319 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=1627083302 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/314, version=0.418.1) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 314 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/315, version=0.418.1) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 315 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/316, version=0.418.1) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 316 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=1627083286 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627083272 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/317, version=0.418.1) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 317 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/318) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/319) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/320) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/318, version=0.418.1) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 318 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=1627083318 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=1627083303 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.418.1 2 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.418.2 (null) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/319, version=0.418.2) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 319 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.418.2 2 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.418.3 (null) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/320, version=0.418.3) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 320 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627083304 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/1123, version=0.418.3) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1124) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1126) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1124, version=0.418.3) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1126, version=0.418.3) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1128) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1130) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-25 offer to an-a02n02 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-25 offer to an-a02n01 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-25 requests from 2 outstanding nodes Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1128, version=0.418.3) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1130, version=0.418.3) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: d7e6a2a969785486d521a2271d560079 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.418.3 with 0.418.3 from an-a02n01 Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/1133, version=0.418.3) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1134) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1135) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-28.raw Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1134, version=0.418.3) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.418.0 of the CIB to disk (digest: 52ce41b40ade94fd4ff9d89be86e0e60) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.nKVJHQ (digest: /var/lib/pacemaker/cib/cib.1UfSzQ) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1135, version=0.418.3) Jul 23 16:37:26 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[7929] exited with status 0 Jul 23 16:37:27 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[7931] exited with status 0 Jul 23 16:37:27 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[7933] exited with status 0 Jul 23 16:37:27 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[7945] exited with status 0 Jul 23 16:37:27 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[7946] exited with status 0 Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[7948] exited with status 0 Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[7949] exited with status 0 Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[7950] exited with status 0 Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 1136) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/1136) Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 1138) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1137) Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.418.3 2 Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.418.4 (null) Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/1136, version=0.418.4) Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.418.4 2 Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.418.5 (null) Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1137, version=0.418.5) Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/1138) Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.418.5 2 Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.418.6 (null) Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/1138, version=0.418.6) Jul 23 16:37:28 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[7951] exited with status 0 Jul 23 16:37:29 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[7952] exited with status 0 Jul 23 16:37:29 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[7953] exited with status 0 Jul 23 16:37:29 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[7962] exited with status 0 Jul 23 16:37:29 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[7970] exited with status 0 Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[7972] exited with status 0 Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[7973] exited with status 0 Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[7975] exited with status 0 Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1139) Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 164 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.418.6 2 Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.418.7 (null) Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1139, version=0.418.7) Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 164 aborted: History refresh | source=process_resource_updates:268 complete=true Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1142) Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1143) Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1144) Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1142, version=0.418.7) Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.418.7 2 Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.418.8 (null) Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1143, version=0.418.8) Jul 23 16:37:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1144, version=0.418.8) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n01 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now 5e7d5a6e316938194ebb385cfee4a6f2 (reload:3.10.2) 0:1;32:159:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n01 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:34:32 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n02 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now 5e7d5a6e316938194ebb385cfee4a6f2 (reload:3.10.2) 0:1;33:157:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:34:19 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 165, saving inputs in /var/lib/pacemaker/pengine/pe-input-185.bz2 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 165 (ref=pe_calc-dc-1627083451-621) derived from /var/lib/pacemaker/pengine/pe-input-185.bz2 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-controld [121633] (te_crm_command) info: Executing crm-event (11) without waiting: clear_failcount on an-a02n02 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-controld [121633] (te_crm_command) info: Executing crm-event (2) locally without waiting: clear_failcount on an-a02n01 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 165 (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-185.bz2): Complete Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-controld [121633] (update_attrd_clear_failures) info: Asking pacemaker-attrd to clear failure of all operations for apc_snmp_node1_an-pdu03 on cluster node an-a02n01 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section /cib/status/node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0'] to all (origin=local/crmd/1148) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 322 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: 1627083286 -> (unset) | from an-a02n01 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/322) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 323 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/323) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n02 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node1_an-pdu03#start_0' delayed: update 322 in progress Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.418.8 2 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.418.9 (null) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0'] Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: 1627083272 -> (unset) | from an-a02n02 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node1_an-pdu03#start_0' delayed: update 323 in progress Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section /cib/status/node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0']: OK (rc=0, origin=an-a02n01/crmd/1148, version=0.418.9) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 165 aborted by deletion of lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0']: Resource operation removal | cib=0.418.9 source=process_op_deletion:358 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0'] complete=true Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.418.9 2 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.418.10 (null) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0'] Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/322, version=0.418.10) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 322 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 165 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.418.10 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 324 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.418.10 2 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.418.11 (null) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0'] Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/323, version=0.418.11) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 323 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627083272 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 165 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.418.11 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 325 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.418.11 2 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.418.12 (null) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0'] Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section /cib/status/node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0']: OK (rc=0, origin=an-a02n02/crmd/301, version=0.418.12) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 165 aborted by deletion of lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0']: Resource operation removal | cib=0.418.12 source=process_op_deletion:358 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0'] complete=true Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/324) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/325) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.418.12 2 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.418.13 (null) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/324, version=0.418.13) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 324 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.418.13 2 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.418.14 (null) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/325, version=0.418.14) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 325 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 165 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.418.13 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 165 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.418.14 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 166, saving inputs in /var/lib/pacemaker/pengine/pe-input-186.bz2 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627083451-624 is obsolete Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 167, saving inputs in /var/lib/pacemaker/pengine/pe-input-187.bz2 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 167 (ref=pe_calc-dc-1627083451-625) derived from /var/lib/pacemaker/pengine/pe-input-187.bz2 Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 167 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-187.bz2): Complete Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 16:37:31 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:37:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 27de344c6bf1c487a04a5777f6fa80e5 for 0.418.14 (0x5619760ad2f0 0) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.418.14 2 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.419.0 (null) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=419, @num_updates=0 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2']: @devices=apc_snmp_node1_an-pdu04 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.418.14 -> 0.419.0 from an-a02n02 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.419.0) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 167 aborted by fencing-level.fl-an-a02n01-2 'modify': Configuration change | cib=0.419.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] complete=true Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 326 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 327 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 328 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 329 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 330 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 331 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 332 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 333 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 2 from fencing topology for an-a02n01 (2 active levels remaining) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1157) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1158) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/326) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/327) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/328) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/329) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/330) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-29.raw Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1157, version=0.419.0) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.419.0 2 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.419.1 (null) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.419.0 of the CIB to disk (digest: f52d24e2f6bd8d871fd257f691ad530f) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1158, version=0.419.1) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/326, version=0.419.1) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.ui4oGD (digest: /var/lib/pacemaker/cib/cib.u1rpCz) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 326 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=1627083319 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=1627083302 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/327, version=0.419.1) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 327 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.419.1 2 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.419.2 (null) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/328, version=0.419.2) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 328 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.419.2 2 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.419.3 (null) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/329, version=0.419.3) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 329 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/330, version=0.419.3) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 330 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/331) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/332) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/333) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/1160, version=0.419.3) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1161) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/331, version=0.419.3) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 331 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=1627083318 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=1627083303 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.419.3 2 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.419.4 (null) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/332, version=0.419.4) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 332 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.419.4 2 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.419.5 (null) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/333, version=0.419.5) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 333 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627083304 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1161, version=0.419.5) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1163) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1165) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-26 offer to an-a02n02 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-26 offer to an-a02n01 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-26 requests from 2 outstanding nodes Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1163, version=0.419.5) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1165, version=0.419.5) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1167) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1167, version=0.419.5) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: 45969eeff4090bba4e968732bc5c5af6 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.419.5 with 0.419.5 from an-a02n01 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/1170, version=0.419.5) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1171) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1172) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1171, version=0.419.5) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1172, version=0.419.5) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[8051] exited with status 0 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-30.raw Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.419.0 of the CIB to disk (digest: f52d24e2f6bd8d871fd257f691ad530f) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.vkSsva (digest: /var/lib/pacemaker/cib/cib.41BUpa) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.419.5 2 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.420.0 (null) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03'] Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=420, @num_updates=0 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.420.0) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Device 'apc_snmp_node1_an-pdu03' not found (4 active devices) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-31.raw Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.420.0 of the CIB to disk (digest: bb801a27abbe2e97edfd495b901c9521) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.3wM5Ps (digest: /var/lib/pacemaker/cib/cib.N9qI5o) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[8054] exited with status 0 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.420.0 2 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.421.0 (null) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=421 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources: Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.421.0) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create resources Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.421.0 Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu04 has been disabled Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node1_an-pdu03' to device list (5 active devices) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-32.raw Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.421.0 of the CIB to disk (digest: c4dfbd72ba6c12deeb43d31c4652dc9c) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.ez5sey (digest: /var/lib/pacemaker/cib/cib.SOIuVA) Jul 23 16:37:38 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[8058] exited with status 0 Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[8061] exited with status 0 Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[8062] exited with status 0 Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.421.0 2 Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.422.0 (null) Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=422 Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.422.0) Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.422.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (1 active entry) Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-33.raw Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.422.0 of the CIB to disk (digest: bb5cd00714e97958f8d47642672c7b37) Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.En6cub (digest: /var/lib/pacemaker/cib/cib.pvfGId) Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[8064] exited with status 0 Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.422.0 2 Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.423.0 (null) Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=423 Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.423.0) Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.423.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (1 active entry) Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-34.raw Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.423.0 of the CIB to disk (digest: fe2cecd4006ade7283a98a4d11c3f813) Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.EMjhHj (digest: /var/lib/pacemaker/cib/cib.CfDQAm) Jul 23 16:37:39 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[8068] exited with status 0 Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[8070] exited with status 0 Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 1173) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/1173) Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1174) Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 1175) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.423.0 2 Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.423.1 (null) Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/1173, version=0.423.1) Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.423.1 2 Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.423.2 (null) Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1174, version=0.423.2) Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/1175) Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.423.2 2 Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.424.0 (null) Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3'] Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=424, @num_updates=0 Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.424.0) Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.424.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3'] Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.424.0 2 Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.424.1 (null) Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[8072] exited with status 0 Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/1175, version=0.424.1) Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-35.raw Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.424.0 of the CIB to disk (digest: 44a12e2151b1cbe08ea09aa9bf7ca05c) Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.qkVY5I (digest: /var/lib/pacemaker/cib/cib.F353RJ) Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[8073] exited with status 0 Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.424.1 2 Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.425.0 (null) Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=425, @num_updates=0 Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.425.0) Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.425.0 for /cib/configuration/fencing-topology Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-36.raw Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.425.0 of the CIB to disk (digest: 9a00c19ab8cde4b792ab6e969cc29df9) Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.ojzBeJ (digest: /var/lib/pacemaker/cib/cib.kqqz0H) Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[8086] exited with status 0 Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.425.0 2 Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.426.0 (null) Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=426 Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.426.0) Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.426.0 for /cib/configuration/fencing-topology Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-37.raw Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.426.0 of the CIB to disk (digest: a89c4a8184f692d6034285250a4f7291) Jul 23 16:37:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.EC97ej (digest: /var/lib/pacemaker/cib/cib.dLUVOi) Jul 23 16:37:41 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[8094] exited with status 0 Jul 23 16:37:41 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[8096] exited with status 0 Jul 23 16:37:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.426.0 2 Jul 23 16:37:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.427.0 (null) Jul 23 16:37:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=427 Jul 23 16:37:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 16:37:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.427.0) Jul 23 16:37:41 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.427.0 for /cib/configuration/fencing-topology Jul 23 16:37:41 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 16:37:41 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:37:41 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:37:41 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 16:37:41 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 16:37:41 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:37:41 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 16:37:41 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 16:37:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-38.raw Jul 23 16:37:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.427.0 of the CIB to disk (digest: 334cf98ef373010a650e1981e956305a) Jul 23 16:37:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.AWMvVC (digest: /var/lib/pacemaker/cib/cib.l2nZfD) Jul 23 16:37:41 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[8098] exited with status 0 Jul 23 16:37:41 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[8100] exited with status 0 Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[8107] exited with status 0 Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1176) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 167 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.427.0 2 Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.427.1 (null) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1176, version=0.427.1) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 167 aborted: History refresh | source=process_resource_updates:268 complete=true Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1179) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1180) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1181) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1179, version=0.427.1) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.427.1 2 Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.427.2 (null) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1180, version=0.427.2) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1181, version=0.427.2) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 168, saving inputs in /var/lib/pacemaker/pengine/pe-input-188.bz2 Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 168 (ref=pe_calc-dc-1627083462-633) derived from /var/lib/pacemaker/pengine/pe-input-188.bz2 Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 on an-a02n02 | action 31 Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.427.2 2 Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.427.3 (null) Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=31:168:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;31:168:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:37:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/304, version=0.427.3) Jul 23 16:37:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 89c684909b7c09f837d9c402637346a7 for 0.427.3 (0x5619764cac10 0) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.427.3 2 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.427.4 (null) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:1;31:168:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=210, @rc-code=1, @op-status=0, @exec-time=13317 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/335, version=0.427.5) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 335 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.427.5 2 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.427.6 (null) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/336, version=0.427.6) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 336 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627083475 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 168 aborted by status-2-fail-count-apc_snmp_node1_an-pdu03.start_0 doing create fail-count-apc_snmp_node1_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.427.5 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 168 aborted by status-2-last-failure-apc_snmp_node1_an-pdu03.start_0 doing create last-failure-apc_snmp_node1_an-pdu03#start_0=1627083475: Transient attribute change | cib=0.427.6 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:37:42 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:37:42 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 169, saving inputs in /var/lib/pacemaker/pengine/pe-input-189.bz2 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627083475-635 is obsolete Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:37:42 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:37:42 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n01 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n02 -> an-a02n01 ) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 170, saving inputs in /var/lib/pacemaker/pengine/pe-input-190.bz2 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 170 (ref=pe_calc-dc-1627083475-636) derived from /var/lib/pacemaker/pengine/pe-input-190.bz2 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 on an-a02n02 | action 10 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.427.6 2 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.427.7 (null) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @transition-key=10:170:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;10:170:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/306, version=0.427.7) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.427.7 2 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.427.8 (null) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;10:170:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=211, @rc-code=0, @op-status=0 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/307, version=0.427.8) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 170 action 10 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=211 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 locally on an-a02n01 | action 32 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=32:170:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_start_0 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1187) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu03 action:start call_id:214 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.427.8 2 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.427.9 (null) Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=32:170:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;32:170:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:37:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1187, version=0.427.9) Jul 23 16:38:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 130f07e2a81c383805f699fc49feee3f for 0.427.9 (0x5619764cac10 0) Jul 23 16:38:01 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[8912] error output [ 2021-07-23 16:38:01,694 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:38:01 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[8912] error output [ ] Jul 23 16:38:01 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[8912] error output [ ] Jul 23 16:38:01 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[8912] error output [ 2021-07-23 16:38:01,694 ERROR: Please use '-h' for usage ] Jul 23 16:38:01 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[8912] error output [ ] Jul 23 16:38:01 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[8912] stderr: [ 2021-07-23 16:38:01,694 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:38:01 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[8912] stderr: [ ] Jul 23 16:38:01 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[8912] stderr: [ ] Jul 23 16:38:01 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[8912] stderr: [ 2021-07-23 16:38:01,694 ERROR: Please use '-h' for usage ] Jul 23 16:38:01 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[8912] stderr: [ ] Jul 23 16:38:01 an-a02n01.alteeve.com pacemaker-fenced [121629] (internal_stonith_action_execute) info: Attempt 2 to execute fence_apc_snmp (monitor). remaining timeout is 14 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[9880] error output [ 2021-07-23 16:38:08,852 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[9880] error output [ ] Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[9880] error output [ ] Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[9880] error output [ 2021-07-23 16:38:08,853 ERROR: Please use '-h' for usage ] Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[9880] error output [ ] Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[9880] stderr: [ 2021-07-23 16:38:08,852 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[9880] stderr: [ ] Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[9880] stderr: [ ] Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[9880] stderr: [ 2021-07-23 16:38:08,853 ERROR: Please use '-h' for usage ] Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[9880] stderr: [ ] Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_remaining_timeout) info: Attempted to execute agent fence_apc_snmp (monitor) the maximum number of times (2) allowed Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_operation) notice: Operation 'monitor' [9880] using apc_snmp_node1_an-pdu03 returned -201 (Generic Pacemaker error) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu03 start (call 214) exited with status 1 (execution time 13319ms, queue time 0ms) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for apc_snmp_node1_an-pdu03 on an-a02n01: error | rc=1 call=214 key=apc_snmp_node1_an-pdu03_start_0 confirmed=true cib-update=1188 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1188) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.427.9 2 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.427.10 (null) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:1;32:170:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=214, @rc-code=1, @op-status=0, @exec-time=13319 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/337, version=0.427.11) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 337 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.427.11 2 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.427.12 (null) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/338, version=0.427.12) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 338 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=1627083488 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627083475 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 170 aborted by status-1-fail-count-apc_snmp_node1_an-pdu03.start_0 doing create fail-count-apc_snmp_node1_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.427.11 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 170 aborted by status-1-last-failure-apc_snmp_node1_an-pdu03.start_0 doing create last-failure-apc_snmp_node1_an-pdu03#start_0=1627083488: Transient attribute change | cib=0.427.12 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:37:55 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:37:55 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:37:42 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n01 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n01 ) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 171, saving inputs in /var/lib/pacemaker/pengine/pe-input-191.bz2 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627083488-639 is obsolete Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:37:55 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:37:55 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:37:42 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu03 ( an-a02n01 ) due to node availability Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 172, saving inputs in /var/lib/pacemaker/pengine/pe-input-192.bz2 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 172 (ref=pe_calc-dc-1627083488-640) derived from /var/lib/pacemaker/pengine/pe-input-192.bz2 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 locally on an-a02n01 | action 2 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=2:172:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_stop_0 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu03 action:stop call_id:215 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu03 stop (call 215) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1192) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01: ok | rc=0 call=215 key=apc_snmp_node1_an-pdu03_stop_0 confirmed=true cib-update=1193 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1193) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.427.12 2 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.427.13 (null) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @transition-key=2:172:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;2:172:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1627 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1192, version=0.427.13) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.427.13 2 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.427.14 (null) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;2:172:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=215, @rc-code=0, @op-status=0 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1193, version=0.427.14) Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 172 action 2 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=215 Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 172 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-192.bz2): Complete Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 16:38:08 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:38:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 106e0f51ae843e5bfab8f433f2354e5d for 0.427.14 (0x5619764cac10 0) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.427.14 2 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.428.0 24ef9e143815d27d2a5694cbdb237d4d Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=428, @num_updates=0 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03']/instance_attributes[@id='apc_snmp_node1_an-pdu03-instance_attributes']/nvpair[@id='apc_snmp_node1_an-pdu03-instance_attributes-ip']: @value=10.201.2.3 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03']/instance_attributes[@id='apc_snmp_node1_an-pdu03-instance_attributes']/nvpair[@id='apc_snmp_node1_an-pdu03-instance_attributes-port']: @value=3 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1']: @id=fl-an-a02n01-1, @index=1 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2']: @id=fl-an-a02n01-2, @index=2 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3']: @id=fl-an-a02n01-3, @index=3 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.427.14 -> 0.428.0 from an-a02n01 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.428.0) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 172 aborted by apc_snmp_node1_an-pdu03-instance_attributes-ip doing modify ip=10.201.2.3: Configuration change | cib=0.428.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03']/instance_attributes[@id='apc_snmp_node1_an-pdu03-instance_attributes']/nvpair[@id='apc_snmp_node1_an-pdu03-instance_attributes-ip'] complete=true Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 339 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 340 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 341 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 342 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 343 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 344 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 345 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 346 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1196) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1197) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 1 from fencing topology for an-a02n01 (2 active levels remaining) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 2 from fencing topology for an-a02n01 (2 active levels remaining) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 3 from fencing topology for an-a02n01 (2 active levels remaining) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: modify nvpair[@id='apc_snmp_node1_an-pdu03-instance_attributes-ip'] Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.428.0 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-39.raw Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1196, version=0.428.0) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.428.0 of the CIB to disk (digest: 0cf49edf4cf65fd90e3e04e2bf784142) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu04 has been disabled Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.stR4e4 (digest: /var/lib/pacemaker/cib/cib.Wf3BM3) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.428.0 2 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.428.1 (null) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1197, version=0.428.1) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node1_an-pdu03' to device list (5 active devices) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/339) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/340) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/341) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/342) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/343) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/339, version=0.428.1) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 339 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=1627083319 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=1627083302 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/340, version=0.428.1) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 340 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/341, version=0.428.1) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 341 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/342, version=0.428.1) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 342 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=1627083488 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627083475 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/343, version=0.428.1) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 343 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/344) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/345) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/346) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/344, version=0.428.1) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 344 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=1627083318 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=1627083303 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.428.1 2 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.428.2 (null) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/345, version=0.428.2) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 345 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.428.2 2 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.428.3 (null) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/346, version=0.428.3) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 346 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627083304 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/1199, version=0.428.3) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1200) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1202) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1204) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1206) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-27 offer to an-a02n02 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-27 offer to an-a02n01 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-27 requests from 2 outstanding nodes Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1200, version=0.428.3) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1202, version=0.428.3) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1204, version=0.428.3) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1206, version=0.428.3) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: 1a8d82936eeb747d34e93bd34cab6301 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.428.3 with 0.428.3 from an-a02n01 Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/1209, version=0.428.3) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1210) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1211) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-40.raw Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1210, version=0.428.3) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.428.0 of the CIB to disk (digest: 0cf49edf4cf65fd90e3e04e2bf784142) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.DqyNAf (digest: /var/lib/pacemaker/cib/cib.f0ap6e) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1211, version=0.428.3) Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[29200] exited with status 0 Jul 23 16:44:44 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[29202] exited with status 0 Jul 23 16:44:44 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[29203] exited with status 0 Jul 23 16:44:44 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[29204] exited with status 0 Jul 23 16:44:44 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[29205] exited with status 0 Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[29207] exited with status 0 Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[29208] exited with status 0 Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[29212] exited with status 0 Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 1212) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 1214) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/1212) Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1213) Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/1214) Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.428.3 2 Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.428.4 (null) Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/1212, version=0.428.4) Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.428.4 2 Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.428.5 (null) Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1213, version=0.428.5) Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.428.5 2 Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.428.6 (null) Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/1214, version=0.428.6) Jul 23 16:44:45 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[29214] exited with status 0 Jul 23 16:44:46 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[29216] exited with status 0 Jul 23 16:44:46 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[29220] exited with status 0 Jul 23 16:44:46 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[29231] exited with status 0 Jul 23 16:44:46 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[29232] exited with status 0 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[29234] exited with status 0 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[29235] exited with status 0 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[29236] exited with status 0 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1215) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 172 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.428.6 2 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.428.7 (null) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1215, version=0.428.7) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1218) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1219) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1220) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 172 aborted: History refresh | source=process_resource_updates:268 complete=true Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1218, version=0.428.7) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.428.7 2 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.428.8 (null) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1219, version=0.428.8) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1220, version=0.428.8) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n01 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now 5e7d5a6e316938194ebb385cfee4a6f2 (reload:3.10.2) 0:1;32:170:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n01 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:37:55 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n02 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now 5e7d5a6e316938194ebb385cfee4a6f2 (reload:3.10.2) 0:1;31:168:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:37:42 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 173, saving inputs in /var/lib/pacemaker/pengine/pe-input-193.bz2 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 173 (ref=pe_calc-dc-1627083887-649) derived from /var/lib/pacemaker/pengine/pe-input-193.bz2 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 on an-a02n02 | action 33 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-controld [121633] (te_crm_command) info: Executing crm-event (11) without waiting: clear_failcount on an-a02n02 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-controld [121633] (te_crm_command) info: Executing crm-event (2) locally without waiting: clear_failcount on an-a02n01 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-controld [121633] (update_attrd_clear_failures) info: Asking pacemaker-attrd to clear failure of all operations for apc_snmp_node1_an-pdu03 on cluster node an-a02n01 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section /cib/status/node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0'] to all (origin=local/crmd/1223) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 348 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: 1627083488 -> (unset) | from an-a02n01 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 349 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.428.8 2 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.428.9 (null) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0'] Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n02 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node1_an-pdu03#start_0' delayed: update 348 in progress Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: 1627083475 -> (unset) | from an-a02n02 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node1_an-pdu03#start_0' delayed: update 349 in progress Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section /cib/status/node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0']: OK (rc=0, origin=an-a02n01/crmd/1223, version=0.428.9) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 173 aborted by deletion of lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0']: Resource operation removal | cib=0.428.9 source=process_op_deletion:358 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0'] complete=false Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.428.9 2 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.428.10 (null) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=33:173:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;33:173:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/310, version=0.428.10) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/348) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/349) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.428.10 2 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.428.11 (null) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0'] Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/348, version=0.428.11) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 348 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 173 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.428.11 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0'] complete=false Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 350 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.428.11 2 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.428.12 (null) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0'] Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section /cib/status/node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0']: OK (rc=0, origin=an-a02n02/crmd/311, version=0.428.12) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 173 aborted by deletion of lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0']: Resource operation removal | cib=0.428.12 source=process_op_deletion:358 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0'] complete=false Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.428.12 2 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.428.13 (null) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0'] Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/349, version=0.428.13) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 349 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 173 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.428.13 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0'] complete=false Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627083475 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/350) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 351 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/351) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.428.13 2 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.428.14 (null) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/350, version=0.428.14) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 350 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 173 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.428.14 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] complete=false Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.428.14 2 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.428.15 (null) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=15 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/351, version=0.428.15) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 173 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.428.15 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] complete=false Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 351 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.428.15 2 Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.428.16 (null) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=16 Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;33:173:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=212, @rc-code=0, @op-status=0, @exec-time=366 Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/312, version=0.428.16) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 173 action 33 (apc_snmp_node1_an-pdu03_start_0 on an-a02n02) confirmed: ok | rc=0 call-id=212 Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 173 (Complete=3, Pending=0, Fired=0, Skipped=1, Incomplete=1, Source=/var/lib/pacemaker/pengine/pe-input-193.bz2): Stopped Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Started an-a02n02) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 174, saving inputs in /var/lib/pacemaker/pengine/pe-input-194.bz2 Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 174 (ref=pe_calc-dc-1627083888-653) derived from /var/lib/pacemaker/pengine/pe-input-194.bz2 Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu03_monitor_60000 on an-a02n02 | action 33 Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.428.16 2 Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.428.17 (null) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=17 Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.429.0) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 174 aborted by meta_attributes.apc_snmp_node1_an-pdu03-meta_attributes 'create': Configuration change | cib=0.429.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03'] complete=true Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create primitive[@id='apc_snmp_node1_an-pdu03'] Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.429.0 Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu04 has been disabled Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu03 has been disabled Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-41.raw Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.429.0 of the CIB to disk (digest: 3b6ea266cb83069c8dfd7b1a9f3668c5) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.54vR0u (digest: /var/lib/pacemaker/cib/cib.sx22wv) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu03 ( an-a02n02 ) due to node availability Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 175, saving inputs in /var/lib/pacemaker/pengine/pe-input-195.bz2 Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 175 (ref=pe_calc-dc-1627083909-655) derived from /var/lib/pacemaker/pengine/pe-input-195.bz2 Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 on an-a02n02 | action 32 Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.429.0 2 Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.429.1 (null) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @transition-key=32:175:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;32:175:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/315, version=0.429.1) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.429.1 2 Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.429.2 (null) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;32:175:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=215, @rc-code=0, @op-status=0 Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/316, version=0.429.2) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 175 action 32 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=215 Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 175 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-195.bz2): Complete Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.429.2 2 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.430.0 (null) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=430, @num_updates=0 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2']: @devices=apc_snmp_node1_an-pdu04 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.429.2 -> 0.430.0 from an-a02n02 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.430.0) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 175 aborted by fencing-level.fl-an-a02n01-2 'modify': Configuration change | cib=0.430.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] complete=true Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 352 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 353 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 354 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 355 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 2 from fencing topology for an-a02n01 (2 active levels remaining) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 356 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 357 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 358 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 359 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/352) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/353) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/354) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/355) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/356) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/352, version=0.430.0) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 352 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=1627083319 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=1627083302 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/353, version=0.430.0) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 353 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.430.0 2 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.430.1 (null) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/354, version=0.430.1) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 354 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.430.1 2 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.430.2 (null) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-42.raw Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/355, version=0.430.2) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 355 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/356, version=0.430.2) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 356 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1228) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1229) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.430.0 of the CIB to disk (digest: 846d7065eca3e8dd9e0967d0c2c674e1) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.7rHVGz (digest: /var/lib/pacemaker/cib/cib.nhrhmA) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/357) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/358) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/359) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1228, version=0.430.2) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.430.2 2 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.430.3 (null) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1229, version=0.430.3) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/357, version=0.430.3) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 357 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=1627083318 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=1627083303 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.430.3 2 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.430.4 (null) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/358, version=0.430.4) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 358 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.430.4 2 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.430.5 (null) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/359, version=0.430.5) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 359 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627083304 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/1231, version=0.430.5) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1232) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1232, version=0.430.5) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1234) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1234, version=0.430.5) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1236) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-28 offer to an-a02n02 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-28 offer to an-a02n01 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-28 requests from 2 outstanding nodes Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1236, version=0.430.5) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1238) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1238, version=0.430.5) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: b1e2c513762d587241c086160bf360ee Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.430.5 with 0.430.5 from an-a02n01 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/1241, version=0.430.5) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1242) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1243) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-43.raw Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1242, version=0.430.5) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.430.0 of the CIB to disk (digest: 846d7065eca3e8dd9e0967d0c2c674e1) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.ePbZNL (digest: /var/lib/pacemaker/cib/cib.loWXaJ) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1243, version=0.430.5) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[29273] exited with status 0 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.430.5 2 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.431.0 (null) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03'] Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=431, @num_updates=0 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.431.0) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Device 'apc_snmp_node1_an-pdu03' not found (4 active devices) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-44.raw Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.431.0 of the CIB to disk (digest: aa2e9e029cf99f37b654b2d12d821d43) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.fLSZBT (digest: /var/lib/pacemaker/cib/cib.H9VPdV) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[29275] exited with status 0 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.431.0 2 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.432.0 (null) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=432 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources: Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.432.0) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create resources Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.432.0 Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu04 has been disabled Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node1_an-pdu03' to device list (5 active devices) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-45.raw Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.432.0 of the CIB to disk (digest: be7da1cbb06451652799b7f2f547013c) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.6ITsh5 (digest: /var/lib/pacemaker/cib/cib.MERSV1) Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[29277] exited with status 0 Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[29279] exited with status 0 Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[29280] exited with status 0 Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.432.0 2 Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.433.0 (null) Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=433 Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.433.0) Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.433.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-46.raw Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.433.0 of the CIB to disk (digest: dbb2389475fb8bc8f2aff8c060b5962d) Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.W8iRqs (digest: /var/lib/pacemaker/cib/cib.NLyF2q) Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[29282] exited with status 0 Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.433.0 2 Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.434.0 (null) Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=434 Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.434.0) Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.434.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-47.raw Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.434.0 of the CIB to disk (digest: fd2f4a53cdd6bf09e0b89917f9c59643) Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.1krUMz (digest: /var/lib/pacemaker/cib/cib.iKkcbD) Jul 23 16:45:12 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[29284] exited with status 0 Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[29286] exited with status 0 Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 1244) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/1244) Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1245) Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 1246) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.434.0 2 Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.434.1 (null) Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/1244, version=0.434.1) Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.434.1 2 Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.434.2 (null) Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1245, version=0.434.2) Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/1246) Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.434.2 2 Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.434.3 (null) Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/1246, version=0.434.3) Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[29287] exited with status 0 Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.434.3 2 Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.435.0 (null) Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3'] Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=435, @num_updates=0 Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.435.0) Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.435.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3'] Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-48.raw Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.435.0 of the CIB to disk (digest: c3f47d702834cc172ef81d82d3be3bb6) Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.05C7MR (digest: /var/lib/pacemaker/cib/cib.B8S5aV) Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[29288] exited with status 0 Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.435.0 2 Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.436.0 (null) Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=436 Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.436.0) Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.436.0 for /cib/configuration/fencing-topology Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-49.raw Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.436.0 of the CIB to disk (digest: 0299b702f168d3c7b8d14c9784e73c4b) Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.8FAtL4 (digest: /var/lib/pacemaker/cib/cib.fHIMa3) Jul 23 16:45:13 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[29290] exited with status 0 Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.436.0 2 Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.437.0 (null) Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=437 Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.437.0) Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.437.0 for /cib/configuration/fencing-topology Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-50.raw Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.437.0 of the CIB to disk (digest: 5f22afb4926bf06436816694cedd0269) Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.ZQni3a (digest: /var/lib/pacemaker/cib/cib.NS1ODc) Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[29292] exited with status 0 Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[29294] exited with status 0 Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.437.0 2 Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.438.0 (null) Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=438 Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.438.0) Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.438.0 for /cib/configuration/fencing-topology Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-51.raw Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.438.0 of the CIB to disk (digest: e70e1b4a83341bdff3c786fbf82001d8) Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.JmoJSt (digest: /var/lib/pacemaker/cib/cib.wj93wt) Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[29296] exited with status 0 Jul 23 16:45:14 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[29298] exited with status 0 Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[29299] exited with status 0 Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1247) Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 175 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1250) Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1251) Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1252) Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.438.0 2 Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.438.1 (null) Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1247, version=0.438.1) Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1250, version=0.438.1) Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.438.1 2 Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.438.2 (null) Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1251, version=0.438.2) Jul 23 16:45:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1252, version=0.438.2) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 175 aborted: History refresh | source=process_resource_updates:268 complete=true Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 176, saving inputs in /var/lib/pacemaker/pengine/pe-input-196.bz2 Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 176 (ref=pe_calc-dc-1627083916-664) derived from /var/lib/pacemaker/pengine/pe-input-196.bz2 Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 on an-a02n02 | action 31 Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.438.2 2 Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.438.3 (null) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=31:176:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;31:176:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/319, version=0.438.3) Jul 23 16:45:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 287fe50ecc02715ff2ac66ec4f55e75f for 0.438.3 (0x561975c274d0 0) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.438.3 2 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.438.4 (null) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:1;31:176:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=216, @rc-code=1, @op-status=0, @exec-time=13323 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/361, version=0.438.5) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 361 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.438.5 2 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.438.6 (null) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/362, version=0.438.6) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 362 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627083929 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 176 aborted by status-2-fail-count-apc_snmp_node1_an-pdu03.start_0 doing create fail-count-apc_snmp_node1_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.438.5 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 176 aborted by status-2-last-failure-apc_snmp_node1_an-pdu03.start_0 doing create last-failure-apc_snmp_node1_an-pdu03#start_0=1627083929: Transient attribute change | cib=0.438.6 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:45:16 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:45:16 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 177, saving inputs in /var/lib/pacemaker/pengine/pe-input-197.bz2 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627083929-666 is obsolete Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:45:16 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:45:16 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n01 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n02 -> an-a02n01 ) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 178, saving inputs in /var/lib/pacemaker/pengine/pe-input-198.bz2 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 178 (ref=pe_calc-dc-1627083929-667) derived from /var/lib/pacemaker/pengine/pe-input-198.bz2 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 on an-a02n02 | action 10 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.438.6 2 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.438.7 (null) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @transition-key=10:178:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;10:178:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/321, version=0.438.7) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.438.7 2 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.438.8 (null) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;10:178:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=217, @rc-code=0, @op-status=0 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/322, version=0.438.8) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 178 action 10 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=217 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 locally on an-a02n01 | action 32 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=32:178:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_start_0 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1258) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu03 action:start call_id:216 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.438.8 2 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.438.9 (null) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=32:178:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;32:178:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1258, version=0.438.9) Jul 23 16:45:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 0145b7a9fe177c6bc8994e4cb9a6a327 for 0.438.9 (0x561975c274d0 0) Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[30692] error output [ 2021-07-23 16:45:35,820 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[30692] error output [ ] Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[30692] error output [ ] Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[30692] error output [ 2021-07-23 16:45:35,820 ERROR: Please use '-h' for usage ] Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[30692] error output [ ] Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[30692] stderr: [ 2021-07-23 16:45:35,820 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[30692] stderr: [ ] Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[30692] stderr: [ ] Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[30692] stderr: [ 2021-07-23 16:45:35,820 ERROR: Please use '-h' for usage ] Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[30692] stderr: [ ] Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (internal_stonith_action_execute) info: Attempt 2 to execute fence_apc_snmp (monitor). remaining timeout is 14 Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[31341] error output [ 2021-07-23 16:45:42,979 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[31341] error output [ ] Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[31341] error output [ ] Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[31341] error output [ 2021-07-23 16:45:42,979 ERROR: Please use '-h' for usage ] Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[31341] error output [ ] Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[31341] stderr: [ 2021-07-23 16:45:42,979 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[31341] stderr: [ ] Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[31341] stderr: [ ] Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[31341] stderr: [ 2021-07-23 16:45:42,979 ERROR: Please use '-h' for usage ] Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[31341] stderr: [ ] Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_remaining_timeout) info: Attempted to execute agent fence_apc_snmp (monitor) the maximum number of times (2) allowed Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_operation) notice: Operation 'monitor' [31341] using apc_snmp_node1_an-pdu03 returned -201 (Generic Pacemaker error) Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu03 start (call 216) exited with status 1 (execution time 13319ms, queue time 0ms) Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for apc_snmp_node1_an-pdu03 on an-a02n01: error | rc=1 call=216 key=apc_snmp_node1_an-pdu03_start_0 confirmed=true cib-update=1259 Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1259) Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.438.9 2 Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.438.10 (null) Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:1;32:178:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=216, @rc-code=1, @op-status=0, @exec-time=13319 Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/363, version=0.438.11) Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 363 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.438.11 2 Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.438.12 (null) Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/364, version=0.438.12) Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 364 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=1627083942 Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627083929 Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 178 aborted by status-1-fail-count-apc_snmp_node1_an-pdu03.start_0 doing create fail-count-apc_snmp_node1_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.438.11 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 178 aborted by status-1-last-failure-apc_snmp_node1_an-pdu03.start_0 doing create last-failure-apc_snmp_node1_an-pdu03#start_0=1627083942: Transient attribute change | cib=0.438.12 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:45:29 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:45:29 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:45:16 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n01 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n01 ) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 179, saving inputs in /var/lib/pacemaker/pengine/pe-input-199.bz2 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627083942-670 is obsolete Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:45:29 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:45:29 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:45:16 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu03 ( an-a02n01 ) due to node availability Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 180, saving inputs in /var/lib/pacemaker/pengine/pe-input-200.bz2 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 180 (ref=pe_calc-dc-1627083943-671) derived from /var/lib/pacemaker/pengine/pe-input-200.bz2 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 locally on an-a02n01 | action 2 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=2:180:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_stop_0 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu03 action:stop call_id:217 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu03 stop (call 217) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1263) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01: ok | rc=0 call=217 key=apc_snmp_node1_an-pdu03_stop_0 confirmed=true cib-update=1264 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1264) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.438.12 2 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.438.13 (null) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @transition-key=2:180:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;2:180:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1627 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1263, version=0.438.13) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.438.13 2 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.438.14 (null) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;2:180:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=217, @rc-code=0, @op-status=0 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1264, version=0.438.14) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 180 action 2 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=217 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 180 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-200.bz2): Complete Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 16:45:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: b90ecb3bbb149aed4a10bd7b7ab35a58 for 0.438.14 (0x561975c274d0 0) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] to all (origin=local/crmd/1265) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]: 1627083319 -> (unset) | from an-a02n01 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 365 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]: 1627083302 -> (unset) | from an-a02n01 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node2_an-pdu03#start_0' delayed: update 365 in progress Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 366 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node2_an-pdu03#start_0' delayed: update 366 in progress Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 367 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node1_an-pdu03#start_0' delayed: update 367 in progress Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: 1627083942 -> (unset) | from an-a02n01 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 368 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: 1627083929 -> (unset) | from an-a02n01 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node1_an-pdu03#start_0' delayed: update 368 in progress Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 369 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node2_an-pdu04#start_0' delayed: update 369 in progress Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]: 1627083318 -> (unset) | from an-a02n01 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 370 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]: 1627083303 -> (unset) | from an-a02n01 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node2_an-pdu04#start_0' delayed: update 370 in progress Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 371 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]: 1627083304 -> (unset) | from an-a02n01 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 372 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.438.14 2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.438.15 117ec9e15adb3d9b22c86a66402b18ae Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=15 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: OK (rc=0, origin=an-a02n01/crmd/1265, version=0.438.14) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-controld [121633] (delete_resource) info: Removing resource apc_snmp_node2_an-pdu04 from executor for 115d70cd-1fd9-474e-839e-6b74e8c2335f as root Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_resource_history) info: Clearing resource history for apc_snmp_node2_an-pdu04 on an-a02n01 (via CIB call 1266) | xpath=//node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-controld [121633] (notify_deleted) info: Notifying 115d70cd-1fd9-474e-839e-6b74e8c2335f on an-a02n01 that apc_snmp_node2_an-pdu04 was deleted Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/365) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/366) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/367) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/368) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/369) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] to all (origin=local/crmd/1266) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/370) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/371) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/372) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.438.14 2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.438.15 (null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu03.start_0'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=15 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/365, version=0.438.15) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 365 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=1627083302 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 373 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.438.15 2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.438.16 (null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu03.start_0'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=16 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/366, version=0.438.16) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 366 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 374 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.438.16 2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.438.17 (null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=17 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/367, version=0.438.17) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 367 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 375 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.438.17 2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.438.18 (null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=18 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/368, version=0.438.18) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 368 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627083929 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 376 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.438.18 2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.438.19 (null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu04.start_0'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=19 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/369, version=0.438.19) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 369 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 377 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.438.19 2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.438.20 (null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=20 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: OK (rc=0, origin=an-a02n01/crmd/1266, version=0.438.20) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.438.20 2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.438.21 (null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu04.start_0'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=21 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/370, version=0.438.21) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 370 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=1627083303 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 378 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.438.21 2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.438.22 (null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu04.start_0'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=22 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/371, version=0.438.22) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 371 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.438.22 2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.438.23 (null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu04.start_0'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=23 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/372, version=0.438.23) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 372 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/373) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/374) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/375) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/376) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/377) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 180 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu03.start_0']: Transient attribute change | cib=0.438.15 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu03.start_0'] complete=true Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 180 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu03.start_0']: Transient attribute change | cib=0.438.16 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu03.start_0'] complete=true Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.438.23 2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.438.24 (null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu03.start_0'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=24 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 180 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.438.17 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 180 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.438.18 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/373, version=0.438.24) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 180 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu04.start_0']: Transient attribute change | cib=0.438.19 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu04.start_0'] complete=true Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 373 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 180 aborted by deletion of lrm_resource[@id='apc_snmp_node2_an-pdu04']: Resource state removal | cib=0.438.20 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] complete=true Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 180 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu04.start_0']: Transient attribute change | cib=0.438.21 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu04.start_0'] complete=true Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 180 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.438.22 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu04.start_0'] complete=true Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 180 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.438.23 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu04.start_0'] complete=true Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.438.24 2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.438.25 (null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu03.start_0'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=25 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/374, version=0.438.25) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 180 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu03.start_0']: Transient attribute change | cib=0.438.24 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu03.start_0'] complete=true Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 180 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu03.start_0']: Transient attribute change | cib=0.438.25 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu03.start_0'] complete=true Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 374 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.438.25 2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.438.26 (null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=26 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/375, version=0.438.26) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 375 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.438.26 2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.438.27 (null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=27 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/376, version=0.438.27) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 376 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.438.27 2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.438.28 (null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu04.start_0'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=28 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/377, version=0.438.28) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 377 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1268) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/378) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.438.28 2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.439.0 (null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=439, @num_updates=0 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']: @value=1627084837 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1268, version=0.439.0) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.439.0 2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.439.1 (null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu04.start_0'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/378, version=0.439.1) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 378 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.439.1 2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.439.2 (null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: OK (rc=0, origin=an-a02n02/crmd/323, version=0.439.1) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.439.1 2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.439.2 (null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: OK (rc=0, origin=an-a02n02/crmd/324, version=0.439.2) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n02/crmd/326, version=0.439.2) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.439.2 2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.439.3 (null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/328, version=0.439.2) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (do_local_notify) warning: Asynchronous reply to client crmd failed: Resource temporarily unavailable | rc=11 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/crmd/1280) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.439.2 2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.439.3 (null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/329, version=0.439.3) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n02/crmd/331, version=0.439.3) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.439.3 2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.439.4 (null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/1280, version=0.439.3) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-controld [121633] (delete_resource) info: Removing resource apc_snmp_node1_an-pdu03 from executor for 115d70cd-1fd9-474e-839e-6b74e8c2335f as root Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_resource_history) info: Clearing resource history for apc_snmp_node1_an-pdu03 on an-a02n01 (via CIB call 1281) | xpath=//node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-controld [121633] (notify_deleted) info: Notifying 115d70cd-1fd9-474e-839e-6b74e8c2335f on an-a02n01 that apc_snmp_node1_an-pdu03 was deleted Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.439.3 2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.439.4 (null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n02/crmd/332, version=0.439.3) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.439.3 2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.439.4 (null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n02/crmd/333, version=0.439.4) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n02/crmd/335, version=0.439.4) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/crmd/1281) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.439.4 2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.439.5 (null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-52.raw Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/1281, version=0.439.5) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1283) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] to all (origin=local/crmd/1284) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.439.0 of the CIB to disk (digest: dd23b6c208f099243a6f1704a47b5630) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.NVdtJZ (digest: /var/lib/pacemaker/cib/cib.rcb6KY) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1283, version=0.439.5) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.439.5 2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.439.6 (null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/1284, version=0.439.5) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-controld [121633] (delete_resource) info: Removing resource apc_snmp_node2_an-pdu03 from executor for 115d70cd-1fd9-474e-839e-6b74e8c2335f as root Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_resource_history) info: Clearing resource history for apc_snmp_node2_an-pdu03 on an-a02n01 (via CIB call 1285) | xpath=//node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-controld [121633] (notify_deleted) info: Notifying 115d70cd-1fd9-474e-839e-6b74e8c2335f on an-a02n01 that apc_snmp_node2_an-pdu03 was deleted Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] to all (origin=local/crmd/1285) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1287) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 180 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.438.26 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 180 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.438.27 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 180 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu04.start_0']: Transient attribute change | cib=0.438.28 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu04.start_0'] complete=true Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.439.5 2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.439.6 (null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/1285, version=0.439.6) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1287, version=0.439.6) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 180 aborted by cib-bootstrap-options-last-lrm-refresh doing modify last-lrm-refresh=1627084837: Configuration change | cib=0.439.0 source=te_update_diff_v2:474 path=/cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh'] complete=true Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 180 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu04.start_0']: Transient attribute change | cib=0.439.1 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu04.start_0'] complete=true Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 180 aborted by deletion of lrm_resource[@id='apc_snmp_node2_an-pdu04']: Resource state removal | cib=0.439.2 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] complete=true Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.439.6 2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.439.7 (null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/336, version=0.439.6) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.439.6 2 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.439.7 (null) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/337, version=0.439.7) Jul 23 17:00:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n02/crmd/339, version=0.439.7) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 180 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu03']: Resource state removal | cib=0.439.3 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] complete=true Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 180 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu04']: Resource state removal | cib=0.439.4 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] complete=true Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 180 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu03']: Resource state removal | cib=0.439.5 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] complete=true Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 180 aborted by deletion of lrm_resource[@id='apc_snmp_node2_an-pdu03']: Resource state removal | cib=0.439.6 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] complete=true Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 180 aborted by deletion of lrm_resource[@id='apc_snmp_node2_an-pdu03']: Resource state removal | cib=0.439.7 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] complete=true Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 181, saving inputs in /var/lib/pacemaker/pengine/pe-input-201.bz2 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 181 (ref=pe_calc-dc-1627084838-676) derived from /var/lib/pacemaker/pengine/pe-input-201.bz2 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu04_monitor_0 on an-a02n02 | action 14 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node2_an-pdu03_monitor_0 on an-a02n02 | action 15 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node2_an-pdu03_monitor_0 locally on an-a02n01 | action 11 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_get_rsc_info) info: Agent information for 'apc_snmp_node2_an-pdu03' not in cache Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_rsc_register) info: Cached agent information for 'apc_snmp_node2_an-pdu03' Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for apc_snmp_node2_an-pdu03 on an-a02n01 | transition_key=11:181:7:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu03_monitor_0 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node2_an-pdu04_monitor_0 on an-a02n02 | action 16 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node2_an-pdu04_monitor_0 locally on an-a02n01 | action 12 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_get_rsc_info) info: Agent information for 'apc_snmp_node2_an-pdu04' not in cache Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1300) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_rsc_register) info: Cached agent information for 'apc_snmp_node2_an-pdu04' Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for apc_snmp_node2_an-pdu04 on an-a02n01 | transition_key=12:181:7:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu04_monitor_0 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1301) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu03_monitor_0 on an-a02n02 | action 17 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu03_monitor_0 locally on an-a02n01 | action 13 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_get_rsc_info) info: Agent information for 'apc_snmp_node1_an-pdu03' not in cache Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_rsc_register) info: Cached agent information for 'apc_snmp_node1_an-pdu03' Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=13:181:7:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_monitor_0 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.439.7 2 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.439.8 (null) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1300, version=0.439.8) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1302) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of probe operation for apc_snmp_node2_an-pdu03 on an-a02n01: not running | rc=7 call=224 key=apc_snmp_node2_an-pdu03_monitor_0 confirmed=true cib-update=1303 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.439.8 2 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.439.9 (null) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of probe operation for apc_snmp_node2_an-pdu04 on an-a02n01: not running | rc=7 call=228 key=apc_snmp_node2_an-pdu04_monitor_0 confirmed=true cib-update=1304 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of probe operation for apc_snmp_node1_an-pdu03 on an-a02n01: not running | rc=7 call=232 key=apc_snmp_node1_an-pdu03_monitor_0 confirmed=true cib-update=1305 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1301, version=0.439.9) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.439.9 2 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.439.10 (null) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1302, version=0.439.10) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1303) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1304) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1305) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.439.10 2 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.439.11 (null) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:7;11:181:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=224, @rc-code=7, @op-status=0 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1303, version=0.439.11) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 181 action 11 (apc_snmp_node2_an-pdu03_monitor_0 on an-a02n01) confirmed: not running | rc=7 call-id=224 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.439.11 2 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.439.12 (null) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:7;12:181:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=228, @rc-code=7, @op-status=0 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1304, version=0.439.12) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 181 action 12 (apc_snmp_node2_an-pdu04_monitor_0 on an-a02n01) confirmed: not running | rc=7 call-id=228 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.439.12 2 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.439.13 (null) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:7;13:181:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=232, @rc-code=7, @op-status=0 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1305, version=0.439.13) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 181 action 13 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n01) confirmed: not running | rc=7 call-id=232 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.439.13 2 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.439.14 (null) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/340, version=0.439.14) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.439.14 2 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.439.15 (null) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=15 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:7;14:181:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=225, @rc-code=7, @op-status=0 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/341, version=0.439.15) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 181 action 14 (apc_snmp_node1_an-pdu04_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=225 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.439.15 2 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.439.16 (null) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=16 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/342, version=0.439.16) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.439.16 2 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.439.17 (null) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=17 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:7;15:181:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=229, @rc-code=7, @op-status=0 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/343, version=0.439.17) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 181 action 15 (apc_snmp_node2_an-pdu03_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=229 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.439.17 2 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.439.18 (null) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=18 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/344, version=0.439.18) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.439.18 2 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.439.19 (null) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=19 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:7;16:181:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=233, @rc-code=7, @op-status=0 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/345, version=0.439.19) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 181 action 16 (apc_snmp_node2_an-pdu04_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=233 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.439.19 2 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.439.20 (null) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=20 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/346, version=0.439.20) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.439.20 2 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.439.21 (null) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=21 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:7;17:181:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=237, @rc-code=7, @op-status=0 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/347, version=0.439.21) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 181 action 17 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=237 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 on an-a02n02 | action 38 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.439.21 2 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.439.22 (null) Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=22 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @transition-key=38:181:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;38:181:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1 Jul 23 17:00:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/348, version=0.439.22) Jul 23 17:00:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 71979e837b82487bd81c1a6045c41994 for 0.439.22 (0x5619763f4e90 0) Jul 23 17:00:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:00:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.439.22 2 Jul 23 17:00:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.440.0 (null) Jul 23 17:00:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=440, @num_updates=0 Jul 23 17:00:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03']: Jul 23 17:00:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:00:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:00:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.440.0) Jul 23 17:00:48 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 181 aborted by meta_attributes.apc_snmp_node1_an-pdu03-meta_attributes 'create': Configuration change | cib=0.440.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03'] complete=false Jul 23 17:00:48 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create primitive[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:00:48 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.440.0 Jul 23 17:00:48 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu04 has been disabled Jul 23 17:00:48 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 17:00:48 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 17:00:48 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu03 has been disabled Jul 23 17:00:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-53.raw Jul 23 17:00:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.440.0 of the CIB to disk (digest: c344487aad296c3ee06f11d2c15fa809) Jul 23 17:00:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.VtMqRn (digest: /var/lib/pacemaker/cib/cib.fcb6lo) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.440.0 2 Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.440.1 (null) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:1;38:181:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=238, @rc-code=1, @op-status=0, @exec-time=13316 Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/379, version=0.440.2) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 379 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.440.2 2 Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.440.3 (null) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/380, version=0.440.3) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 380 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627084851 Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 181 aborted by status-2-fail-count-apc_snmp_node1_an-pdu03.start_0 doing create fail-count-apc_snmp_node1_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.440.2 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 181 aborted by status-2-last-failure-apc_snmp_node1_an-pdu03.start_0 doing create last-failure-apc_snmp_node1_an-pdu03#start_0=1627084851: Transient attribute change | cib=0.440.3 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:00:38 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:00:38 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu03 ( an-a02n02 ) due to node availability Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 182, saving inputs in /var/lib/pacemaker/pengine/pe-input-202.bz2 Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627084851-685 is obsolete Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:00:38 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:00:38 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu03 ( an-a02n02 ) due to node availability Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 183, saving inputs in /var/lib/pacemaker/pengine/pe-input-203.bz2 Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 183 (ref=pe_calc-dc-1627084851-686) derived from /var/lib/pacemaker/pengine/pe-input-203.bz2 Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 on an-a02n02 | action 11 Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.440.3 2 Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.440.4 (null) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @transition-key=11:183:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;11:183:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/350, version=0.440.4) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.440.4 2 Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.440.5 (null) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;11:183:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=239, @rc-code=0, @op-status=0 Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/351, version=0.440.5) Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 183 action 11 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=239 Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 183 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-203.bz2): Complete Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:00:51 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:00:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: d565ae664b2722782ef08c2bf48377bf for 0.440.5 (0x56197608e6d0 0) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 381 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: 1627084851 -> (unset) | from an-a02n01 Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 382 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/381) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/382) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.440.5 2 Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.440.6 (null) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/381, version=0.440.6) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 183 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.440.6 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 381 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.440.6 2 Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.440.7 (null) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/382, version=0.440.7) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 183 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.440.7 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 382 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.440.7 2 Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.440.8 (null) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/352, version=0.440.7) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.440.7 2 Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.440.8 (null) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/353, version=0.440.8) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.440.8 2 Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.441.0 (null) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=441, @num_updates=0 Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']: @value=1627084857 Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n02/crmd/355, version=0.441.0) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-54.raw Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.441.0 of the CIB to disk (digest: 9314709f4bb1ef70f4b7f27eb470c40a) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.OXr2KV (digest: /var/lib/pacemaker/cib/cib.3fIKUU) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 183 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu03']: Resource state removal | cib=0.440.8 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] complete=true Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 183 aborted by cib-bootstrap-options-last-lrm-refresh doing modify last-lrm-refresh=1627084857: Configuration change | cib=0.441.0 source=te_update_diff_v2:474 path=/cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh'] complete=true Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:00:38 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:00:57 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 184, saving inputs in /var/lib/pacemaker/pengine/pe-input-204.bz2 Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627084857-688 is obsolete Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 185, saving inputs in /var/lib/pacemaker/pengine/pe-input-205.bz2 Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 185 (ref=pe_calc-dc-1627084858-689) derived from /var/lib/pacemaker/pengine/pe-input-205.bz2 Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu03_monitor_0 on an-a02n02 | action 11 Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.441.0 2 Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.441.1 (null) Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/357, version=0.441.1) Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.441.1 2 Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.441.2 (null) Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:7;11:185:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=244, @rc-code=7, @op-status=0 Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/358, version=0.441.2) Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 185 action 11 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=244 Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 185 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-205.bz2): Complete Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:00:58 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:01:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 19dd0018dba7660f6967aabfc6910df1 for 0.441.2 (0x5619764c2590 0) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.441.2 2 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.442.0 (null) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=442, @num_updates=0 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03']/instance_attributes[@id='apc_snmp_node1_an-pdu03-instance_attributes']/nvpair[@id='apc_snmp_node1_an-pdu03-instance_attributes-ip']: @value=10.201.2.3 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03']/instance_attributes[@id='apc_snmp_node1_an-pdu03-instance_attributes']/nvpair[@id='apc_snmp_node1_an-pdu03-instance_attributes-port']: @value=3 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1']: @id=fl-an-a02n01-1, @index=1 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2']: @id=fl-an-a02n01-2, @index=2 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3']: @id=fl-an-a02n01-3, @index=3 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.441.2 -> 0.442.0 from an-a02n01 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.442.0) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 185 aborted by apc_snmp_node1_an-pdu03-instance_attributes-ip doing modify ip=10.201.2.3: Configuration change | cib=0.442.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03']/instance_attributes[@id='apc_snmp_node1_an-pdu03-instance_attributes']/nvpair[@id='apc_snmp_node1_an-pdu03-instance_attributes-ip'] complete=true Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 1 from fencing topology for an-a02n01 (2 active levels remaining) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 383 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 2 from fencing topology for an-a02n01 (2 active levels remaining) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 384 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 3 from fencing topology for an-a02n01 (2 active levels remaining) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 385 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: modify nvpair[@id='apc_snmp_node1_an-pdu03-instance_attributes-ip'] Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.442.0 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 386 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 387 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 388 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 389 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 390 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1317) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1318) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/383) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/384) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/385) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/386) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/387) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu04 has been disabled Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu03 has been disabled Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1317, version=0.442.0) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.442.0 2 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.442.1 (null) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1318, version=0.442.1) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.442.1 2 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.442.2 (null) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/383, version=0.442.2) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 383 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.442.2 2 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.442.3 (null) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/384, version=0.442.3) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 384 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.442.3 2 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.442.4 (null) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/385, version=0.442.4) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 385 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.442.4 2 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.442.5 (null) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/386, version=0.442.5) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 386 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.442.5 2 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.442.6 (null) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/387, version=0.442.6) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/388) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 387 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/389) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/390) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.442.6 2 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.442.7 (null) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/388, version=0.442.7) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 388 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.442.7 2 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.442.8 (null) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/389, version=0.442.8) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 389 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.442.8 2 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.442.9 (null) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/390, version=0.442.9) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 390 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-55.raw Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.442.0 of the CIB to disk (digest: a82f36c057188d7837cd3e42c2048dc8) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.hkNcJB (digest: /var/lib/pacemaker/cib/cib.3mW5Ux) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/1320, version=0.442.9) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1321) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1323) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1325) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-29 offer to an-a02n02 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-29 offer to an-a02n01 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-29 requests from 2 outstanding nodes Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1321, version=0.442.9) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1323, version=0.442.9) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1325, version=0.442.9) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1327) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1327, version=0.442.9) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: 26afeb6eabf7f95dce6d8004d127a8ee Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.442.9 with 0.442.9 from an-a02n01 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/1330, version=0.442.9) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1331) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1332) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-56.raw Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1331, version=0.442.9) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.442.0 of the CIB to disk (digest: a82f36c057188d7837cd3e42c2048dc8) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.ksFmFH (digest: /var/lib/pacemaker/cib/cib.tzidnH) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1332, version=0.442.9) Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[78416] exited with status 0 Jul 23 17:01:29 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[78418] exited with status 0 Jul 23 17:01:30 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[78419] exited with status 0 Jul 23 17:01:30 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[78420] exited with status 0 Jul 23 17:01:30 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[78421] exited with status 0 Jul 23 17:01:30 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[78423] exited with status 0 Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[78424] exited with status 0 Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[78425] exited with status 0 Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 1333) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 1335) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/1333) Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1334) Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/1335) Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.442.9 2 Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.442.10 (null) Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/1333, version=0.442.10) Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.442.10 2 Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.442.11 (null) Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1334, version=0.442.11) Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.442.11 2 Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.442.12 (null) Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/1335, version=0.442.12) Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[78426] exited with status 0 Jul 23 17:01:31 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[78427] exited with status 0 Jul 23 17:01:32 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[78428] exited with status 0 Jul 23 17:01:32 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[78429] exited with status 0 Jul 23 17:01:32 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[78430] exited with status 0 Jul 23 17:01:32 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[78432] exited with status 0 Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[78433] exited with status 0 Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[78434] exited with status 0 Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1336) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 185 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.442.12 2 Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.442.13 (null) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1336, version=0.442.13) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 185 aborted: History refresh | source=process_resource_updates:268 complete=true Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1339) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1340) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1341) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1339, version=0.442.13) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.442.13 2 Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.442.14 (null) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1340, version=0.442.14) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1341, version=0.442.14) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 186, saving inputs in /var/lib/pacemaker/pengine/pe-input-206.bz2 Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 186 (ref=pe_calc-dc-1627084893-698) derived from /var/lib/pacemaker/pengine/pe-input-206.bz2 Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 186 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-206.bz2): Complete Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:01:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.442.14 2 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.443.0 (null) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=443, @num_updates=0 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2']: @devices=apc_snmp_node1_an-pdu04 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.442.14 -> 0.443.0 from an-a02n02 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.443.0) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 186 aborted by fencing-level.fl-an-a02n01-2 'modify': Configuration change | cib=0.443.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] complete=true Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 392 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 393 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 394 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 395 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 396 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 397 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 398 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 399 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 2 from fencing topology for an-a02n01 (2 active levels remaining) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1346) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1347) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/392) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/393) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/394) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/395) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/396) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-57.raw Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.443.0 of the CIB to disk (digest: 1dc1215770a6aa958b452e700e4f7d0f) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.AU3QTI (digest: /var/lib/pacemaker/cib/cib.4xy9mG) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1346, version=0.443.0) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.443.0 2 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.443.1 (null) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1347, version=0.443.1) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.443.1 2 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.443.2 (null) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/392, version=0.443.2) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 392 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.443.2 2 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.443.3 (null) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/393, version=0.443.3) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 393 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.443.3 2 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.443.4 (null) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/394, version=0.443.4) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 394 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.443.4 2 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.443.5 (null) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/395, version=0.443.5) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 395 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.443.5 2 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.443.6 (null) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/396, version=0.443.6) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 396 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/1349, version=0.443.6) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1350) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/397) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/398) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/399) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1352) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1350, version=0.443.6) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.443.6 2 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.443.7 (null) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/397, version=0.443.7) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 397 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.443.7 2 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.443.8 (null) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/398, version=0.443.8) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 398 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.443.8 2 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.443.9 (null) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/399, version=0.443.9) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 399 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1352, version=0.443.9) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1354) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1356) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-30 offer to an-a02n02 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-30 offer to an-a02n01 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-30 requests from 2 outstanding nodes Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1354, version=0.443.9) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1356, version=0.443.9) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: 639b7193175b2a3946128c481476a778 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.443.9 with 0.443.9 from an-a02n01 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/1359, version=0.443.9) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1360) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1361) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1360, version=0.443.9) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1361, version=0.443.9) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-58.raw Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.443.0 of the CIB to disk (digest: 1dc1215770a6aa958b452e700e4f7d0f) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.BzUqzQ (digest: /var/lib/pacemaker/cib/cib.KY4M3N) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[78444] exited with status 0 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.443.9 2 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.444.0 (null) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=444, @num_updates=0 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.444.0) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Device 'apc_snmp_node1_an-pdu03' not found (4 active devices) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-59.raw Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.444.0 of the CIB to disk (digest: c0a41b6849946ce88a91441ee4680910) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.jjXQ3X (digest: /var/lib/pacemaker/cib/cib.Azv8k1) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[78446] exited with status 0 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.444.0 2 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.445.0 (null) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=445 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources: Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.445.0) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create resources Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.445.0 Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu04 has been disabled Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node1_an-pdu03' to device list (5 active devices) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-60.raw Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.445.0 of the CIB to disk (digest: a6a3a8e8b9ad2e460204c5a31e00538f) Jul 23 17:01:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.84O289 (digest: /var/lib/pacemaker/cib/cib.RaAoL9) Jul 23 17:01:37 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[78448] exited with status 0 Jul 23 17:01:37 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[78450] exited with status 0 Jul 23 17:01:37 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[78451] exited with status 0 Jul 23 17:01:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.445.0 2 Jul 23 17:01:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.446.0 (null) Jul 23 17:01:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] Jul 23 17:01:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=446 Jul 23 17:01:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.446.0) Jul 23 17:01:37 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.446.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] Jul 23 17:01:37 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 17:01:37 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 17:01:37 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 17:01:37 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 17:01:37 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 17:01:37 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:01:37 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 17:01:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-61.raw Jul 23 17:01:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.446.0 of the CIB to disk (digest: b58bc79a7e94758372711b63d06ffdd6) Jul 23 17:01:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.8P9Jxu (digest: /var/lib/pacemaker/cib/cib.B2zYbw) Jul 23 17:01:37 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[78453] exited with status 0 Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.446.0 2 Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.447.0 e15b8ac81188ce421016e05bf0f94c35 Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=447 Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.447.0) Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.447.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[78455] exited with status 0 Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-62.raw Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.447.0 of the CIB to disk (digest: d9adf45bb8ec8aaf6958d59f97b941f0) Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.4EnxKE (digest: /var/lib/pacemaker/cib/cib.a6vfLD) Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[78457] exited with status 0 Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 1362) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/1362) Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1363) Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 1364) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.447.0 2 Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.447.1 (null) Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/1362, version=0.447.1) Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.447.1 2 Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.447.2 (null) Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1363, version=0.447.2) Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/1364) Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.447.2 2 Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.447.3 (null) Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/1364, version=0.447.3) Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[78458] exited with status 0 Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.447.3 2 Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.448.0 (null) Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3'] Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=448, @num_updates=0 Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.448.0) Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.448.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3'] Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-63.raw Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.448.0 of the CIB to disk (digest: 9a1dcb68856a1e4782a3648571bf4a29) Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.LD6HEW (digest: /var/lib/pacemaker/cib/cib.PawNs0) Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[78459] exited with status 0 Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.448.0 2 Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.449.0 (null) Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=449 Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.449.0) Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.449.0 for /cib/configuration/fencing-topology Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-64.raw Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.449.0 of the CIB to disk (digest: 1939ce03fe56d78b4198c8f20d117487) Jul 23 17:01:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.cV0027 (digest: /var/lib/pacemaker/cib/cib.h6imH9) Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[78461] exited with status 0 Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.449.0 2 Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.450.0 (null) Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=450 Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.450.0) Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.450.0 for /cib/configuration/fencing-topology Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-65.raw Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.450.0 of the CIB to disk (digest: 132ddc0433f5cd73d8c64e16e34ba8f1) Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.NM8Emi (digest: /var/lib/pacemaker/cib/cib.p0bOTg) Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[78463] exited with status 0 Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[78466] exited with status 0 Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.450.0 2 Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.451.0 (null) Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=451 Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.451.0) Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.451.0 for /cib/configuration/fencing-topology Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-66.raw Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.451.0 of the CIB to disk (digest: 3d095fd5f66804777228b964c3c41e17) Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.0wOnqc (digest: /var/lib/pacemaker/cib/cib.XYG5qb) Jul 23 17:01:39 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[78468] exited with status 0 Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[78477] exited with status 0 Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[78478] exited with status 0 Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1365) Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 186 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1368) Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1369) Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1370) Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.451.0 2 Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.451.1 (null) Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1365, version=0.451.1) Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1368, version=0.451.1) Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.451.1 2 Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.451.2 (null) Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1369, version=0.451.2) Jul 23 17:01:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1370, version=0.451.2) Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 186 aborted: History refresh | source=process_resource_updates:268 complete=true Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 187, saving inputs in /var/lib/pacemaker/pengine/pe-input-207.bz2 Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 187 (ref=pe_calc-dc-1627084901-706) derived from /var/lib/pacemaker/pengine/pe-input-207.bz2 Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 on an-a02n02 | action 31 Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.451.2 2 Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.451.3 (null) Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=31:187:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;31:187:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 17:01:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/363, version=0.451.3) Jul 23 17:01:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 046a6481cfee4f1ee29f1367a4758ca6 for 0.451.3 (0x5619761a6630 0) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.451.3 2 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.451.4 (null) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:1;31:187:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=245, @rc-code=1, @op-status=0, @exec-time=13319 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/401, version=0.451.5) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 401 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.451.5 2 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.451.6 (null) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/402, version=0.451.6) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 402 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627084914 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 187 aborted by status-2-fail-count-apc_snmp_node1_an-pdu03.start_0 doing create fail-count-apc_snmp_node1_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.451.5 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 187 aborted by status-2-last-failure-apc_snmp_node1_an-pdu03.start_0 doing create last-failure-apc_snmp_node1_an-pdu03#start_0=1627084914: Transient attribute change | cib=0.451.6 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:01:41 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:01:41 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 188, saving inputs in /var/lib/pacemaker/pengine/pe-input-208.bz2 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627084914-708 is obsolete Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:01:41 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:01:41 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n01 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n02 -> an-a02n01 ) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 189, saving inputs in /var/lib/pacemaker/pengine/pe-input-209.bz2 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 189 (ref=pe_calc-dc-1627084914-709) derived from /var/lib/pacemaker/pengine/pe-input-209.bz2 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 on an-a02n02 | action 10 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.451.6 2 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.451.7 (null) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @transition-key=10:189:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;10:189:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/365, version=0.451.7) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.451.7 2 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.451.8 (null) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;10:189:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=246, @rc-code=0, @op-status=0 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/366, version=0.451.8) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 189 action 10 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=246 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 locally on an-a02n01 | action 32 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=32:189:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_start_0 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1376) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu03 action:start call_id:233 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.451.8 2 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.451.9 (null) Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=32:189:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;32:189:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 17:01:54 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1376, version=0.451.9) Jul 23 17:01:59 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 0d3a5a196b0ccb57e3426dc506c513d4 for 0.451.9 (0x5619761a6630 0) Jul 23 17:02:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[78501] error output [ 2021-07-23 17:02:00,973 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:02:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[78501] error output [ ] Jul 23 17:02:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[78501] error output [ ] Jul 23 17:02:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[78501] error output [ 2021-07-23 17:02:00,973 ERROR: Please use '-h' for usage ] Jul 23 17:02:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[78501] error output [ ] Jul 23 17:02:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[78501] stderr: [ 2021-07-23 17:02:00,973 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:02:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[78501] stderr: [ ] Jul 23 17:02:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[78501] stderr: [ ] Jul 23 17:02:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[78501] stderr: [ 2021-07-23 17:02:00,973 ERROR: Please use '-h' for usage ] Jul 23 17:02:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[78501] stderr: [ ] Jul 23 17:02:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (internal_stonith_action_execute) info: Attempt 2 to execute fence_apc_snmp (monitor). remaining timeout is 14 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[79426] error output [ 2021-07-23 17:02:08,133 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[79426] error output [ ] Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[79426] error output [ ] Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[79426] error output [ 2021-07-23 17:02:08,133 ERROR: Please use '-h' for usage ] Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[79426] error output [ ] Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[79426] stderr: [ 2021-07-23 17:02:08,133 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[79426] stderr: [ ] Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[79426] stderr: [ ] Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[79426] stderr: [ 2021-07-23 17:02:08,133 ERROR: Please use '-h' for usage ] Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[79426] stderr: [ ] Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_remaining_timeout) info: Attempted to execute agent fence_apc_snmp (monitor) the maximum number of times (2) allowed Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_operation) notice: Operation 'monitor' [79426] using apc_snmp_node1_an-pdu03 returned -201 (Generic Pacemaker error) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu03 start (call 233) exited with status 1 (execution time 13321ms, queue time 0ms) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for apc_snmp_node1_an-pdu03 on an-a02n01: error | rc=1 call=233 key=apc_snmp_node1_an-pdu03_start_0 confirmed=true cib-update=1377 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1377) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.451.9 2 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.451.10 (null) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:1;32:189:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=233, @rc-code=1, @op-status=0, @exec-time=13321 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/403, version=0.451.11) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 403 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.451.11 2 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.451.12 (null) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/404, version=0.451.12) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 404 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=1627084928 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627084914 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 189 aborted by status-1-fail-count-apc_snmp_node1_an-pdu03.start_0 doing create fail-count-apc_snmp_node1_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.451.11 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 189 aborted by status-1-last-failure-apc_snmp_node1_an-pdu03.start_0 doing create last-failure-apc_snmp_node1_an-pdu03#start_0=1627084928: Transient attribute change | cib=0.451.12 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:01:54 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:01:54 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:01:41 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n01 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n01 ) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 190, saving inputs in /var/lib/pacemaker/pengine/pe-input-210.bz2 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627084928-712 is obsolete Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:01:54 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:01:54 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:01:41 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu03 ( an-a02n01 ) due to node availability Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 191, saving inputs in /var/lib/pacemaker/pengine/pe-input-211.bz2 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 191 (ref=pe_calc-dc-1627084928-713) derived from /var/lib/pacemaker/pengine/pe-input-211.bz2 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 locally on an-a02n01 | action 2 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=2:191:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_stop_0 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1381) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu03 action:stop call_id:234 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu03 stop (call 234) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01: ok | rc=0 call=234 key=apc_snmp_node1_an-pdu03_stop_0 confirmed=true cib-update=1382 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1382) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.451.12 2 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.451.13 (null) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @transition-key=2:191:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;2:191:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1627 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1381, version=0.451.13) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.451.13 2 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.451.14 (null) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;2:191:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=234, @rc-code=0, @op-status=0 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1382, version=0.451.14) Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 191 action 2 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=234 Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 191 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-211.bz2): Complete Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:02:08 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:02:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 56d37f2c3761d84d820f447fa508275d for 0.451.14 (0x5619761a6630 0) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.451.14 2 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.452.0 (null) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=452, @num_updates=0 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1']: @id=fl-an-a02n01-1, @index=1 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2']: @devices=apc_snmp_node1_an-pdu04, @id=fl-an-a02n01-2, @index=2 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3']: @id=fl-an-a02n01-3, @index=3 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.451.14 -> 0.452.0 from an-a02n01 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.452.0) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 191 aborted by fencing-level.fl-an-a02n01-1 'modify': Configuration change | cib=0.452.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] complete=true Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 1 from fencing topology for an-a02n01 (2 active levels remaining) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 2 from fencing topology for an-a02n01 (2 active levels remaining) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 3 from fencing topology for an-a02n01 (2 active levels remaining) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 405 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 406 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 407 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 408 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 409 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 410 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 411 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 412 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1385) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1386) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-67.raw Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.452.0 of the CIB to disk (digest: fef58c32e5fbf2d4d20d78328d48e553) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1385, version=0.452.0) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.V8ymZo (digest: /var/lib/pacemaker/cib/cib.xy1Hfq) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.452.0 2 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.452.1 (null) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1386, version=0.452.1) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/405) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/406) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/407) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/408) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/409) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.452.1 2 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.452.2 (null) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/405, version=0.452.2) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 405 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.452.2 2 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.452.3 (null) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/406, version=0.452.3) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 406 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/407, version=0.452.3) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 407 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/408, version=0.452.3) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 408 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=1627084928 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627084914 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.452.3 2 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.452.4 (null) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/409, version=0.452.4) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 409 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/410) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/411) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/412) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.452.4 2 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.452.5 (null) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/410, version=0.452.5) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 410 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.452.5 2 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.452.6 (null) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/411, version=0.452.6) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 411 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.452.6 2 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.452.7 (null) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/412, version=0.452.7) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 412 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/1388, version=0.452.7) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1389) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1389, version=0.452.7) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1391) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1393) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1391, version=0.452.7) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1393, version=0.452.7) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-31 offer to an-a02n02 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-31 offer to an-a02n01 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-31 requests from 2 outstanding nodes Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1395) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1395, version=0.452.7) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: 03bed2fdca4c397b4415cc6cb5bf2b42 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.452.7 with 0.452.7 from an-a02n01 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/1398, version=0.452.7) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1399) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1400) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-68.raw Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1399, version=0.452.7) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.452.0 of the CIB to disk (digest: fef58c32e5fbf2d4d20d78328d48e553) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.siBZZS (digest: /var/lib/pacemaker/cib/cib.Uq0N8U) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1400, version=0.452.7) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[80031] exited with status 0 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/cibadmin/2) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.452.7 2 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.453.0 (null) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=453, @num_updates=0 Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.453.0) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Removed 'apc_snmp_node1_an-pdu03' from device list (4 active devices) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-69.raw Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.453.0 of the CIB to disk (digest: fac417317a65c0ae6a967887fb475e13) Jul 23 17:02:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.EGa0qc (digest: /var/lib/pacemaker/cib/cib.qYqX8e) Jul 23 17:02:18 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80034] exited with status 0 Jul 23 17:02:18 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80045] exited with status 0 Jul 23 17:02:18 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80061] exited with status 0 Jul 23 17:02:18 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[80062] exited with status 0 Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80064] exited with status 0 Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80068] exited with status 0 Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80072] exited with status 0 Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 1401) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/1401) Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 1403) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.453.0 2 Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.453.1 (null) Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/1401, version=0.453.1) Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1402) Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/1403) Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.453.1 2 Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.453.2 (null) Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1402, version=0.453.2) Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.453.2 2 Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.453.3 (null) Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/1403, version=0.453.3) Jul 23 17:02:19 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[80092] exited with status 0 Jul 23 17:02:20 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80093] exited with status 0 Jul 23 17:02:20 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80101] exited with status 0 Jul 23 17:02:20 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80243] exited with status 0 Jul 23 17:02:20 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[80258] exited with status 0 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80266] exited with status 0 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80272] exited with status 0 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80275] exited with status 0 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1404) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 191 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.453.3 2 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.453.4 (null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1404, version=0.453.4) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1407) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1408) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1409) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 191 aborted: History refresh | source=process_resource_updates:268 complete=true Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1407, version=0.453.4) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.453.4 2 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.453.5 (null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1408, version=0.453.5) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1409, version=0.453.5) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n01 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now f2317cad3d54cec5d7d7aa7d0bf35cf8 (reload:3.10.2) 0:1;32:189:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n01 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:01:54 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n02 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now f2317cad3d54cec5d7d7aa7d0bf35cf8 (reload:3.10.2) 0:1;31:187:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:01:41 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node1_an-pdu03 from an-a02n01 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node1_an-pdu03 from an-a02n02 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n01 because it is orphaned | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02 because it is orphaned | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 192, saving inputs in /var/lib/pacemaker/pengine/pe-input-212.bz2 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 192 (ref=pe_calc-dc-1627084941-722) derived from /var/lib/pacemaker/pengine/pe-input-212.bz2 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating delete operation apc_snmp_node1_an-pdu03_delete_0 on an-a02n02 | action 14 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating delete operation apc_snmp_node1_an-pdu03_delete_0 locally on an-a02n01 | action 13 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/crmd/1412) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.453.5 2 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.453.6 (null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/1412, version=0.453.5) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (delete_resource) info: Removing resource apc_snmp_node1_an-pdu03 from executor for tengine Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_resource_history) info: Clearing resource history for apc_snmp_node1_an-pdu03 on an-a02n01 (via CIB call 1413) | xpath=//node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (notify_deleted) info: Notifying tengine on localhost that apc_snmp_node1_an-pdu03 was deleted Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 192 action 13 (apc_snmp_node1_an-pdu03_delete_0 on an-a02n01) confirmed: ok | rc=0 call-id=0 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (te_crm_command) info: Executing crm-event (11) without waiting: clear_failcount on an-a02n02 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (te_crm_command) info: Executing crm-event (2) locally without waiting: clear_failcount on an-a02n01 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/crmd/1413) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (update_attrd_clear_failures) info: Asking pacemaker-attrd to clear failure of all operations for apc_snmp_node1_an-pdu03 on cluster node an-a02n01 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 414 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: 1627084928 -> (unset) | from an-a02n01 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 415 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.453.5 2 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.453.6 (null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/369, version=0.453.5) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 192 action 14 (apc_snmp_node1_an-pdu03_delete_0 on an-a02n02) confirmed: ok | rc=0 call-id=0 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 192 (Complete=4, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-212.bz2): Complete Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n02 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node1_an-pdu03#start_0' delayed: update 414 in progress Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: 1627084914 -> (unset) | from an-a02n02 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node1_an-pdu03#start_0' delayed: update 415 in progress Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.453.5 2 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.453.6 (null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/1413, version=0.453.6) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section /cib/status/node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0'] to all (origin=local/crmd/1414) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 192 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu03']: Resource state removal | cib=0.453.6 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] complete=true Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/414) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/415) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.453.6 2 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.453.7 (null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/370, version=0.453.7) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 192 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu03']: Resource state removal | cib=0.453.7 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] complete=true Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section /cib/status/node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0']: OK (rc=0, origin=an-a02n02/crmd/371, version=0.453.7) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section /cib/status/node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0']: OK (rc=0, origin=an-a02n01/crmd/1414, version=0.453.7) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.453.7 2 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.453.8 (null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/414, version=0.453.8) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 414 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 192 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.453.8 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 416 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.453.8 2 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.453.9 (null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/415, version=0.453.9) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 192 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.453.9 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 415 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627084914 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 417 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/416) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/417) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.453.9 2 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.453.10 (null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/416, version=0.453.10) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 416 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.453.10 2 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.453.11 (null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/417, version=0.453.11) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 417 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 192 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.453.10 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 192 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.453.11 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 193, saving inputs in /var/lib/pacemaker/pengine/pe-input-213.bz2 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627084941-728 is obsolete Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 194, saving inputs in /var/lib/pacemaker/pengine/pe-input-214.bz2 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 194 (ref=pe_calc-dc-1627084941-729) derived from /var/lib/pacemaker/pengine/pe-input-214.bz2 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 194 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-214.bz2): Complete Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.453.11 2 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.454.0 (null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=454, @num_updates=0 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.453.11 -> 0.454.0 from an-a02n01 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.454.0) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 194 aborted by deletion of fencing-level[@id='fl-an-a02n01-2']: Configuration change | cib=0.454.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] complete=true Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 418 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 419 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 420 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 421 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 422 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 423 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 424 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 425 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.454.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1423) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1424) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/418) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/419) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/420) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/421) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/422) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1423, version=0.454.0) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.454.0 2 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.454.1 (null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1424, version=0.454.1) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.454.1 2 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.454.2 (null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/418, version=0.454.2) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 418 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.454.2 2 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.454.3 (null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/419, version=0.454.3) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 419 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.454.3 2 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.454.4 (null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/420, version=0.454.4) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 420 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.454.4 2 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.454.5 (null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/421, version=0.454.5) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 421 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.454.5 2 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.454.6 (null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/422, version=0.454.6) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 422 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/423) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/424) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/425) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.454.6 2 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.454.7 (null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/423, version=0.454.7) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 423 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.454.7 2 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.454.8 (null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/424, version=0.454.8) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 424 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.454.8 2 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.454.9 (null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/425, version=0.454.9) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 425 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/1426, version=0.454.9) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1427) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1427, version=0.454.9) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1429) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1429, version=0.454.9) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1431) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-32 offer to an-a02n02 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-32 offer to an-a02n01 Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-32 requests from 2 outstanding nodes Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1431, version=0.454.9) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1433) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1433, version=0.454.9) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-70.raw Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.454.0 of the CIB to disk (digest: 7b7443d6de39b536e7457bcf4e68e19c) Jul 23 17:02:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.W6irB4 (digest: /var/lib/pacemaker/cib/cib.EUlC75) Jul 23 17:02:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: c959fca94857e9814057c0697327279a Jul 23 17:02:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.454.9 with 0.454.9 from an-a02n01 Jul 23 17:02:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/1436, version=0.454.9) Jul 23 17:02:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1437) Jul 23 17:02:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1438) Jul 23 17:02:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-71.raw Jul 23 17:02:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1437, version=0.454.9) Jul 23 17:02:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.454.0 of the CIB to disk (digest: 8d78944fa18458a630f7f972f1adb5a4) Jul 23 17:02:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.ggzcVj (digest: /var/lib/pacemaker/cib/cib.AvFJHm) Jul 23 17:02:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1438, version=0.454.9) Jul 23 17:02:22 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[80302] exited with status 0 Jul 23 17:02:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu04'] to all (origin=local/cibadmin/2) Jul 23 17:02:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.454.9 2 Jul 23 17:02:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.455.0 (null) Jul 23 17:02:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:02:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=455, @num_updates=0 Jul 23 17:02:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.455.0) Jul 23 17:02:22 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Device 'apc_snmp_node1_an-pdu04' not found (4 active devices) Jul 23 17:02:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-72.raw Jul 23 17:02:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.455.0 of the CIB to disk (digest: b66387fb8fd28d61deb2bc4b9ff3b668) Jul 23 17:02:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.XOMcuG (digest: /var/lib/pacemaker/cib/cib.JAoUBK) Jul 23 17:02:22 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80304] exited with status 0 Jul 23 17:02:22 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80327] exited with status 0 Jul 23 17:02:22 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80329] exited with status 0 Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[80333] exited with status 0 Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.455.0 2 Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.456.0 (null) Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=456 Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources: Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.456.0) Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create resources Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.456.0 Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node1_an-pdu03' to device list (5 active devices) Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-73.raw Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.456.0 of the CIB to disk (digest: 03d4b8e19935d67eacb9060cd352a9a8) Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.CFcW6m (digest: /var/lib/pacemaker/cib/cib.BElUIo) Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80337] exited with status 0 Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80360] exited with status 0 Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.456.0 2 Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.457.0 (null) Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=457 Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.457.0) Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.457.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-74.raw Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.457.0 of the CIB to disk (digest: 747f024c7bbd18a950c977779c668b4d) Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.gTZjG0 (digest: /var/lib/pacemaker/cib/cib.JHx71Y) Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80374] exited with status 0 Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 1439) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/1439) Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 1441) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.457.0 2 Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.457.1 (null) Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/1439, version=0.457.1) Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1440) Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/1441) Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.457.1 2 Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.457.2 (null) Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1440, version=0.457.2) Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.457.2 2 Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.457.3 (null) Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/1441, version=0.457.3) Jul 23 17:02:23 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[80382] exited with status 0 Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.457.3 2 Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.458.0 (null) Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3'] Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=458, @num_updates=0 Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.458.0) Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.458.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3'] Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-75.raw Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.458.0 of the CIB to disk (digest: 7391185eb5632643198ad53327aa2bca) Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.nqBfsE (digest: /var/lib/pacemaker/cib/cib.VL9p3G) Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80383] exited with status 0 Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80391] exited with status 0 Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.458.0 2 Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.459.0 (null) Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=459 Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.459.0) Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.459.0 for /cib/configuration/fencing-topology Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-76.raw Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.459.0 of the CIB to disk (digest: a729c8e462430c4191cc82a6c8795665) Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.Kjqvnv (digest: /var/lib/pacemaker/cib/cib.KEb0Hv) Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80396] exited with status 0 Jul 23 17:02:24 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[80403] exited with status 0 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80412] exited with status 0 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80419] exited with status 0 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80435] exited with status 0 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1442) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 194 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.459.0 2 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.459.1 (null) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1442, version=0.459.1) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 194 aborted: History refresh | source=process_resource_updates:268 complete=true Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1445) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1446) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1447) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1445, version=0.459.1) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.459.1 2 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.459.2 (null) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1446, version=0.459.2) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1447, version=0.459.2) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node1_an-pdu04 from an-a02n01 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node1_an-pdu04 from an-a02n02 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 195, saving inputs in /var/lib/pacemaker/pengine/pe-input-215.bz2 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 195 (ref=pe_calc-dc-1627084945-737) derived from /var/lib/pacemaker/pengine/pe-input-215.bz2 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu03_monitor_0 on an-a02n02 | action 14 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu03_monitor_0 locally on an-a02n01 | action 13 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_get_rsc_info) info: Agent information for 'apc_snmp_node1_an-pdu03' not in cache Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_rsc_register) info: Cached agent information for 'apc_snmp_node1_an-pdu03' Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=13:195:7:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_monitor_0 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating delete operation apc_snmp_node1_an-pdu04_delete_0 on an-a02n02 | action 12 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating delete operation apc_snmp_node1_an-pdu04_delete_0 locally on an-a02n01 | action 11 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1450) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] to all (origin=local/crmd/1451) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.459.2 2 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.459.3 (null) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1450, version=0.459.3) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.459.3 2 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.459.4 (null) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n01/crmd/1451, version=0.459.3) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-controld [121633] (delete_resource) info: Removing resource apc_snmp_node1_an-pdu04 from executor for tengine Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_resource_history) info: Clearing resource history for apc_snmp_node1_an-pdu04 on an-a02n01 (via CIB call 1452) | xpath=//node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-controld [121633] (notify_deleted) info: Notifying tengine on localhost that apc_snmp_node1_an-pdu04 was deleted Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 195 action 11 (apc_snmp_node1_an-pdu04_delete_0 on an-a02n01) confirmed: ok | rc=0 call-id=0 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.459.3 2 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.459.4 (null) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of probe operation for apc_snmp_node1_an-pdu03 on an-a02n01: not running | rc=7 call=239 key=apc_snmp_node1_an-pdu03_monitor_0 confirmed=true cib-update=1453 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/374, version=0.459.4) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] to all (origin=local/crmd/1452) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1453) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.459.4 2 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.459.5 (null) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n01/crmd/1452, version=0.459.5) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.459.5 2 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.459.6 (null) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:7;13:195:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=239, @rc-code=7, @op-status=0 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1453, version=0.459.6) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 195 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu04']: Resource state removal | cib=0.459.5 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] complete=false Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 195 action 13 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n01) confirmed: not running | rc=7 call-id=239 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.459.6 2 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.459.7 (null) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:7;14:195:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=251, @rc-code=7, @op-status=0 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/375, version=0.459.7) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 195 action 14 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=251 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.459.7 2 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.459.8 (null) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n02/crmd/376, version=0.459.7) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 195 action 12 (apc_snmp_node1_an-pdu04_delete_0 on an-a02n02) confirmed: ok | rc=0 call-id=0 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 195 (Complete=4, Pending=0, Fired=0, Skipped=1, Incomplete=2, Source=/var/lib/pacemaker/pengine/pe-input-215.bz2): Stopped Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.459.7 2 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.459.8 (null) Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:02:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n02/crmd/377, version=0.459.8) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 195 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu04']: Resource state removal | cib=0.459.8 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] complete=true Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node1_an-pdu04 from an-a02n02 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 196, saving inputs in /var/lib/pacemaker/pengine/pe-input-216.bz2 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627084945-743 is obsolete Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 197, saving inputs in /var/lib/pacemaker/pengine/pe-input-217.bz2 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 197 (ref=pe_calc-dc-1627084946-744) derived from /var/lib/pacemaker/pengine/pe-input-217.bz2 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 on an-a02n02 | action 31 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.459.8 2 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.459.9 (null) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @transition-key=31:197:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;31:197:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change= Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/378, version=0.459.9) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.459.9 2 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.460.0 (null) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=460, @num_updates=0 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-2']: @devices=apc_snmp_node2_an-pdu04 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1']: @id=fl-an-a02n01-1, @index=1 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.459.9 -> 0.460.0 from an-a02n01 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.460.0) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 2 from fencing topology for an-a02n02 (2 active levels remaining) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 1 from fencing topology for an-a02n01 (0 active levels remaining) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 427 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 428 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 429 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 430 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 431 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 432 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 197 aborted by fencing-level.fl-an-a02n02-2 'modify': Configuration change | cib=0.460.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-2'] complete=false Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 433 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 434 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/427) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/428) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/429) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/430) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/431) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1458) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1459) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/432) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/433) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/434) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.460.0 2 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.460.1 (null) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/427, version=0.460.1) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 427 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-77.raw Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.460.1 2 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.460.2 (null) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/428, version=0.460.2) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 428 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.460.2 2 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.460.3 (null) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/429, version=0.460.3) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 429 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.460.3 2 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.460.4 (null) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/430, version=0.460.4) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 430 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.460.4 2 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.460.5 (null) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.460.0 of the CIB to disk (digest: a954a061b12f2b671f4357df671dc8d4) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/431, version=0.460.5) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 431 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.QcHK3s (digest: /var/lib/pacemaker/cib/cib.2Rh88p) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1458, version=0.460.5) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.460.5 2 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.460.6 (null) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1459, version=0.460.6) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.460.6 2 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.460.7 (null) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/432, version=0.460.7) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 432 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.460.7 2 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.460.8 (null) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/433, version=0.460.8) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 433 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.460.8 2 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.460.9 (null) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/434, version=0.460.9) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 434 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/1460, version=0.460.9) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1461) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1463) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1461, version=0.460.9) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1463, version=0.460.9) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1465) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-33 offer to an-a02n02 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-33 offer to an-a02n01 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-33 requests from 2 outstanding nodes Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1465, version=0.460.9) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1467) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1467, version=0.460.9) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: 9b3222f83f4bab3256f3fa8a53d2c015 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.460.9 with 0.460.9 from an-a02n01 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/1470, version=0.460.9) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1471) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1472) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-78.raw Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1471, version=0.460.9) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.460.0 of the CIB to disk (digest: a954a061b12f2b671f4357df671dc8d4) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.NfTReK (digest: /var/lib/pacemaker/cib/cib.JjTyOL) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1472, version=0.460.9) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[80466] exited with status 0 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //primitive[@id='apc_snmp_node2_an-pdu03'] to all (origin=local/cibadmin/2) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.460.9 2 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.461.0 (null) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu03'] Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=461, @num_updates=0 Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //primitive[@id='apc_snmp_node2_an-pdu03']: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.461.0) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Device 'apc_snmp_node2_an-pdu03' not found (5 active devices) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-79.raw Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.461.0 of the CIB to disk (digest: 81657bc9d41e6447531fb90c5ab59ee9) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.iruxL3 (digest: /var/lib/pacemaker/cib/cib.GNR4m3) Jul 23 17:02:26 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80468] exited with status 0 Jul 23 17:02:27 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80622] exited with status 0 Jul 23 17:02:27 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80631] exited with status 0 Jul 23 17:02:27 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[80653] exited with status 0 Jul 23 17:02:27 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80655] exited with status 0 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80668] exited with status 0 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80770] exited with status 0 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by deletion of primitive[@id='apc_snmp_node2_an-pdu03']: Configuration change | cib=0.461.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu03'] complete=false Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 1473) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/1473) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1474) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 1475) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.461.0 2 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.461.1 (null) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/1473, version=0.461.1) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.461.1 2 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.461.2 (null) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1474, version=0.461.2) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/1475) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.461.2 2 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.461.3 (null) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/1475, version=0.461.3) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.461.3 2 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.462.0 (null) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-2'] Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=462, @num_updates=0 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.461.3 -> 0.462.0 from an-a02n01 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.462.0) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[80834] exited with status 0 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.462.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-2'] Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 436 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 437 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 438 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 439 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 440 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 441 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 442 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 443 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/436) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/437) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/438) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/439) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.462.0 2 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.462.1 (null) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/436, version=0.462.1) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 436 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.462.1 2 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.462.2 (null) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/437, version=0.462.2) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 437 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.462.2 2 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.462.3 (null) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-80.raw Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/438, version=0.462.3) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 438 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.462.3 2 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.462.4 (null) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/439, version=0.462.4) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 439 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/440) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/441) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/442) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/443) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.462.4 2 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.462.5 (null) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/440, version=0.462.5) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 440 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.462.5 2 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.462.6 (null) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/441, version=0.462.6) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 441 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.462.6 2 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.462.7 (null) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.462.0 of the CIB to disk (digest: 3315d4d4480540b21867a91d2f35d406) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/442, version=0.462.7) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 442 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.462.7 2 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.462.8 (null) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.4CDJ1z (digest: /var/lib/pacemaker/cib/cib.2Y6Mhy) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/443, version=0.462.8) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 443 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //primitive[@id='apc_snmp_node2_an-pdu04'] to all (origin=local/cibadmin/2) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.462.8 2 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.463.0 (null) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu04'] Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=463, @num_updates=0 Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //primitive[@id='apc_snmp_node2_an-pdu04']: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.463.0) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Device 'apc_snmp_node2_an-pdu04' not found (5 active devices) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-81.raw Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.463.0 of the CIB to disk (digest: 2ac144eb25821d8f83af42cc6d6a0e35) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.WUli42 (digest: /var/lib/pacemaker/cib/cib.igluP2) Jul 23 17:02:28 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80836] exited with status 0 Jul 23 17:02:29 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80851] exited with status 0 Jul 23 17:02:29 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80867] exited with status 0 Jul 23 17:02:29 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[80921] exited with status 0 Jul 23 17:02:29 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[80967] exited with status 0 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[81040] exited with status 0 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[81051] exited with status 0 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1476) Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by deletion of lrm[@id='2']: Resource state removal | cib=0.461.1 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2'] complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv14-file-processor_monitor_0 'create' on an-a02n02: Old event | magic=0:7;20:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.461.2 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 20 (srv14-file-processor_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=41 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation delay_node1_stop_0 'create' on an-a02n02: Old event | magic=0:0;20:123:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.461.2 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 123 action 20 (delay_node1_stop_0 on an-a02n02): arrived really late | rc=0 target-rc=0 call-id=142 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation delay_node2_start_0 'create' on an-a02n02: Old event | magic=0:0;16:123:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.461.2 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 123 action 16 (delay_node2_start_0 on an-a02n02): arrived really late | rc=0 target-rc=0 call-id=144 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation delay_node2_monitor_60000 'create' on an-a02n02: Change in recurring result | magic=0:0;17:123:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.461.2 source=process_graph_event:408 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 123 action 17 (delay_node2_monitor_60000 on an-a02n02): arrived after initial scheduling | rc=0 target-rc=0 call-id=145 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation apc_snmp_node1_an-pdu03_monitor_0 'create' on an-a02n02: Old event | magic=0:7;14:195:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.461.2 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 195 action 14 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=251 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv22-machineinterface_monitor_0 'create' on an-a02n02: Old event | magic=0:7;28:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.461.2 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 28 (srv22-machineinterface_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=73 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv13-psql_monitor_0 'create' on an-a02n02: Old event | magic=0:7;19:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.461.2 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 19 (srv13-psql_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=37 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv24-inventory_monitor_0 'create' on an-a02n02: Old event | magic=0:7;30:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.461.2 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 30 (srv24-inventory_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=81 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv15-tickets_monitor_0 'create' on an-a02n02: Old event | magic=0:7;21:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.461.2 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 21 (srv15-tickets_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=45 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv18-inventory_monitor_0 'create' on an-a02n02: Old event | magic=0:7;24:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.461.2 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 24 (srv18-inventory_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=57 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv21-tickets_monitor_0 'create' on an-a02n02: Old event | magic=0:7;27:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.461.2 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 27 (srv21-tickets_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=69 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv17-order-processing_monitor_0 'create' on an-a02n02: Old event | magic=0:7;23:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.461.2 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 23 (srv17-order-processing_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=53 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation ipmilan_node1_start_0 'create' on an-a02n02: Old event | magic=0:0;22:122:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.461.2 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 122 action 22 (ipmilan_node1_start_0 on an-a02n02): arrived really late | rc=0 target-rc=0 call-id=137 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation ipmilan_node1_monitor_60000 'create' on an-a02n02: Change in recurring result | magic=0:0;23:122:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.461.2 source=process_graph_event:408 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 122 action 23 (ipmilan_node1_monitor_60000 on an-a02n02): arrived after initial scheduling | rc=0 target-rc=0 call-id=138 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation ipmilan_node2_stop_0 'create' on an-a02n02: Old event | magic=0:0;16:122:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.461.2 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 122 action 16 (ipmilan_node2_stop_0 on an-a02n02): arrived really late | rc=0 target-rc=0 call-id=136 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv19-uls_monitor_0 'create' on an-a02n02: Old event | magic=0:7;25:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.461.2 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 25 (srv19-uls_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=61 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation apc_snmp_node2_an-pdu03_monitor_0 'create' on an-a02n02: Old event | magic=0:7;15:181:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.461.2 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 181 action 15 (apc_snmp_node2_an-pdu03_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=229 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation apc_snmp_node2_an-pdu04_monitor_0 'create' on an-a02n02: Old event | magic=0:7;16:181:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.461.2 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 181 action 16 (apc_snmp_node2_an-pdu04_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=233 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv20-fileprocessor_monitor_0 'create' on an-a02n02: Old event | magic=0:7;26:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.461.2 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 26 (srv20-fileprocessor_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=65 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv16-machine-interface_monitor_0 'create' on an-a02n02: Old event | magic=0:7;22:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.461.2 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 22 (srv16-machine-interface_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=49 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv23-orderprocessing_monitor_0 'create' on an-a02n02: Old event | magic=0:7;29:101:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.461.2 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 101 action 29 (srv23-orderprocessing_monitor_0 on an-a02n02): arrived really late | rc=7 target-rc=7 call-id=77 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by deletion of lrm[@id='1']: Resource state removal | cib=0.461.3 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1'] complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by deletion of fencing-level[@id='fl-an-a02n02-2']: Configuration change | cib=0.462.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-2'] complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted: Peer Cancelled | source=do_te_invoke:129 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by deletion of primitive[@id='apc_snmp_node2_an-pdu04']: Configuration change | cib=0.463.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu04'] complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.463.0 2 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.463.1 (null) Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1476, version=0.463.1) Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1479) Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1480) Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv14-file-processor_monitor_0 'create' on an-a02n01: Old event | magic=0:7;10:0:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.463.1 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 0 action 10 (srv14-file-processor_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=41 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation delay_node1_start_0 'create' on an-a02n01: Old event | magic=0:0;21:123:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.463.1 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 123 action 21 (delay_node1_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=164 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation delay_node1_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;22:123:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.463.1 source=process_graph_event:408 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 123 action 22 (delay_node1_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=165 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation delay_node2_stop_0 'create' on an-a02n01: Old event | magic=0:0;15:123:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.463.1 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 123 action 15 (delay_node2_stop_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=163 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation apc_snmp_node1_an-pdu03_monitor_0 'create' on an-a02n01: Old event | magic=0:7;13:195:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.463.1 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 195 action 13 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=239 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv13-psql_monitor_0 'create' on an-a02n01: Old event | magic=0:7;9:0:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.463.1 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 0 action 9 (srv13-psql_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=37 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv22-machineinterface_start_0 'create' on an-a02n01: Old event | magic=0:0;22:8:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.463.1 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 8 action 22 (srv22-machineinterface_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=100 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv22-machineinterface_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;24:9:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.463.1 source=process_graph_event:408 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 9 action 24 (srv22-machineinterface_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=101 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv24-inventory_start_0 'create' on an-a02n01: Old event | magic=0:0;28:10:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.463.1 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 10 action 28 (srv24-inventory_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=104 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv24-inventory_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;29:10:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.463.1 source=process_graph_event:408 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 10 action 29 (srv24-inventory_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=105 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv15-tickets_monitor_0 'create' on an-a02n01: Old event | magic=0:7;11:0:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.463.1 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 0 action 11 (srv15-tickets_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=45 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv18-inventory_monitor_0 'create' on an-a02n01: Old event | magic=0:7;14:0:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.463.1 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 0 action 14 (srv18-inventory_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=57 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv21-tickets_start_0 'create' on an-a02n01: Old event | magic=0:0;19:7:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.463.1 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 7 action 19 (srv21-tickets_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=98 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv21-tickets_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;21:8:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.463.1 source=process_graph_event:408 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 8 action 21 (srv21-tickets_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=99 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv17-order-processing_monitor_0 'create' on an-a02n01: Old event | magic=0:7;13:0:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.463.1 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 0 action 13 (srv17-order-processing_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=53 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation ipmilan_node1_stop_0 'create' on an-a02n01: Old event | magic=0:0;21:122:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.463.1 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 122 action 21 (ipmilan_node1_stop_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=159 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation ipmilan_node2_start_0 'create' on an-a02n01: Old event | magic=0:0;17:122:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.463.1 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 122 action 17 (ipmilan_node2_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=160 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation ipmilan_node2_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;18:122:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.463.1 source=process_graph_event:408 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 122 action 18 (ipmilan_node2_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=161 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv19-uls_start_0 'create' on an-a02n01: Old event | magic=0:0;13:5:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.463.1 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 5 action 13 (srv19-uls_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=94 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv19-uls_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;15:6:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.463.1 source=process_graph_event:408 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 6 action 15 (srv19-uls_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=95 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation apc_snmp_node2_an-pdu03_monitor_0 'create' on an-a02n01: Old event | magic=0:7;11:181:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.463.1 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 181 action 11 (apc_snmp_node2_an-pdu03_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=224 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation apc_snmp_node2_an-pdu04_monitor_0 'create' on an-a02n01: Old event | magic=0:7;12:181:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.463.1 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 181 action 12 (apc_snmp_node2_an-pdu04_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=228 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv20-fileprocessor_start_0 'create' on an-a02n01: Old event | magic=0:0;16:6:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.463.1 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 6 action 16 (srv20-fileprocessor_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=96 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv20-fileprocessor_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;18:7:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.463.1 source=process_graph_event:408 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 7 action 18 (srv20-fileprocessor_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=97 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv23-orderprocessing_start_0 'create' on an-a02n01: Old event | magic=0:0;25:9:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.463.1 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 9 action 25 (srv23-orderprocessing_start_0 on an-a02n01): arrived really late | rc=0 target-rc=0 call-id=102 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv23-orderprocessing_monitor_60000 'create' on an-a02n01: Change in recurring result | magic=0:0;27:10:0:789131c0-5164-4a52-9822-30c5026c3827 cib=0.463.1 source=process_graph_event:408 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 10 action 27 (srv23-orderprocessing_monitor_60000 on an-a02n01): arrived after initial scheduling | rc=0 target-rc=0 call-id=103 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted by operation srv16-machine-interface_monitor_0 'create' on an-a02n01: Old event | magic=0:7;12:0:7:789131c0-5164-4a52-9822-30c5026c3827 cib=0.463.1 source=process_graph_event:412 complete=false Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 0 action 12 (srv16-machine-interface_monitor_0 on an-a02n01): arrived really late | rc=7 target-rc=7 call-id=49 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1479, version=0.463.1) Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.463.1 2 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.463.2 (null) Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1480, version=0.463.2) Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1483) Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1484) Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1485) Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1483, version=0.463.2) Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.463.2 2 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.463.3 (null) Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1484, version=0.463.3) Jul 23 17:02:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1485, version=0.463.3) Jul 23 17:02:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 3a5ab7cdb0411ac638cb2f509437dd08 for 0.463.3 (0x561976090670 0) Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.463.3 2 Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.463.4 (null) Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=31:197:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=0:1;31:197:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=253, @rc-code= Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/445, version=0.463.5) Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 445 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.463.5 2 Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.463.6 (null) Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/446, version=0.463.6) Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 446 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627084959 Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/1486, version=0.463.6) Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1487) Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1489) Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1491) Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1493) Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-34 offer to an-a02n02 Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-34 offer to an-a02n01 Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-34 requests from 2 outstanding nodes Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1487, version=0.463.6) Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1489, version=0.463.6) Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1491, version=0.463.6) Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1493, version=0.463.6) Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: 592dca6fc0fb08027384c08728351172 Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.463.6 with 0.463.6 from an-a02n01 Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/1496, version=0.463.6) Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1497) Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1498) Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-82.raw Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1497, version=0.463.6) Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.463.0 of the CIB to disk (digest: d01a983c916104dfb09fd232736ca6e9) Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.FSIes1 (digest: /var/lib/pacemaker/cib/cib.TRzhe3) Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1498, version=0.463.6) Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[81422] exited with status 0 Jul 23 17:02:39 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[81424] exited with status 0 Jul 23 17:02:40 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[81426] exited with status 0 Jul 23 17:02:40 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[81434] exited with status 0 Jul 23 17:02:40 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[81435] exited with status 0 Jul 23 17:02:40 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[81437] exited with status 0 Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[81441] exited with status 0 Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[81446] exited with status 0 Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 1499) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/1499) Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 1501) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1500) Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/1501) Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.463.6 2 Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.463.7 (null) Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/1499, version=0.463.7) Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.463.7 2 Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.463.8 (null) Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1500, version=0.463.8) Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.463.8 2 Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.463.9 (null) Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/1501, version=0.463.9) Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[81592] exited with status 0 Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[81598] exited with status 0 Jul 23 17:02:41 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[81608] exited with status 0 Jul 23 17:02:42 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[81618] exited with status 0 Jul 23 17:02:42 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[81639] exited with status 0 Jul 23 17:02:42 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[81642] exited with status 0 Jul 23 17:02:42 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[81649] exited with status 0 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[81791] exited with status 0 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1502) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.463.9 2 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.463.10 (null) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1502, version=0.463.10) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 197 aborted: History refresh | source=process_resource_updates:268 complete=true Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1505) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1506) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1507) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1505, version=0.463.10) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.463.10 2 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.463.11 (null) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1506, version=0.463.11) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1507, version=0.463.11) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:02:26 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node2_an-pdu03 from an-a02n01 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node2_an-pdu04 from an-a02n01 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node2_an-pdu03 from an-a02n02 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node2_an-pdu04 from an-a02n02 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n01 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n02 -> an-a02n01 ) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 198, saving inputs in /var/lib/pacemaker/pengine/pe-input-218.bz2 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 198 (ref=pe_calc-dc-1627084963-760) derived from /var/lib/pacemaker/pengine/pe-input-218.bz2 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 on an-a02n02 | action 10 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating delete operation apc_snmp_node2_an-pdu03_delete_0 on an-a02n02 | action 14 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating delete operation apc_snmp_node2_an-pdu03_delete_0 locally on an-a02n01 | action 12 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] to all (origin=local/crmd/1510) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.463.11 2 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.463.12 (null) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/1510, version=0.463.11) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (delete_resource) info: Removing resource apc_snmp_node2_an-pdu03 from executor for tengine Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_resource_history) info: Clearing resource history for apc_snmp_node2_an-pdu03 on an-a02n01 (via CIB call 1511) | xpath=//node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (notify_deleted) info: Notifying tengine on localhost that apc_snmp_node2_an-pdu03 was deleted Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 198 action 12 (apc_snmp_node2_an-pdu03_delete_0 on an-a02n01) confirmed: ok | rc=0 call-id=0 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating delete operation apc_snmp_node2_an-pdu04_delete_0 on an-a02n02 | action 15 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating delete operation apc_snmp_node2_an-pdu04_delete_0 locally on an-a02n01 | action 13 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.463.11 2 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.463.12 (null) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=10:198:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;10:198:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code= Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/384, version=0.463.12) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.463.12 2 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.463.13 (null) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/385, version=0.463.12) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.463.12 2 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.463.13 (null) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/386, version=0.463.13) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.463.13 2 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.463.14 (null) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;10:198:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=254, @rc-code=0, @op-status=0 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/387, version=0.463.14) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.463.14 2 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.463.15 (null) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=15 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: OK (rc=0, origin=an-a02n02/crmd/388, version=0.463.14) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.463.14 2 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.463.15 (null) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=15 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: OK (rc=0, origin=an-a02n02/crmd/389, version=0.463.15) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] to all (origin=local/crmd/1511) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] to all (origin=local/crmd/1512) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.463.15 2 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.463.16 (null) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=16 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/1511, version=0.463.16) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.463.16 2 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.463.17 (null) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=17 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: OK (rc=0, origin=an-a02n01/crmd/1512, version=0.463.16) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (delete_resource) info: Removing resource apc_snmp_node2_an-pdu04 from executor for tengine Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_resource_history) info: Clearing resource history for apc_snmp_node2_an-pdu04 on an-a02n01 (via CIB call 1513) | xpath=//node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (notify_deleted) info: Notifying tengine on localhost that apc_snmp_node2_an-pdu04 was deleted Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 198 action 13 (apc_snmp_node2_an-pdu04_delete_0 on an-a02n01) confirmed: ok | rc=0 call-id=0 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 198 aborted by deletion of lrm_resource[@id='apc_snmp_node2_an-pdu03']: Resource state removal | cib=0.463.13 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] complete=false Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 198 action 10 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=254 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 198 aborted by deletion of lrm_resource[@id='apc_snmp_node2_an-pdu04']: Resource state removal | cib=0.463.15 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] complete=false Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 198 aborted by deletion of lrm_resource[@id='apc_snmp_node2_an-pdu03']: Resource state removal | cib=0.463.16 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] complete=false Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] to all (origin=local/crmd/1513) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 198 action 14 (apc_snmp_node2_an-pdu03_delete_0 on an-a02n02) confirmed: ok | rc=0 call-id=0 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 198 action 15 (apc_snmp_node2_an-pdu04_delete_0 on an-a02n02) confirmed: ok | rc=0 call-id=0 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 198 (Complete=5, Pending=0, Fired=0, Skipped=1, Incomplete=2, Source=/var/lib/pacemaker/pengine/pe-input-218.bz2): Stopped Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.463.16 2 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.463.17 (null) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=17 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: OK (rc=0, origin=an-a02n01/crmd/1513, version=0.463.17) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 198 aborted by deletion of lrm_resource[@id='apc_snmp_node2_an-pdu04']: Resource state removal | cib=0.463.17 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] complete=true Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:02:26 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n01 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu03 ( an-a02n01 ) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 199, saving inputs in /var/lib/pacemaker/pengine/pe-input-219.bz2 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 199 (ref=pe_calc-dc-1627084963-768) derived from /var/lib/pacemaker/pengine/pe-input-219.bz2 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 locally on an-a02n01 | action 31 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=31:199:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_start_0 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1516) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu03 action:start call_id:243 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.463.17 2 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.463.18 (null) Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=18 Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=31:199:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;31:199:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 17:02:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1516, version=0.463.18) Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 447 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: 1627084959 -> (unset) | from an-a02n01 Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 448 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/447) Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/448) Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.463.18 2 Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.463.19 (null) Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=19 Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/447, version=0.463.19) Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 447 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 199 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.463.19 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] complete=false Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.463.19 2 Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.463.20 (null) Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=20 Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/448, version=0.463.20) Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 448 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 199 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.463.20 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] complete=false Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.463.20 2 Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.463.21 (null) Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=21 Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/390, version=0.463.20) Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.463.20 2 Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.463.21 (null) Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=21 Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/391, version=0.463.21) Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 199 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu03']: Resource state removal | cib=0.463.21 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] complete=false Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.463.21 2 Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.464.0 (null) Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=464, @num_updates=0 Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']: @value=1627084965 Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n02/crmd/393, version=0.464.0) Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 199 aborted by cib-bootstrap-options-last-lrm-refresh doing modify last-lrm-refresh=1627084965: Configuration change | cib=0.464.0 source=te_update_diff_v2:474 path=/cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh'] complete=false Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-83.raw Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.464.0 of the CIB to disk (digest: 8ec1401193efb3b5917943fdc4ca9f06) Jul 23 17:02:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.8iOkc1 (digest: /var/lib/pacemaker/cib/cib.bdKEY0) Jul 23 17:02:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[81813] error output [ 2021-07-23 17:02:49,530 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:02:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[81813] error output [ ] Jul 23 17:02:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[81813] error output [ ] Jul 23 17:02:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[81813] error output [ 2021-07-23 17:02:49,530 ERROR: Please use '-h' for usage ] Jul 23 17:02:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[81813] error output [ ] Jul 23 17:02:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[81813] stderr: [ 2021-07-23 17:02:49,530 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:02:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[81813] stderr: [ ] Jul 23 17:02:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[81813] stderr: [ ] Jul 23 17:02:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[81813] stderr: [ 2021-07-23 17:02:49,530 ERROR: Please use '-h' for usage ] Jul 23 17:02:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[81813] stderr: [ ] Jul 23 17:02:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (internal_stonith_action_execute) info: Attempt 2 to execute fence_apc_snmp (monitor). remaining timeout is 14 Jul 23 17:02:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: bac22de2c09593682367b25388805e65 for 0.464.0 (0x561975c5c940 0) Jul 23 17:02:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 17:02:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.464.0) Jul 23 17:02:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-84.raw Jul 23 17:02:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.464.0 of the CIB to disk (digest: b66011815367a38117e1415fba9059a4) Jul 23 17:02:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.MqvQww (digest: /var/lib/pacemaker/cib/cib.g7tWeu) Jul 23 17:02:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/cibadmin/2) Jul 23 17:02:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.464.0 2 Jul 23 17:02:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.465.0 (null) Jul 23 17:02:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:02:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=465 Jul 23 17:02:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.465.0) Jul 23 17:02:53 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 199 aborted by deletion of primitive[@id='apc_snmp_node1_an-pdu03']: Configuration change | cib=0.465.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03'] complete=false Jul 23 17:02:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-85.raw Jul 23 17:02:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.465.0 of the CIB to disk (digest: 2af3375b2d1f932fd9ac41f75d5806c5) Jul 23 17:02:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.0UQhQR (digest: /var/lib/pacemaker/cib/cib.WNBLHS) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[82635] error output [ 2021-07-23 17:02:56,688 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[82635] error output [ ] Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[82635] error output [ ] Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[82635] error output [ 2021-07-23 17:02:56,688 ERROR: Please use '-h' for usage ] Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[82635] error output [ ] Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[82635] stderr: [ 2021-07-23 17:02:56,688 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[82635] stderr: [ ] Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[82635] stderr: [ ] Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[82635] stderr: [ 2021-07-23 17:02:56,688 ERROR: Please use '-h' for usage ] Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[82635] stderr: [ ] Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_remaining_timeout) info: Attempted to execute agent fence_apc_snmp (monitor) the maximum number of times (2) allowed Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_operation) notice: Operation 'monitor' [82635] using apc_snmp_node1_an-pdu03 returned -201 (Generic Pacemaker error) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu03 start (call 243) exited with status 1 (execution time 13318ms, queue time 0ms) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for apc_snmp_node1_an-pdu03 on an-a02n01: error | rc=1 call=243 key=apc_snmp_node1_an-pdu03_start_0 confirmed=true cib-update=1518 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1518) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.465.0 2 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.465.1 (null) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:1;31:199:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=243, @rc-code=1, @op-status=0, @exec-time=13318 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/449, version=0.465.2) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 449 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.465.2 2 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.465.3 (null) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/450, version=0.465.3) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 450 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=1627084976 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 199 aborted by status-1-fail-count-apc_snmp_node1_an-pdu03.start_0 doing create fail-count-apc_snmp_node1_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.465.2 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 199 aborted by status-1-last-failure-apc_snmp_node1_an-pdu03.start_0 doing create last-failure-apc_snmp_node1_an-pdu03#start_0=1627084976: Transient attribute change | cib=0.465.3 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n01 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now f2317cad3d54cec5d7d7aa7d0bf35cf8 (reload:3.10.2) 0:1;31:199:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n01 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:02:43 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:02:43 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (process_rsc_state) warning: Detected active orphan apc_snmp_node1_an-pdu03 running on an-a02n01 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Stopping orphan resource apc_snmp_node1_an-pdu03 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu03 ( an-a02n01 ) due to node availability Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 200, saving inputs in /var/lib/pacemaker/pengine/pe-input-220.bz2 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627084976-770 is obsolete Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n01 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now f2317cad3d54cec5d7d7aa7d0bf35cf8 (reload:3.10.2) 0:1;31:199:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n01 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:02:43 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:02:43 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (process_rsc_state) warning: Detected active orphan apc_snmp_node1_an-pdu03 running on an-a02n01 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Stopping orphan resource apc_snmp_node1_an-pdu03 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n01 because it is orphaned | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu03 ( an-a02n01 ) due to node availability Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 201, saving inputs in /var/lib/pacemaker/pengine/pe-input-221.bz2 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 201 (ref=pe_calc-dc-1627084976-771) derived from /var/lib/pacemaker/pengine/pe-input-221.bz2 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (te_crm_command) info: Executing crm-event (2) locally without waiting: clear_failcount on an-a02n01 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 locally on an-a02n01 | action 3 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=3:201:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_stop_0 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu03 action:stop call_id:244 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1522) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Removed 'apc_snmp_node1_an-pdu03' from device list (4 active devices) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu03 stop (call 244) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.465.3 2 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.465.4 (null) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @transition-key=3:201:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;3:201:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1627 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01: ok | rc=0 call=244 key=apc_snmp_node1_an-pdu03_stop_0 confirmed=true cib-update=1523 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1522, version=0.465.4) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1523) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (update_attrd_clear_failures) info: Asking pacemaker-attrd to clear failure of all operations for apc_snmp_node1_an-pdu03 on cluster node an-a02n01 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.465.4 2 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.465.5 (null) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;3:201:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=244, @rc-code=0, @op-status=0 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1523, version=0.465.5) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 201 action 3 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=244 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section /cib/status/node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0'] to all (origin=local/crmd/1524) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 201 (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-221.bz2): Complete Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 451 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/451) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: 1627084976 -> (unset) | from an-a02n01 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 452 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/452) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.465.5 2 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.465.6 (null) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0'] Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section /cib/status/node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0']: OK (rc=0, origin=an-a02n01/crmd/1524, version=0.465.6) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 201 aborted by deletion of lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0']: Resource operation removal | cib=0.465.6 source=process_op_deletion:358 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0'] complete=true Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.465.6 2 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.465.7 (null) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/451, version=0.465.7) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 201 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.465.7 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 451 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.465.7 2 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.465.8 (null) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/452, version=0.465.8) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 201 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.465.8 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 452 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node1_an-pdu03 from an-a02n01 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 202, saving inputs in /var/lib/pacemaker/pengine/pe-input-222.bz2 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 202 (ref=pe_calc-dc-1627084976-774) derived from /var/lib/pacemaker/pengine/pe-input-222.bz2 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating delete operation apc_snmp_node1_an-pdu03_delete_0 locally on an-a02n01 | action 11 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/crmd/1528) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.465.8 2 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.465.9 (null) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/1528, version=0.465.8) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (delete_resource) info: Removing resource apc_snmp_node1_an-pdu03 from executor for tengine Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_resource_history) info: Clearing resource history for apc_snmp_node1_an-pdu03 on an-a02n01 (via CIB call 1529) | xpath=//node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (notify_deleted) info: Notifying tengine on localhost that apc_snmp_node1_an-pdu03 was deleted Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 202 action 11 (apc_snmp_node1_an-pdu03_delete_0 on an-a02n01) confirmed: ok | rc=0 call-id=0 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 202 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-222.bz2): Complete Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/crmd/1529) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.465.8 2 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.465.9 (null) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/1529, version=0.465.9) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 202 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu03']: Resource state removal | cib=0.465.9 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] complete=true Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 203, saving inputs in /var/lib/pacemaker/pengine/pe-input-223.bz2 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 203 (ref=pe_calc-dc-1627084976-777) derived from /var/lib/pacemaker/pengine/pe-input-223.bz2 Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 203 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-223.bz2): Complete Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:02:56 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:03:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: c9b62841ef620b33ecf792c9fd7f7fe3 for 0.465.9 (0x5619762ade80 0) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.465.9 2 Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.466.0 (null) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=466, @num_updates=0 Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources: Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.466.0) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 203 aborted by primitive.apc_snmp_node1_an-pdu03 'create': Configuration change | cib=0.466.0 source=te_update_diff_v2:474 path=/cib/configuration/resources complete=true Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create resources Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.466.0 Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node1_an-pdu03' to device list (5 active devices) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-86.raw Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.466.0 of the CIB to disk (digest: 11c93e5989643c10016c1cb530bd9cf9) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.6xlTzw (digest: /var/lib/pacemaker/cib/cib.yQDkiA) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 204, saving inputs in /var/lib/pacemaker/pengine/pe-input-224.bz2 Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 204 (ref=pe_calc-dc-1627084988-778) derived from /var/lib/pacemaker/pengine/pe-input-224.bz2 Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu03_monitor_0 on an-a02n02 | action 12 Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu03_monitor_0 locally on an-a02n01 | action 11 Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_get_rsc_info) info: Agent information for 'apc_snmp_node1_an-pdu03' not in cache Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_rsc_register) info: Cached agent information for 'apc_snmp_node1_an-pdu03' Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=11:204:7:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_monitor_0 Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1532) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of probe operation for apc_snmp_node1_an-pdu03 on an-a02n01: not running | rc=7 call=249 key=apc_snmp_node1_an-pdu03_monitor_0 confirmed=true cib-update=1533 Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1533) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.466.0 2 Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.466.1 (null) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1532, version=0.466.1) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.466.1 2 Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.466.2 (null) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:7;11:204:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=249, @rc-code=7, @op-status=0 Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1533, version=0.466.2) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 204 action 11 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n01) confirmed: not running | rc=7 call-id=249 Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.466.2 2 Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.466.3 (null) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/395, version=0.466.3) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.466.3 2 Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.466.4 (null) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:7;12:204:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=261, @rc-code=7, @op-status=0 Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/396, version=0.466.4) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 204 action 12 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=261 Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 on an-a02n02 | action 33 Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.466.4 2 Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.466.5 (null) Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @transition-key=33:204:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;33:204:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1 Jul 23 17:03:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/397, version=0.466.5) Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.466.5 2 Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.467.0 (null) Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=467, @num_updates=0 Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources: Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.467.0) Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 204 aborted by primitive.apc_snmp_node1_an-pdu04 'create': Configuration change | cib=0.467.0 source=te_update_diff_v2:474 path=/cib/configuration/resources complete=false Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create resources Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.467.0 Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node1_an-pdu04' to device list (6 active devices) Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-87.raw Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.467.0 of the CIB to disk (digest: 1887720796e188265dec8b73c4601d4f) Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.nhcerC (digest: /var/lib/pacemaker/cib/cib.TbuEID) Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.467.0 2 Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.468.0 (null) Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=468 Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources: Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.468.0) Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 204 aborted by primitive.apc_snmp_node2_an-pdu03 'create': Configuration change | cib=0.468.0 source=te_update_diff_v2:474 path=/cib/configuration/resources complete=false Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create resources Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.468.0 Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node2_an-pdu03' to device list (7 active devices) Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-88.raw Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.468.0 of the CIB to disk (digest: 55342da94c1058a8aa8f3bb11dbf1cb5) Jul 23 17:03:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.GNH7dJ (digest: /var/lib/pacemaker/cib/cib.1ECraI) Jul 23 17:03:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.468.0 2 Jul 23 17:03:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.469.0 (null) Jul 23 17:03:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=469 Jul 23 17:03:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources: Jul 23 17:03:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.469.0) Jul 23 17:03:10 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 204 aborted by primitive.apc_snmp_node2_an-pdu04 'create': Configuration change | cib=0.469.0 source=te_update_diff_v2:474 path=/cib/configuration/resources complete=false Jul 23 17:03:10 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create resources Jul 23 17:03:10 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.469.0 Jul 23 17:03:10 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node2_an-pdu04' to device list (8 active devices) Jul 23 17:03:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-89.raw Jul 23 17:03:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.469.0 of the CIB to disk (digest: 3abd658a0780444e7f081677594821e2) Jul 23 17:03:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.Z8R3YM (digest: /var/lib/pacemaker/cib/cib.wJfZpO) Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.469.0 2 Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.470.0 (null) Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=470 Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.470.0) Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 204 aborted by deletion of fencing-level[@id='fl-an-a02n01-1']: Configuration change | cib=0.470.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] complete=false Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.470.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-90.raw Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.470.0 of the CIB to disk (digest: 1cd1f0ba2b8a262a23606975ea1378a2) Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.JIwZnP (digest: /var/lib/pacemaker/cib/cib.v4wVXO) Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.470.0 2 Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.471.0 (null) Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=471 Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.471.0) Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 204 aborted by fencing-level.fl-an-a02n01-1 'create': Configuration change | cib=0.471.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology complete=false Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.471.0 for /cib/configuration/fencing-topology Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-91.raw Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.471.0 of the CIB to disk (digest: 680c5eff2bb75cfb8dc180b072f02919) Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.7rBWuU (digest: /var/lib/pacemaker/cib/cib.B6gWBX) Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.471.0 2 Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.472.0 (null) Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=472 Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.472.0) Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 204 aborted by fencing-level.fl-an-a02n01-2 'create': Configuration change | cib=0.472.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology complete=false Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.472.0 for /cib/configuration/fencing-topology Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-92.raw Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.472.0 of the CIB to disk (digest: 960254598af38f01003a5a0ac1f34703) Jul 23 17:03:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.oBnQHY (digest: /var/lib/pacemaker/cib/cib.1m5bx2) Jul 23 17:03:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.472.0 2 Jul 23 17:03:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.473.0 (null) Jul 23 17:03:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=473 Jul 23 17:03:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 17:03:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.473.0) Jul 23 17:03:12 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 204 aborted by fencing-level.fl-an-a02n01-3 'create': Configuration change | cib=0.473.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology complete=false Jul 23 17:03:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.473.0 for /cib/configuration/fencing-topology Jul 23 17:03:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 17:03:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 17:03:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 17:03:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 17:03:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:03:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 17:03:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 17:03:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-93.raw Jul 23 17:03:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.473.0 of the CIB to disk (digest: aff885f10d7818dafad50c2a6c8a3b85) Jul 23 17:03:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.Mu7go5 (digest: /var/lib/pacemaker/cib/cib.GLbiw3) Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.473.0 2 Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.474.0 (null) Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-1'] Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=474 Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.474.0) Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 204 aborted by deletion of fencing-level[@id='fl-an-a02n02-1']: Configuration change | cib=0.474.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-1'] complete=false Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.474.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-1'] Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (0 active entries) Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (1 active entry) Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-94.raw Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.474.0 of the CIB to disk (digest: 8d6ec786bb67eb8ff444cbce5c1a9a9c) Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.7VA4oa (digest: /var/lib/pacemaker/cib/cib.iCueEb) Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.474.0 2 Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.475.0 (null) Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-3'] Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=475 Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.475.0) Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 204 aborted by deletion of fencing-level[@id='fl-an-a02n02-3']: Configuration change | cib=0.475.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-3'] complete=false Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.475.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-3'] Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-95.raw Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.475.0 of the CIB to disk (digest: da6a9fa60ff2f3c658641df0802d9528) Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.bcpqVe (digest: /var/lib/pacemaker/cib/cib.xZlxzg) Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.475.0 2 Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.476.0 (null) Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=476 Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.476.0) Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 204 aborted by fencing-level.fl-an-a02n02-1 'create': Configuration change | cib=0.476.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology complete=false Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.476.0 for /cib/configuration/fencing-topology Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (1 active entry) Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-96.raw Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.476.0 of the CIB to disk (digest: a836bd8479c1f0a527773664641ea99c) Jul 23 17:03:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.DiO84h (digest: /var/lib/pacemaker/cib/cib.iskTzi) Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.476.0 2 Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.477.0 (null) Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=477 Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.477.0) Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 204 aborted by fencing-level.fl-an-a02n02-2 'create': Configuration change | cib=0.477.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology complete=false Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.477.0 for /cib/configuration/fencing-topology Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (1 active entry) Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-97.raw Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.477.0 of the CIB to disk (digest: 768db63418abce65ee05494629afe39c) Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.w6Et5t (digest: /var/lib/pacemaker/cib/cib.o3pBQq) Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.477.0 2 Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.0 (null) Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=478 Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.478.0) Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 204 aborted by fencing-level.fl-an-a02n02-3 'create': Configuration change | cib=0.478.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology complete=false Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.478.0 for /cib/configuration/fencing-topology Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (1 active entry) Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-98.raw Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.478.0 of the CIB to disk (digest: 93824249fa475e313d27d37b69938764) Jul 23 17:03:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.dTESK2 (digest: /var/lib/pacemaker/cib/cib.1U2ZJ5) Jul 23 17:03:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 7b4df5f27e24cff8e9fb67f75d04f2c4 for 0.478.0 (0x56197630d6c0 0) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.0 2 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.1 (null) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:1;33:204:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=262, @rc-code=1, @op-status=0, @exec-time=13318 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/453, version=0.478.2) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 453 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.2 2 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.3 (null) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/454, version=0.478.3) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 454 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627085002 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 204 aborted by status-2-fail-count-apc_snmp_node1_an-pdu03.start_0 doing create fail-count-apc_snmp_node1_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.478.2 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 204 aborted by status-2-last-failure-apc_snmp_node1_an-pdu03.start_0 doing create last-failure-apc_snmp_node1_an-pdu03#start_0=1627085002: Transient attribute change | cib=0.478.3 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:03:08 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:03:08 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n02 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu03 on an-a02n02 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu04 on an-a02n02 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu04 ( an-a02n02 ) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node2_an-pdu03 ( an-a02n02 ) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node2_an-pdu04 ( an-a02n02 ) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 205, saving inputs in /var/lib/pacemaker/pengine/pe-input-225.bz2 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627085002-782 is obsolete Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:03:08 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:03:08 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n01 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n02 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu03 on an-a02n02 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu04 on an-a02n02 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n02 -> an-a02n01 ) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu04 ( an-a02n02 ) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node2_an-pdu03 ( an-a02n02 ) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node2_an-pdu04 ( an-a02n02 ) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 206, saving inputs in /var/lib/pacemaker/pengine/pe-input-226.bz2 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 206 (ref=pe_calc-dc-1627085002-783) derived from /var/lib/pacemaker/pengine/pe-input-226.bz2 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 on an-a02n02 | action 11 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu04_monitor_0 on an-a02n02 | action 15 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu04_monitor_0 locally on an-a02n01 | action 12 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_get_rsc_info) info: Agent information for 'apc_snmp_node1_an-pdu04' not in cache Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_rsc_register) info: Cached agent information for 'apc_snmp_node1_an-pdu04' Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for apc_snmp_node1_an-pdu04 on an-a02n01 | transition_key=12:206:7:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu04_monitor_0 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1537) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node2_an-pdu03_monitor_0 on an-a02n02 | action 16 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node2_an-pdu03_monitor_0 locally on an-a02n01 | action 13 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_get_rsc_info) info: Agent information for 'apc_snmp_node2_an-pdu03' not in cache Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_rsc_register) info: Cached agent information for 'apc_snmp_node2_an-pdu03' Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for apc_snmp_node2_an-pdu03 on an-a02n01 | transition_key=13:206:7:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu03_monitor_0 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node2_an-pdu04_monitor_0 on an-a02n02 | action 17 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node2_an-pdu04_monitor_0 locally on an-a02n01 | action 14 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.3 2 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.4 (null) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_get_rsc_info) info: Agent information for 'apc_snmp_node2_an-pdu04' not in cache Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_rsc_register) info: Cached agent information for 'apc_snmp_node2_an-pdu04' Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for apc_snmp_node2_an-pdu04 on an-a02n01 | transition_key=14:206:7:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu04_monitor_0 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1537, version=0.478.4) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1538) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.4 2 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.5 (null) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @transition-key=11:206:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;11:206:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of probe operation for apc_snmp_node1_an-pdu04 on an-a02n01: not running | rc=7 call=253 key=apc_snmp_node1_an-pdu04_monitor_0 confirmed=true cib-update=1540 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/399, version=0.478.5) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of probe operation for apc_snmp_node2_an-pdu03 on an-a02n01: not running | rc=7 call=257 key=apc_snmp_node2_an-pdu03_monitor_0 confirmed=true cib-update=1541 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of probe operation for apc_snmp_node2_an-pdu04 on an-a02n01: not running | rc=7 call=261 key=apc_snmp_node2_an-pdu04_monitor_0 confirmed=true cib-update=1542 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.5 2 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.6 (null) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1538, version=0.478.6) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1539) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1540) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1541) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1542) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.6 2 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.7 (null) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/400, version=0.478.7) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.7 2 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.8 (null) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;11:206:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=263, @rc-code=0, @op-status=0 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/401, version=0.478.8) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 206 action 11 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=263 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 locally on an-a02n01 | action 38 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=38:206:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_start_0 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.8 2 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.9 (null) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu03 action:start call_id:262 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1539, version=0.478.9) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.9 2 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.10 (null) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:7;12:206:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=253, @rc-code=7, @op-status=0 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1540, version=0.478.10) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 206 action 12 (apc_snmp_node1_an-pdu04_monitor_0 on an-a02n01) confirmed: not running | rc=7 call-id=253 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.10 2 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.11 (null) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:7;13:206:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=257, @rc-code=7, @op-status=0 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1541, version=0.478.11) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 206 action 13 (apc_snmp_node2_an-pdu03_monitor_0 on an-a02n01) confirmed: not running | rc=7 call-id=257 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.11 2 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.12 (null) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:7;14:206:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=261, @rc-code=7, @op-status=0 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1542, version=0.478.12) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 206 action 14 (apc_snmp_node2_an-pdu04_monitor_0 on an-a02n01) confirmed: not running | rc=7 call-id=261 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1543) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.12 2 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.13 (null) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:7;15:206:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=267, @rc-code=7, @op-status=0 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/402, version=0.478.13) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 206 action 15 (apc_snmp_node1_an-pdu04_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=267 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu04_start_0 on an-a02n02 | action 40 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.13 2 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.14 (null) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/403, version=0.478.14) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.14 2 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.15 (null) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=15 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:7;16:206:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=271, @rc-code=7, @op-status=0 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/404, version=0.478.15) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 206 action 16 (apc_snmp_node2_an-pdu03_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=271 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node2_an-pdu03_start_0 on an-a02n02 | action 42 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.15 2 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.16 (null) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=16 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/405, version=0.478.16) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.16 2 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.17 (null) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=17 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @transition-key=38:206:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;38:206:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change= Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1543, version=0.478.17) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.17 2 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.18 (null) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=18 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:7;17:206:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=275, @rc-code=7, @op-status=0 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/406, version=0.478.18) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 206 action 17 (apc_snmp_node2_an-pdu04_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=275 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node2_an-pdu04_start_0 on an-a02n02 | action 44 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.18 2 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.19 (null) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=19 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_start_0, @operation=start, @transition-key=40:206:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;40:206:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/407, version=0.478.19) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.19 2 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.20 (null) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=20 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @operation_key=apc_snmp_node2_an-pdu03_start_0, @operation=start, @transition-key=42:206:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;42:206:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/408, version=0.478.20) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.20 2 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.21 (null) Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=21 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @operation_key=apc_snmp_node2_an-pdu04_start_0, @operation=start, @transition-key=44:206:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;44:206:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1 Jul 23 17:03:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/409, version=0.478.21) Jul 23 17:03:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 5b64721ffaa9a1b8fdd781132590af55 for 0.478.21 (0x56197630d6c0 0) Jul 23 17:03:28 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[83164] error output [ 2021-07-23 17:03:28,497 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:03:28 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[83164] error output [ ] Jul 23 17:03:28 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[83164] error output [ ] Jul 23 17:03:28 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[83164] error output [ 2021-07-23 17:03:28,498 ERROR: Please use '-h' for usage ] Jul 23 17:03:28 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[83164] error output [ ] Jul 23 17:03:28 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[83164] stderr: [ 2021-07-23 17:03:28,497 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:03:28 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[83164] stderr: [ ] Jul 23 17:03:28 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[83164] stderr: [ ] Jul 23 17:03:28 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[83164] stderr: [ 2021-07-23 17:03:28,498 ERROR: Please use '-h' for usage ] Jul 23 17:03:28 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[83164] stderr: [ ] Jul 23 17:03:28 an-a02n01.alteeve.com pacemaker-fenced [121629] (internal_stonith_action_execute) info: Attempt 2 to execute fence_apc_snmp (monitor). remaining timeout is 14 Jul 23 17:03:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.21 2 Jul 23 17:03:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.22 (null) Jul 23 17:03:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=22 Jul 23 17:03:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:1;42:206:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=277, @rc-code=1, @op-status=0, @exec-time=12316 Jul 23 17:03:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: Jul 23 17:03:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/455, version=0.478.23) Jul 23 17:03:34 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 206 aborted by status-2-fail-count-apc_snmp_node2_an-pdu03.start_0 doing create fail-count-apc_snmp_node2_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.478.23 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=false Jul 23 17:03:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 455 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:03:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:03:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 17:03:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.23 2 Jul 23 17:03:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.24 (null) Jul 23 17:03:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=24 Jul 23 17:03:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 17:03:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/456, version=0.478.24) Jul 23 17:03:34 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 206 aborted by status-2-last-failure-apc_snmp_node2_an-pdu03.start_0 doing create last-failure-apc_snmp_node2_an-pdu03#start_0=1627085014: Transient attribute change | cib=0.478.24 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=false Jul 23 17:03:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 456 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:03:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:03:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=1627085014 Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[83186] error output [ 2021-07-23 17:03:35,655 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[83186] error output [ ] Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[83186] error output [ ] Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[83186] error output [ 2021-07-23 17:03:35,655 ERROR: Please use '-h' for usage ] Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[83186] error output [ ] Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[83186] stderr: [ 2021-07-23 17:03:35,655 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[83186] stderr: [ ] Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[83186] stderr: [ ] Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[83186] stderr: [ 2021-07-23 17:03:35,655 ERROR: Please use '-h' for usage ] Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[83186] stderr: [ ] Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_remaining_timeout) info: Attempted to execute agent fence_apc_snmp (monitor) the maximum number of times (2) allowed Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_operation) notice: Operation 'monitor' [83186] using apc_snmp_node1_an-pdu03 returned -201 (Generic Pacemaker error) Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu03 start (call 262) exited with status 1 (execution time 13316ms, queue time 0ms) Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for apc_snmp_node1_an-pdu03 on an-a02n01: error | rc=1 call=262 key=apc_snmp_node1_an-pdu03_start_0 confirmed=true cib-update=1544 Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1544) Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.24 2 Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.25 (null) Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=25 Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:1;38:206:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=262, @rc-code=1, @op-status=0, @exec-time=13316 Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/457, version=0.478.26) Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 206 aborted by status-1-fail-count-apc_snmp_node1_an-pdu03.start_0 doing create fail-count-apc_snmp_node1_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.478.26 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=false Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/458) Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 457 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.26 2 Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.27 (null) Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=27 Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/458, version=0.478.27) Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 206 aborted by status-1-last-failure-apc_snmp_node1_an-pdu03.start_0 doing create last-failure-apc_snmp_node1_an-pdu03#start_0=1627085015: Transient attribute change | cib=0.478.27 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=false Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 458 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=1627085015 Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627085002 Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.27 2 Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.28 (null) Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=28 Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:1;44:206:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=278, @rc-code=1, @op-status=0, @exec-time=13317 Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/459, version=0.478.29) Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/460) Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 206 aborted by status-2-fail-count-apc_snmp_node2_an-pdu04.start_0 doing create fail-count-apc_snmp_node2_an-pdu04#start_0=INFINITY: Transient attribute change | cib=0.478.29 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=false Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 459 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.29 2 Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.30 (null) Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=30 Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/460, version=0.478.30) Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 206 aborted by status-2-last-failure-apc_snmp_node2_an-pdu04.start_0 doing create last-failure-apc_snmp_node2_an-pdu04#start_0=1627085015: Transient attribute change | cib=0.478.30 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=false Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 460 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:03:35 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=1627085015 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.30 2 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.31 (null) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=31 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:1;40:206:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=276, @rc-code=1, @op-status=0, @exec-time=14322 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/461, version=0.478.32) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 461 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.32 2 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.33 (null) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=33 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/462, version=0.478.33) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 462 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627085016 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 206 aborted by status-2-fail-count-apc_snmp_node1_an-pdu04.start_0 doing create fail-count-apc_snmp_node1_an-pdu04#start_0=INFINITY: Transient attribute change | cib=0.478.32 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 206 aborted by status-2-last-failure-apc_snmp_node1_an-pdu04.start_0 doing create last-failure-apc_snmp_node1_an-pdu04#start_0=1627085016: Transient attribute change | cib=0.478.33 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:03:08 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_0 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_0 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n02 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu03 on an-a02n01 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu04 on an-a02n01 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu03 ( an-a02n01 ) due to node availability Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu04 ( an-a02n02 ) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node2_an-pdu03 ( an-a02n02 -> an-a02n01 ) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node2_an-pdu04 ( an-a02n02 -> an-a02n01 ) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 207, saving inputs in /var/lib/pacemaker/pengine/pe-input-227.bz2 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627085016-795 is obsolete Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:03:08 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_0 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_0 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n01 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu03 on an-a02n01 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu04 on an-a02n01 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu03 ( an-a02n01 ) due to node availability Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu04 ( an-a02n02 -> an-a02n01 ) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node2_an-pdu03 ( an-a02n02 -> an-a02n01 ) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node2_an-pdu04 ( an-a02n02 -> an-a02n01 ) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 208, saving inputs in /var/lib/pacemaker/pengine/pe-input-228.bz2 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 208 (ref=pe_calc-dc-1627085016-796) derived from /var/lib/pacemaker/pengine/pe-input-228.bz2 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 locally on an-a02n01 | action 9 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=9:208:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_stop_0 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu03 action:stop call_id:263 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1548) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu04_stop_0 on an-a02n02 | action 12 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu03 stop (call 263) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node2_an-pdu03_stop_0 on an-a02n02 | action 13 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node2_an-pdu04_stop_0 on an-a02n02 | action 14 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01: ok | rc=0 call=263 key=apc_snmp_node1_an-pdu03_stop_0 confirmed=true cib-update=1549 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1549) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.33 2 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.34 (null) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=34 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @transition-key=9:208:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;9:208:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1627 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1548, version=0.478.34) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.34 2 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.35 (null) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=35 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;9:208:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=263, @rc-code=0, @op-status=0 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1549, version=0.478.35) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 208 action 9 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=263 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.35 2 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.36 (null) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=36 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_stop_0, @operation=stop, @transition-key=12:208:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;12:208:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/413, version=0.478.36) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.36 2 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.37 (null) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=37 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @operation_key=apc_snmp_node2_an-pdu03_stop_0, @operation=stop, @transition-key=13:208:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;13:208:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/414, version=0.478.37) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.37 2 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.38 (null) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=38 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:0;12:208:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=279, @rc-code=0, @op-status=0 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/415, version=0.478.38) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 208 action 12 (apc_snmp_node1_an-pdu04_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=279 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu04_start_0 locally on an-a02n01 | action 35 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node1_an-pdu04 on an-a02n01 | transition_key=35:208:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu04_start_0 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.38 2 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.39 (null) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=39 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @operation_key=apc_snmp_node2_an-pdu04_stop_0, @operation=stop, @transition-key=14:208:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;14:208:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu04 action:start call_id:264 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/416, version=0.478.39) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1550) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.39 2 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.40 (null) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=40 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:0;13:208:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=280, @rc-code=0, @op-status=0 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/417, version=0.478.40) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 208 action 13 (apc_snmp_node2_an-pdu03_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=280 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node2_an-pdu03_start_0 locally on an-a02n01 | action 37 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node2_an-pdu03 on an-a02n01 | transition_key=37:208:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu03_start_0 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.40 2 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.41 (null) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=41 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:0;14:208:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=281, @rc-code=0, @op-status=0 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node2_an-pdu03 action:start call_id:265 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/418, version=0.478.41) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 208 action 14 (apc_snmp_node2_an-pdu04_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=281 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node2_an-pdu04_start_0 locally on an-a02n01 | action 39 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node2_an-pdu04 on an-a02n01 | transition_key=39:208:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu04_start_0 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.41 2 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.42 (null) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=42 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_start_0, @operation=start, @transition-key=35:208:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;35:208:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change= Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node2_an-pdu04 action:start call_id:266 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1550, version=0.478.42) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1551) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1552) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.42 2 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.43 (null) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=43 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @operation_key=apc_snmp_node2_an-pdu03_start_0, @operation=start, @transition-key=37:208:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;37:208:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change= Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1551, version=0.478.43) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.43 2 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.44 (null) Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=44 Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @operation_key=apc_snmp_node2_an-pdu04_start_0, @operation=start, @transition-key=39:208:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;39:208:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change= Jul 23 17:03:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1552, version=0.478.44) Jul 23 17:03:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: dce281c92a24f626ea1f27ff05b79ca7 for 0.478.44 (0x56197630d6c0 0) Jul 23 17:03:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[83249] error output [ 2021-07-23 17:03:42,861 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 17:03:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[83249] error output [ ] Jul 23 17:03:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[83249] error output [ ] Jul 23 17:03:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[83249] error output [ 2021-07-23 17:03:42,861 ERROR: Please use '-h' for usage ] Jul 23 17:03:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[83249] error output [ ] Jul 23 17:03:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[83249] stderr: [ 2021-07-23 17:03:42,861 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 17:03:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[83249] stderr: [ ] Jul 23 17:03:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[83249] stderr: [ ] Jul 23 17:03:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[83249] stderr: [ 2021-07-23 17:03:42,861 ERROR: Please use '-h' for usage ] Jul 23 17:03:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[83249] stderr: [ ] Jul 23 17:03:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (internal_stonith_action_execute) info: Attempt 2 to execute fence_apc_snmp (monitor). remaining timeout is 14 Jul 23 17:03:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[83250] error output [ 2021-07-23 17:03:42,861 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:03:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[83250] error output [ ] Jul 23 17:03:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[83250] error output [ ] Jul 23 17:03:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[83250] error output [ 2021-07-23 17:03:42,861 ERROR: Please use '-h' for usage ] Jul 23 17:03:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[83250] error output [ ] Jul 23 17:03:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[83250] stderr: [ 2021-07-23 17:03:42,861 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:03:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[83250] stderr: [ ] Jul 23 17:03:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[83250] stderr: [ ] Jul 23 17:03:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[83250] stderr: [ 2021-07-23 17:03:42,861 ERROR: Please use '-h' for usage ] Jul 23 17:03:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[83250] stderr: [ ] Jul 23 17:03:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (internal_stonith_action_execute) info: Attempt 2 to execute fence_apc_snmp (monitor). remaining timeout is 14 Jul 23 17:03:43 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[83251] error output [ 2021-07-23 17:03:42,864 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 17:03:43 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[83251] error output [ ] Jul 23 17:03:43 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[83251] error output [ ] Jul 23 17:03:43 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[83251] error output [ 2021-07-23 17:03:42,864 ERROR: Please use '-h' for usage ] Jul 23 17:03:43 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[83251] error output [ ] Jul 23 17:03:43 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[83251] stderr: [ 2021-07-23 17:03:42,864 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 17:03:43 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[83251] stderr: [ ] Jul 23 17:03:43 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[83251] stderr: [ ] Jul 23 17:03:43 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[83251] stderr: [ 2021-07-23 17:03:42,864 ERROR: Please use '-h' for usage ] Jul 23 17:03:43 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[83251] stderr: [ ] Jul 23 17:03:43 an-a02n01.alteeve.com pacemaker-fenced [121629] (internal_stonith_action_execute) info: Attempt 2 to execute fence_apc_snmp (monitor). remaining timeout is 13 Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[84151] error output [ 2021-07-23 17:03:49,027 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[84151] error output [ ] Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[84151] error output [ ] Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[84151] error output [ 2021-07-23 17:03:49,027 ERROR: Please use '-h' for usage ] Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[84151] error output [ ] Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[84151] stderr: [ 2021-07-23 17:03:49,027 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[84151] stderr: [ ] Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[84151] stderr: [ ] Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[84151] stderr: [ 2021-07-23 17:03:49,027 ERROR: Please use '-h' for usage ] Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[84151] stderr: [ ] Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_remaining_timeout) info: Attempted to execute agent fence_apc_snmp (monitor) the maximum number of times (2) allowed Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_operation) notice: Operation 'monitor' [84151] using apc_snmp_node1_an-pdu04 returned -201 (Generic Pacemaker error) Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu04 start (call 264) exited with status 1 (execution time 12327ms, queue time 0ms) Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for apc_snmp_node1_an-pdu04 on an-a02n01: error | rc=1 call=264 key=apc_snmp_node1_an-pdu04_start_0 confirmed=true cib-update=1553 Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1553) Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.44 2 Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.45 (null) Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=45 Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:1;35:208:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=264, @rc-code=1, @op-status=0, @exec-time=12327 Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/463, version=0.478.46) Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 463 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 208 aborted by status-1-fail-count-apc_snmp_node1_an-pdu04.start_0 doing create fail-count-apc_snmp_node1_an-pdu04#start_0=INFINITY: Transient attribute change | cib=0.478.46 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=false Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.46 2 Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.47 (null) Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=47 Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/464, version=0.478.47) Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 464 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=1627085029 Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627085016 Jul 23 17:03:49 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 208 aborted by status-1-last-failure-apc_snmp_node1_an-pdu04.start_0 doing create last-failure-apc_snmp_node1_an-pdu04#start_0=1627085029: Transient attribute change | cib=0.478.47 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=false Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[84154] error output [ 2021-07-23 17:03:50,026 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[84154] error output [ ] Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[84154] error output [ ] Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[84154] error output [ 2021-07-23 17:03:50,026 ERROR: Please use '-h' for usage ] Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[84154] error output [ ] Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[84154] stderr: [ 2021-07-23 17:03:50,026 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[84154] stderr: [ ] Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[84154] stderr: [ ] Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[84154] stderr: [ 2021-07-23 17:03:50,026 ERROR: Please use '-h' for usage ] Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[84154] stderr: [ ] Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_remaining_timeout) info: Attempted to execute agent fence_apc_snmp (monitor) the maximum number of times (2) allowed Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_operation) notice: Operation 'monitor' [84154] using apc_snmp_node2_an-pdu03 returned -201 (Generic Pacemaker error) Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node2_an-pdu03 start (call 265) exited with status 1 (execution time 13325ms, queue time 0ms) Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for apc_snmp_node2_an-pdu03 on an-a02n01: error | rc=1 call=265 key=apc_snmp_node2_an-pdu03_start_0 confirmed=true cib-update=1554 Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1554) Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.47 2 Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.48 (null) Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=48 Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:1;37:208:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=265, @rc-code=1, @op-status=0, @exec-time=13325 Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/465, version=0.478.49) Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 465 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 208 aborted by status-1-fail-count-apc_snmp_node2_an-pdu03.start_0 doing create fail-count-apc_snmp_node2_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.478.49 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=false Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.49 2 Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.50 (null) Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=50 Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/466, version=0.478.50) Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 466 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=1627085030 Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=1627085014 Jul 23 17:03:50 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 208 aborted by status-1-last-failure-apc_snmp_node2_an-pdu03.start_0 doing create last-failure-apc_snmp_node2_an-pdu03#start_0=1627085030: Transient attribute change | cib=0.478.50 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=false Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[84169] error output [ 2021-07-23 17:03:51,027 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[84169] error output [ ] Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[84169] error output [ ] Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[84169] error output [ 2021-07-23 17:03:51,027 ERROR: Please use '-h' for usage ] Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[84169] error output [ ] Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[84169] stderr: [ 2021-07-23 17:03:51,027 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[84169] stderr: [ ] Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[84169] stderr: [ ] Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[84169] stderr: [ 2021-07-23 17:03:51,027 ERROR: Please use '-h' for usage ] Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[84169] stderr: [ ] Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_remaining_timeout) info: Attempted to execute agent fence_apc_snmp (monitor) the maximum number of times (2) allowed Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_operation) notice: Operation 'monitor' [84169] using apc_snmp_node2_an-pdu04 returned -201 (Generic Pacemaker error) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node2_an-pdu04 start (call 266) exited with status 1 (execution time 14325ms, queue time 0ms) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for apc_snmp_node2_an-pdu04 on an-a02n01: error | rc=1 call=266 key=apc_snmp_node2_an-pdu04_start_0 confirmed=true cib-update=1555 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1555) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.50 2 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.51 (null) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=51 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:1;39:208:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=266, @rc-code=1, @op-status=0, @exec-time=14325 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/467, version=0.478.52) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 467 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.52 2 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.53 (null) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=53 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/468, version=0.478.53) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 468 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=1627085031 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=1627085015 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 208 aborted by status-1-fail-count-apc_snmp_node2_an-pdu04.start_0 doing create fail-count-apc_snmp_node2_an-pdu04#start_0=INFINITY: Transient attribute change | cib=0.478.52 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 208 aborted by status-1-last-failure-apc_snmp_node2_an-pdu04.start_0 doing create last-failure-apc_snmp_node2_an-pdu04#start_0=1627085031: Transient attribute change | cib=0.478.53 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 17:03:36 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 17:03:36 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 17:03:36 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 17:03:36 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 17:03:36 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 17:03:36 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:03:08 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu04 on an-a02n01 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu04 ( an-a02n01 ) due to node availability Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node2_an-pdu03 ( an-a02n01 ) due to node availability Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node2_an-pdu04 ( an-a02n01 ) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 209, saving inputs in /var/lib/pacemaker/pengine/pe-input-229.bz2 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627085031-804 is obsolete Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 17:03:36 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 17:03:36 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 17:03:36 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 17:03:36 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 17:03:36 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 17:03:36 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:03:08 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu04 ( an-a02n01 ) due to node availability Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node2_an-pdu03 ( an-a02n01 ) due to node availability Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node2_an-pdu04 ( an-a02n01 ) due to node availability Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 210, saving inputs in /var/lib/pacemaker/pengine/pe-input-230.bz2 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 210 (ref=pe_calc-dc-1627085031-805) derived from /var/lib/pacemaker/pengine/pe-input-230.bz2 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu04_stop_0 locally on an-a02n01 | action 9 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node1_an-pdu04 on an-a02n01 | transition_key=9:210:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu04_stop_0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu04 action:stop call_id:267 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node2_an-pdu03_stop_0 locally on an-a02n01 | action 10 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node2_an-pdu03 on an-a02n01 | transition_key=10:210:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu03_stop_0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1559) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1560) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu04 stop (call 267) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node2_an-pdu03 action:stop call_id:268 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node2_an-pdu04_stop_0 locally on an-a02n01 | action 11 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node2_an-pdu04 on an-a02n01 | transition_key=11:210:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu04_stop_0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node2_an-pdu03 stop (call 268) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1561) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node2_an-pdu04 action:stop call_id:269 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node2_an-pdu04 stop (call 269) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node1_an-pdu04 on an-a02n01: ok | rc=0 call=267 key=apc_snmp_node1_an-pdu04_stop_0 confirmed=true cib-update=1562 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1562) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node2_an-pdu03 on an-a02n01: ok | rc=0 call=268 key=apc_snmp_node2_an-pdu03_stop_0 confirmed=true cib-update=1563 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node2_an-pdu04 on an-a02n01: ok | rc=0 call=269 key=apc_snmp_node2_an-pdu04_stop_0 confirmed=true cib-update=1564 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.53 2 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.54 (null) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=54 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_stop_0, @operation=stop, @transition-key=9:210:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;9:210:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1627 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1559, version=0.478.54) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.54 2 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.55 (null) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=55 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @operation_key=apc_snmp_node2_an-pdu03_stop_0, @operation=stop, @transition-key=10:210:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;10:210:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1560, version=0.478.55) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.55 2 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.56 (null) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=56 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @operation_key=apc_snmp_node2_an-pdu04_stop_0, @operation=stop, @transition-key=11:210:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;11:210:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1561, version=0.478.56) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.56 2 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.57 (null) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=57 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:0;9:210:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=267, @rc-code=0, @op-status=0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1562, version=0.478.57) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 210 action 9 (apc_snmp_node1_an-pdu04_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=267 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1563) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1564) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.57 2 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.58 (null) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=58 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:0;10:210:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=268, @rc-code=0, @op-status=0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1563, version=0.478.58) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 210 action 10 (apc_snmp_node2_an-pdu03_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=268 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.58 2 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.478.59 (null) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=59 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:0;11:210:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=269, @rc-code=0, @op-status=0 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1564, version=0.478.59) Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 210 action 11 (apc_snmp_node2_an-pdu04_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=269 Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 210 (Complete=3, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-230.bz2): Complete Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:03:51 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:03:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: bfbce504dd2f957c0551babdeb91277a for 0.478.59 (0x56197630d6c0 0) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.478.59 2 Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.479.0 a1d4b388efc3790ba53b74609618930b Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3'] Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-1'] Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-2'] Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-3'] Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=479, @num_updates=0 Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.479.0) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 210 aborted by deletion of fencing-level[@id='fl-an-a02n01-1']: Configuration change | cib=0.479.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] complete=true Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.479.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-99.raw Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.479.0 of the CIB to disk (digest: 6d72beb3717faa2ca4371a9ba8945a62) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.HNYgQB (digest: /var/lib/pacemaker/cib/cib.81zUNB) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 17:03:36 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 17:03:36 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 17:03:36 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:03:08 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 211, saving inputs in /var/lib/pacemaker/pengine/pe-input-231.bz2 Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 211 (ref=pe_calc-dc-1627085111-809) derived from /var/lib/pacemaker/pengine/pe-input-231.bz2 Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 211 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-231.bz2): Complete Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:05:11 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:05:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: a1d4b388efc3790ba53b74609618930b for 0.479.0 (0x561976350be0 0) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] to all (origin=local/crmd/1566) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]: 1627085030 -> (unset) | from an-a02n01 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 469 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]: 1627085014 -> (unset) | from an-a02n01 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node2_an-pdu03#start_0' delayed: update 469 in progress Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 470 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node2_an-pdu03#start_0' delayed: update 470 in progress Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 471 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node1_an-pdu03#start_0' delayed: update 471 in progress Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: 1627085015 -> (unset) | from an-a02n01 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 472 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: 1627085002 -> (unset) | from an-a02n01 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node1_an-pdu03#start_0' delayed: update 472 in progress Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 473 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node2_an-pdu04#start_0' delayed: update 473 in progress Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]: 1627085031 -> (unset) | from an-a02n01 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 474 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]: 1627085015 -> (unset) | from an-a02n01 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node2_an-pdu04#start_0' delayed: update 474 in progress Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 475 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node1_an-pdu04#start_0' delayed: update 475 in progress Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]: 1627085029 -> (unset) | from an-a02n01 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 476 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]: 1627085016 -> (unset) | from an-a02n01 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node1_an-pdu04#start_0' delayed: update 476 in progress Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.479.0 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.479.1 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: OK (rc=0, origin=an-a02n01/crmd/1566, version=0.479.0) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (delete_resource) info: Removing resource apc_snmp_node2_an-pdu04 from executor for fb042966-4496-44c8-8969-37f186ed898c as root Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/469) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/470) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/471) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/472) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/473) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_resource_history) info: Clearing resource history for apc_snmp_node2_an-pdu04 on an-a02n01 (via CIB call 1567) | xpath=//node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] to all (origin=local/crmd/1567) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (notify_deleted) info: Notifying fb042966-4496-44c8-8969-37f186ed898c on an-a02n01 that apc_snmp_node2_an-pdu04 was deleted Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/474) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/475) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/476) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.479.0 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.479.1 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu03.start_0'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/469, version=0.479.1) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 469 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=1627085014 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 477 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.479.1 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.479.2 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu03.start_0'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/470, version=0.479.2) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 470 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 478 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.479.2 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.479.3 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/471, version=0.479.3) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 471 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 479 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.479.3 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.479.4 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/472, version=0.479.4) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 472 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627085002 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 480 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.479.4 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.479.5 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu04.start_0'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/473, version=0.479.5) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 473 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 481 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.479.5 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.479.6 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: OK (rc=0, origin=an-a02n01/crmd/1567, version=0.479.6) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.479.6 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.479.7 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu04.start_0'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/474, version=0.479.7) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 474 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=1627085015 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 482 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.479.7 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.479.8 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu04.start_0'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/475, version=0.479.8) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 475 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 483 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.479.8 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.479.9 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu04.start_0'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/476, version=0.479.9) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 476 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627085016 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 484 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/477) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/478) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/479) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 211 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu03.start_0']: Transient attribute change | cib=0.479.1 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu03.start_0'] complete=true Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/480) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 211 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu03.start_0']: Transient attribute change | cib=0.479.2 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu03.start_0'] complete=true Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/481) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1569) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/482) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 211 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.479.3 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/483) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/484) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 211 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.479.4 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 211 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu04.start_0']: Transient attribute change | cib=0.479.5 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu04.start_0'] complete=true Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 211 aborted by deletion of lrm_resource[@id='apc_snmp_node2_an-pdu04']: Resource state removal | cib=0.479.6 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] complete=true Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 211 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu04.start_0']: Transient attribute change | cib=0.479.7 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu04.start_0'] complete=true Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 211 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.479.8 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu04.start_0'] complete=true Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 211 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.479.9 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu04.start_0'] complete=true Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.479.9 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.479.10 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu03.start_0'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/477, version=0.479.10) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 477 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.479.10 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.479.11 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu03.start_0'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/478, version=0.479.11) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 478 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.479.11 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.479.12 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/479, version=0.479.12) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 479 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.479.12 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.479.13 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/480, version=0.479.13) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 480 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.479.13 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.479.14 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu04.start_0'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/481, version=0.479.14) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 481 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.479.14 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.0 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=480, @num_updates=0 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']: @value=1627085127 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1569, version=0.480.0) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.0 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.1 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu04.start_0'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/482, version=0.480.1) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 482 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.1 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.2 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu04.start_0'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/483, version=0.480.2) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 483 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.2 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.3 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu04.start_0'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/484, version=0.480.3) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 484 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.3 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.4 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: OK (rc=0, origin=an-a02n02/crmd/419, version=0.480.3) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.3 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.4 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: OK (rc=0, origin=an-a02n02/crmd/420, version=0.480.4) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n02/crmd/422, version=0.480.4) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.4 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.5 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/424, version=0.480.4) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.4 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.5 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/425, version=0.480.5) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n02/crmd/427, version=0.480.5) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/crmd/1579) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.5 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.6 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/1579, version=0.480.5) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (delete_resource) info: Removing resource apc_snmp_node1_an-pdu03 from executor for fb042966-4496-44c8-8969-37f186ed898c as root Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-0.raw Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_resource_history) info: Clearing resource history for apc_snmp_node1_an-pdu03 on an-a02n01 (via CIB call 1580) | xpath=//node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (notify_deleted) info: Notifying fb042966-4496-44c8-8969-37f186ed898c on an-a02n01 that apc_snmp_node1_an-pdu03 was deleted Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/crmd/1580) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1582) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] to all (origin=local/crmd/1583) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.480.0 of the CIB to disk (digest: a99a92c6e82f721f2dc986cfb884dfb5) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.5 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.6 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.4HAK3Z (digest: /var/lib/pacemaker/cib/cib.zJbqy0) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/1580, version=0.480.6) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1582, version=0.480.6) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.6 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.7 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n01/crmd/1583, version=0.480.6) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (delete_resource) info: Removing resource apc_snmp_node1_an-pdu04 from executor for fb042966-4496-44c8-8969-37f186ed898c as root Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_resource_history) info: Clearing resource history for apc_snmp_node1_an-pdu04 on an-a02n01 (via CIB call 1584) | xpath=//node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (notify_deleted) info: Notifying fb042966-4496-44c8-8969-37f186ed898c on an-a02n01 that apc_snmp_node1_an-pdu04 was deleted Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] to all (origin=local/crmd/1584) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.6 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.7 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n01/crmd/1584, version=0.480.7) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1586) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] to all (origin=local/crmd/1587) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1586, version=0.480.7) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.7 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.8 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/1587, version=0.480.7) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (delete_resource) info: Removing resource apc_snmp_node2_an-pdu03 from executor for fb042966-4496-44c8-8969-37f186ed898c as root Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_resource_history) info: Clearing resource history for apc_snmp_node2_an-pdu03 on an-a02n01 (via CIB call 1588) | xpath=//node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (notify_deleted) info: Notifying fb042966-4496-44c8-8969-37f186ed898c on an-a02n01 that apc_snmp_node2_an-pdu03 was deleted Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] to all (origin=local/crmd/1588) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 211 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu03.start_0']: Transient attribute change | cib=0.479.10 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu03.start_0'] complete=true Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 211 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu03.start_0']: Transient attribute change | cib=0.479.11 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu03.start_0'] complete=true Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 211 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.479.12 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 211 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.479.13 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 211 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu04.start_0']: Transient attribute change | cib=0.479.14 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu04.start_0'] complete=true Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 211 aborted by cib-bootstrap-options-last-lrm-refresh doing modify last-lrm-refresh=1627085127: Configuration change | cib=0.480.0 source=te_update_diff_v2:474 path=/cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh'] complete=true Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 211 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu04.start_0']: Transient attribute change | cib=0.480.1 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu04.start_0'] complete=true Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 211 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.480.2 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu04.start_0'] complete=true Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 211 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.480.3 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu04.start_0'] complete=true Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.7 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.8 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/1588, version=0.480.8) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1590) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (do_local_notify) warning: Asynchronous reply to client crmd failed: Resource temporarily unavailable | rc=11 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1590, version=0.480.8) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 211 aborted by deletion of lrm_resource[@id='apc_snmp_node2_an-pdu04']: Resource state removal | cib=0.480.4 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] complete=true Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 211 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu03']: Resource state removal | cib=0.480.5 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] complete=true Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 211 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu03']: Resource state removal | cib=0.480.6 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] complete=true Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 211 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu04']: Resource state removal | cib=0.480.7 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] complete=true Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 211 aborted by deletion of lrm_resource[@id='apc_snmp_node2_an-pdu03']: Resource state removal | cib=0.480.8 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] complete=true Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.8 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.9 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n02/crmd/428, version=0.480.8) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.8 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.9 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n02/crmd/429, version=0.480.9) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n02/crmd/431, version=0.480.9) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.9 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.10 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/432, version=0.480.9) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.9 2 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.10 (null) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/433, version=0.480.10) Jul 23 17:05:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n02/crmd/435, version=0.480.10) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 211 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu04']: Resource state removal | cib=0.480.9 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] complete=true Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 211 aborted by deletion of lrm_resource[@id='apc_snmp_node2_an-pdu03']: Resource state removal | cib=0.480.10 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] complete=true Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 17:03:22 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n02 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu03 on an-a02n02 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu04 on an-a02n02 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu04 ( an-a02n02 ) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node2_an-pdu03 ( an-a02n02 ) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node2_an-pdu04 ( an-a02n02 ) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 212, saving inputs in /var/lib/pacemaker/pengine/pe-input-232.bz2 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627085129-814 is obsolete Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n02 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu03 on an-a02n02 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu04 on an-a02n02 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu04 ( an-a02n02 ) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node2_an-pdu03 ( an-a02n02 ) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node2_an-pdu04 ( an-a02n02 ) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 213, saving inputs in /var/lib/pacemaker/pengine/pe-input-233.bz2 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 213 (ref=pe_calc-dc-1627085129-815) derived from /var/lib/pacemaker/pengine/pe-input-233.bz2 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu03_monitor_0 on an-a02n02 | action 15 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu03_monitor_0 locally on an-a02n01 | action 11 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_get_rsc_info) info: Agent information for 'apc_snmp_node1_an-pdu03' not in cache Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_rsc_register) info: Cached agent information for 'apc_snmp_node1_an-pdu03' Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=11:213:7:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_monitor_0 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu04_monitor_0 on an-a02n02 | action 16 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu04_monitor_0 locally on an-a02n01 | action 12 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_get_rsc_info) info: Agent information for 'apc_snmp_node1_an-pdu04' not in cache Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1608) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_rsc_register) info: Cached agent information for 'apc_snmp_node1_an-pdu04' Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for apc_snmp_node1_an-pdu04 on an-a02n01 | transition_key=12:213:7:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu04_monitor_0 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1609) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node2_an-pdu03_monitor_0 on an-a02n02 | action 17 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node2_an-pdu03_monitor_0 locally on an-a02n01 | action 13 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_get_rsc_info) info: Agent information for 'apc_snmp_node2_an-pdu03' not in cache Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_rsc_register) info: Cached agent information for 'apc_snmp_node2_an-pdu03' Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for apc_snmp_node2_an-pdu03 on an-a02n01 | transition_key=13:213:7:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu03_monitor_0 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.10 2 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.11 (null) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node2_an-pdu04_monitor_0 on an-a02n02 | action 18 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node2_an-pdu04_monitor_0 locally on an-a02n01 | action 14 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1608, version=0.480.11) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_get_rsc_info) info: Agent information for 'apc_snmp_node2_an-pdu04' not in cache Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_rsc_register) info: Cached agent information for 'apc_snmp_node2_an-pdu04' Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for apc_snmp_node2_an-pdu04 on an-a02n01 | transition_key=14:213:7:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu04_monitor_0 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.11 2 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.12 (null) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1609, version=0.480.12) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1610) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1611) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of probe operation for apc_snmp_node1_an-pdu03 on an-a02n01: not running | rc=7 call=277 key=apc_snmp_node1_an-pdu03_monitor_0 confirmed=true cib-update=1612 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1612) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of probe operation for apc_snmp_node1_an-pdu04 on an-a02n01: not running | rc=7 call=281 key=apc_snmp_node1_an-pdu04_monitor_0 confirmed=true cib-update=1613 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of probe operation for apc_snmp_node2_an-pdu03 on an-a02n01: not running | rc=7 call=285 key=apc_snmp_node2_an-pdu03_monitor_0 confirmed=true cib-update=1614 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.12 2 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.13 (null) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of probe operation for apc_snmp_node2_an-pdu04 on an-a02n01: not running | rc=7 call=289 key=apc_snmp_node2_an-pdu04_monitor_0 confirmed=true cib-update=1615 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1610, version=0.480.13) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.13 2 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.14 (null) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1611, version=0.480.14) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.14 2 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.15 (null) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=15 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:7;11:213:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=277, @rc-code=7, @op-status=0 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1612, version=0.480.15) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 213 action 11 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n01) confirmed: not running | rc=7 call-id=277 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.15 2 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.16 (null) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=16 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/436, version=0.480.16) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.16 2 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.17 (null) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=17 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:7;15:213:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=289, @rc-code=7, @op-status=0 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/437, version=0.480.17) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 213 action 15 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=289 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 on an-a02n02 | action 39 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.17 2 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.18 (null) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=18 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/438, version=0.480.18) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.18 2 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.19 (null) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=19 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:7;16:213:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=293, @rc-code=7, @op-status=0 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/439, version=0.480.19) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 213 action 16 (apc_snmp_node1_an-pdu04_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=293 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1613) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1614) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1615) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.19 2 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.20 (null) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=20 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:7;12:213:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=281, @rc-code=7, @op-status=0 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1613, version=0.480.20) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 213 action 12 (apc_snmp_node1_an-pdu04_monitor_0 on an-a02n01) confirmed: not running | rc=7 call-id=281 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu04_start_0 on an-a02n02 | action 41 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.20 2 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.21 (null) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=21 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:7;13:213:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=285, @rc-code=7, @op-status=0 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1614, version=0.480.21) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 213 action 13 (apc_snmp_node2_an-pdu03_monitor_0 on an-a02n01) confirmed: not running | rc=7 call-id=285 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.21 2 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.22 (null) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=22 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:7;14:213:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=289, @rc-code=7, @op-status=0 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1615, version=0.480.22) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 213 action 14 (apc_snmp_node2_an-pdu04_monitor_0 on an-a02n01) confirmed: not running | rc=7 call-id=289 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.22 2 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.23 (null) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=23 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/440, version=0.480.23) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.23 2 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.24 (null) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=24 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:7;17:213:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=297, @rc-code=7, @op-status=0 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/441, version=0.480.24) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 213 action 17 (apc_snmp_node2_an-pdu03_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=297 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node2_an-pdu03_start_0 on an-a02n02 | action 43 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.24 2 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.25 (null) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=25 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/442, version=0.480.25) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.25 2 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.26 (null) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=26 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:7;18:213:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=301, @rc-code=7, @op-status=0 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/443, version=0.480.26) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 213 action 18 (apc_snmp_node2_an-pdu04_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=301 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node2_an-pdu04_start_0 on an-a02n02 | action 45 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.26 2 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.27 (null) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=27 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @transition-key=39:213:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;39:213:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/444, version=0.480.27) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.27 2 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.28 (null) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=28 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_start_0, @operation=start, @transition-key=41:213:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;41:213:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/445, version=0.480.28) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.28 2 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.29 (null) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=29 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @operation_key=apc_snmp_node2_an-pdu03_start_0, @operation=start, @transition-key=43:213:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;43:213:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/446, version=0.480.29) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.29 2 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.480.30 (null) Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=30 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @operation_key=apc_snmp_node2_an-pdu04_start_0, @operation=start, @transition-key=45:213:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;45:213:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1 Jul 23 17:05:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/447, version=0.480.30) Jul 23 17:05:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: b65d9cb176b17e7fc815016d8f4ef1a6 for 0.480.30 (0x561976135650 0) Jul 23 17:05:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:05:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.480.30 2 Jul 23 17:05:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.481.0 (null) Jul 23 17:05:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=481, @num_updates=0 Jul 23 17:05:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03']: Jul 23 17:05:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:05:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:05:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.481.0) Jul 23 17:05:39 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 213 aborted by meta_attributes.apc_snmp_node1_an-pdu03-meta_attributes 'create': Configuration change | cib=0.481.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03'] complete=false Jul 23 17:05:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create primitive[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:05:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.481.0 Jul 23 17:05:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu03 has been disabled Jul 23 17:05:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-1.raw Jul 23 17:05:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.481.0 of the CIB to disk (digest: 8a6814dcc9fd46c3db627c49b9772c17) Jul 23 17:05:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.Esojzn (digest: /var/lib/pacemaker/cib/cib.CFgBno) Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.481.0 2 Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.481.1 (null) Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:1;41:213:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=303, @rc-code=1, @op-status=0, @exec-time=12324 Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: (unset) -> INFINITY | from an-a02n01 Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 487 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: (unset) -> 1627085142 | from an-a02n01 Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/485, version=0.481.3) Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 488 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 213 aborted by status-2-fail-count-apc_snmp_node1_an-pdu04.start_0 doing create fail-count-apc_snmp_node1_an-pdu04#start_0=INFINITY: Transient attribute change | cib=0.481.3 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=false Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 485 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.481.3 2 Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.481.4 (null) Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/486, version=0.481.4) Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 213 aborted by status-2-last-failure-apc_snmp_node1_an-pdu04.start_0 doing create last-failure-apc_snmp_node1_an-pdu04#start_0=1627085142: Transient attribute change | cib=0.481.4 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=false Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 486 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627085142 Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/487) Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/488) Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.481.4 2 Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.481.5 (null) Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/487, version=0.481.5) Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 487 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 213 aborted by status-2-fail-count-apc_snmp_node1_an-pdu03.start_0 doing create fail-count-apc_snmp_node1_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.481.5 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=false Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.481.5 2 Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.481.6 (null) Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/488, version=0.481.6) Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 213 aborted by status-2-last-failure-apc_snmp_node1_an-pdu03.start_0 doing create last-failure-apc_snmp_node1_an-pdu03#start_0=1627085142: Transient attribute change | cib=0.481.6 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=false Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 488 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:05:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627085142 Jul 23 17:05:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.481.6 2 Jul 23 17:05:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.481.7 (null) Jul 23 17:05:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:05:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:1;43:213:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=304, @rc-code=1, @op-status=0, @exec-time=13322 Jul 23 17:05:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: Jul 23 17:05:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/489, version=0.481.8) Jul 23 17:05:43 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 213 aborted by status-2-fail-count-apc_snmp_node2_an-pdu03.start_0 doing create fail-count-apc_snmp_node2_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.481.8 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=false Jul 23 17:05:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 489 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:05:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:05:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 17:05:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.481.8 2 Jul 23 17:05:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.481.9 (null) Jul 23 17:05:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 17:05:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 17:05:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/490, version=0.481.9) Jul 23 17:05:43 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 213 aborted by status-2-last-failure-apc_snmp_node2_an-pdu03.start_0 doing create last-failure-apc_snmp_node2_an-pdu03#start_0=1627085143: Transient attribute change | cib=0.481.9 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=false Jul 23 17:05:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 490 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:05:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:05:43 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=1627085143 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.481.9 2 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.481.10 (null) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:1;45:213:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=305, @rc-code=1, @op-status=0, @exec-time=14320 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/491, version=0.481.11) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 491 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.481.11 2 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.481.12 (null) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/492, version=0.481.12) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 492 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=1627085144 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 213 aborted by status-2-fail-count-apc_snmp_node2_an-pdu04.start_0 doing create fail-count-apc_snmp_node2_an-pdu04#start_0=INFINITY: Transient attribute change | cib=0.481.11 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 213 aborted by status-2-last-failure-apc_snmp_node2_an-pdu04.start_0 doing create last-failure-apc_snmp_node2_an-pdu04#start_0=1627085144: Transient attribute change | cib=0.481.12 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_0 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_0 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n01 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu03 on an-a02n01 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu04 on an-a02n02 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu03 ( an-a02n02 ) due to node availability Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu04 ( an-a02n02 -> an-a02n01 ) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node2_an-pdu03 ( an-a02n02 -> an-a02n01 ) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node2_an-pdu04 ( an-a02n02 ) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 214, saving inputs in /var/lib/pacemaker/pengine/pe-input-234.bz2 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627085144-828 is obsolete Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_0 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_0 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n01 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu03 on an-a02n01 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu04 on an-a02n01 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu03 ( an-a02n02 ) due to node availability Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu04 ( an-a02n02 -> an-a02n01 ) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node2_an-pdu03 ( an-a02n02 -> an-a02n01 ) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node2_an-pdu04 ( an-a02n02 -> an-a02n01 ) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 215, saving inputs in /var/lib/pacemaker/pengine/pe-input-235.bz2 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 215 (ref=pe_calc-dc-1627085144-829) derived from /var/lib/pacemaker/pengine/pe-input-235.bz2 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 on an-a02n02 | action 11 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu04_stop_0 on an-a02n02 | action 12 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node2_an-pdu03_stop_0 on an-a02n02 | action 13 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node2_an-pdu04_stop_0 on an-a02n02 | action 14 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.481.12 2 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.481.13 (null) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @transition-key=11:215:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;11:215:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/452, version=0.481.13) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.481.13 2 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.481.14 (null) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_stop_0, @operation=stop, @transition-key=12:215:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;12:215:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/453, version=0.481.14) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.481.14 2 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.481.15 (null) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=15 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;11:215:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=306, @rc-code=0, @op-status=0 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/454, version=0.481.15) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 215 action 11 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=306 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.481.15 2 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.481.16 (null) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=16 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:0;12:215:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=307, @rc-code=0, @op-status=0 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/455, version=0.481.16) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 215 action 12 (apc_snmp_node1_an-pdu04_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=307 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu04_start_0 locally on an-a02n01 | action 35 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node1_an-pdu04 on an-a02n01 | transition_key=35:215:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu04_start_0 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.481.16 2 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.481.17 (null) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=17 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @operation_key=apc_snmp_node2_an-pdu03_stop_0, @operation=stop, @transition-key=13:215:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;13:215:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/456, version=0.481.17) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu04 action:start call_id:290 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1619) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.481.17 2 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.481.18 (null) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=18 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @operation_key=apc_snmp_node2_an-pdu04_stop_0, @operation=stop, @transition-key=14:215:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;14:215:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/457, version=0.481.18) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.481.18 2 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.481.19 (null) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=19 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:0;13:215:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=308, @rc-code=0, @op-status=0 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/458, version=0.481.19) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 215 action 13 (apc_snmp_node2_an-pdu03_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=308 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node2_an-pdu03_start_0 locally on an-a02n01 | action 37 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node2_an-pdu03 on an-a02n01 | transition_key=37:215:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu03_start_0 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.481.19 2 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.481.20 (null) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=20 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:0;14:215:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=309, @rc-code=0, @op-status=0 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node2_an-pdu03 action:start call_id:291 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/459, version=0.481.20) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 215 action 14 (apc_snmp_node2_an-pdu04_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=309 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node2_an-pdu04_start_0 locally on an-a02n01 | action 39 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node2_an-pdu04 on an-a02n01 | transition_key=39:215:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu04_start_0 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.481.20 2 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.481.21 (null) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=21 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_start_0, @operation=start, @transition-key=35:215:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;35:215:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change= Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1619, version=0.481.21) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1620) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1621) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node2_an-pdu04 action:start call_id:292 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.481.21 2 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.481.22 (null) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=22 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @operation_key=apc_snmp_node2_an-pdu03_start_0, @operation=start, @transition-key=37:215:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;37:215:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change= Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1620, version=0.481.22) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.481.22 2 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.481.23 (null) Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=23 Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @operation_key=apc_snmp_node2_an-pdu04_start_0, @operation=start, @transition-key=39:215:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;39:215:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change= Jul 23 17:05:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1621, version=0.481.23) Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 493 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: 1627085142 -> (unset) | from an-a02n01 Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 494 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/493) Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/494) Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.481.23 2 Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.481.24 (null) Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=24 Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/493, version=0.481.24) Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 493 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 215 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.481.24 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] complete=false Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.481.24 2 Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.481.25 (null) Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=25 Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/494, version=0.481.25) Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 494 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 215 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.481.25 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] complete=false Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.481.25 2 Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.481.26 (null) Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=26 Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/460, version=0.481.25) Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.481.25 2 Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.481.26 (null) Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=26 Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/461, version=0.481.26) Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 215 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu03']: Resource state removal | cib=0.481.26 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] complete=false Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.481.26 2 Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.482.0 (null) Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=482, @num_updates=0 Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']: @value=1627085145 Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n02/crmd/463, version=0.482.0) Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 215 aborted by cib-bootstrap-options-last-lrm-refresh doing modify last-lrm-refresh=1627085145: Configuration change | cib=0.482.0 source=te_update_diff_v2:474 path=/cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh'] complete=false Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-2.raw Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.482.0 of the CIB to disk (digest: 0655f2922eea990c8b695008c3b74f3f) Jul 23 17:05:45 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.KCpnil (digest: /var/lib/pacemaker/cib/cib.ZBKiIn) Jul 23 17:05:50 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 4f10eedcb935d71cb5c3dc548fe0c946 for 0.482.0 (0x56197608b520 0) Jul 23 17:05:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[91397] error output [ 2021-07-23 17:05:50,480 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 17:05:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[91397] error output [ ] Jul 23 17:05:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[91397] error output [ ] Jul 23 17:05:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[91397] error output [ 2021-07-23 17:05:50,481 ERROR: Please use '-h' for usage ] Jul 23 17:05:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[91397] error output [ ] Jul 23 17:05:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[91397] stderr: [ 2021-07-23 17:05:50,480 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 17:05:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[91397] stderr: [ ] Jul 23 17:05:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[91397] stderr: [ ] Jul 23 17:05:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[91397] stderr: [ 2021-07-23 17:05:50,481 ERROR: Please use '-h' for usage ] Jul 23 17:05:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[91397] stderr: [ ] Jul 23 17:05:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (internal_stonith_action_execute) info: Attempt 2 to execute fence_apc_snmp (monitor). remaining timeout is 14 Jul 23 17:05:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[91398] error output [ 2021-07-23 17:05:50,480 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:05:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[91398] error output [ ] Jul 23 17:05:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[91398] error output [ ] Jul 23 17:05:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[91398] error output [ 2021-07-23 17:05:50,481 ERROR: Please use '-h' for usage ] Jul 23 17:05:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[91398] error output [ ] Jul 23 17:05:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[91398] stderr: [ 2021-07-23 17:05:50,480 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:05:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[91398] stderr: [ ] Jul 23 17:05:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[91398] stderr: [ ] Jul 23 17:05:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[91398] stderr: [ 2021-07-23 17:05:50,481 ERROR: Please use '-h' for usage ] Jul 23 17:05:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[91398] stderr: [ ] Jul 23 17:05:50 an-a02n01.alteeve.com pacemaker-fenced [121629] (internal_stonith_action_execute) info: Attempt 2 to execute fence_apc_snmp (monitor). remaining timeout is 14 Jul 23 17:05:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[91399] error output [ 2021-07-23 17:05:50,483 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 17:05:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[91399] error output [ ] Jul 23 17:05:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[91399] error output [ ] Jul 23 17:05:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[91399] error output [ 2021-07-23 17:05:50,484 ERROR: Please use '-h' for usage ] Jul 23 17:05:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[91399] error output [ ] Jul 23 17:05:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[91399] stderr: [ 2021-07-23 17:05:50,483 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 17:05:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[91399] stderr: [ ] Jul 23 17:05:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[91399] stderr: [ ] Jul 23 17:05:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[91399] stderr: [ 2021-07-23 17:05:50,484 ERROR: Please use '-h' for usage ] Jul 23 17:05:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[91399] stderr: [ ] Jul 23 17:05:51 an-a02n01.alteeve.com pacemaker-fenced [121629] (internal_stonith_action_execute) info: Attempt 2 to execute fence_apc_snmp (monitor). remaining timeout is 13 Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[92570] error output [ 2021-07-23 17:05:56,643 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[92570] error output [ ] Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[92570] error output [ ] Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[92570] error output [ 2021-07-23 17:05:56,643 ERROR: Please use '-h' for usage ] Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[92570] error output [ ] Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[92570] stderr: [ 2021-07-23 17:05:56,643 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[92570] stderr: [ ] Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[92570] stderr: [ ] Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[92570] stderr: [ 2021-07-23 17:05:56,643 ERROR: Please use '-h' for usage ] Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[92570] stderr: [ ] Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_remaining_timeout) info: Attempted to execute agent fence_apc_snmp (monitor) the maximum number of times (2) allowed Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_operation) notice: Operation 'monitor' [92570] using apc_snmp_node1_an-pdu04 returned -201 (Generic Pacemaker error) Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu04 start (call 290) exited with status 1 (execution time 12327ms, queue time 0ms) Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for apc_snmp_node1_an-pdu04 on an-a02n01: error | rc=1 call=290 key=apc_snmp_node1_an-pdu04_start_0 confirmed=true cib-update=1623 Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1623) Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.482.0 2 Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.482.1 (null) Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:1;35:215:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=290, @rc-code=1, @op-status=0, @exec-time=12327 Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/495, version=0.482.2) Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 495 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 215 aborted by status-1-fail-count-apc_snmp_node1_an-pdu04.start_0 doing create fail-count-apc_snmp_node1_an-pdu04#start_0=INFINITY: Transient attribute change | cib=0.482.2 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=false Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.482.2 2 Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.482.3 (null) Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/496, version=0.482.3) Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 496 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=1627085156 Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627085142 Jul 23 17:05:56 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 215 aborted by status-1-last-failure-apc_snmp_node1_an-pdu04.start_0 doing create last-failure-apc_snmp_node1_an-pdu04#start_0=1627085156: Transient attribute change | cib=0.482.3 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=false Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[92683] error output [ 2021-07-23 17:05:57,641 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[92683] error output [ ] Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[92683] error output [ ] Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[92683] error output [ 2021-07-23 17:05:57,641 ERROR: Please use '-h' for usage ] Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[92683] error output [ ] Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[92683] stderr: [ 2021-07-23 17:05:57,641 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[92683] stderr: [ ] Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[92683] stderr: [ ] Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[92683] stderr: [ 2021-07-23 17:05:57,641 ERROR: Please use '-h' for usage ] Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[92683] stderr: [ ] Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_remaining_timeout) info: Attempted to execute agent fence_apc_snmp (monitor) the maximum number of times (2) allowed Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_operation) notice: Operation 'monitor' [92683] using apc_snmp_node2_an-pdu03 returned -201 (Generic Pacemaker error) Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node2_an-pdu03 start (call 291) exited with status 1 (execution time 13322ms, queue time 0ms) Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for apc_snmp_node2_an-pdu03 on an-a02n01: error | rc=1 call=291 key=apc_snmp_node2_an-pdu03_start_0 confirmed=true cib-update=1624 Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1624) Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.482.3 2 Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.482.4 (null) Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:1;37:215:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=291, @rc-code=1, @op-status=0, @exec-time=13322 Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/497, version=0.482.5) Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 215 aborted by status-1-fail-count-apc_snmp_node2_an-pdu03.start_0 doing create fail-count-apc_snmp_node2_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.482.5 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=false Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 497 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.482.5 2 Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.482.6 (null) Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/498, version=0.482.6) Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 215 aborted by status-1-last-failure-apc_snmp_node2_an-pdu03.start_0 doing create last-failure-apc_snmp_node2_an-pdu03#start_0=1627085157: Transient attribute change | cib=0.482.6 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=false Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 498 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=1627085157 Jul 23 17:05:57 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=1627085143 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[92806] error output [ 2021-07-23 17:05:58,642 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[92806] error output [ ] Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[92806] error output [ ] Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[92806] error output [ 2021-07-23 17:05:58,642 ERROR: Please use '-h' for usage ] Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[92806] error output [ ] Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[92806] stderr: [ 2021-07-23 17:05:58,642 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[92806] stderr: [ ] Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[92806] stderr: [ ] Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[92806] stderr: [ 2021-07-23 17:05:58,642 ERROR: Please use '-h' for usage ] Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[92806] stderr: [ ] Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_remaining_timeout) info: Attempted to execute agent fence_apc_snmp (monitor) the maximum number of times (2) allowed Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_operation) notice: Operation 'monitor' [92806] using apc_snmp_node2_an-pdu04 returned -201 (Generic Pacemaker error) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node2_an-pdu04 start (call 292) exited with status 1 (execution time 14322ms, queue time 0ms) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for apc_snmp_node2_an-pdu04 on an-a02n01: error | rc=1 call=292 key=apc_snmp_node2_an-pdu04_start_0 confirmed=true cib-update=1625 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1625) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.482.6 2 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.482.7 (null) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:1;39:215:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=292, @rc-code=1, @op-status=0, @exec-time=14322 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/499, version=0.482.8) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 499 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.482.8 2 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.482.9 (null) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/500, version=0.482.9) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 500 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=1627085158 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=1627085144 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 215 aborted by status-1-fail-count-apc_snmp_node2_an-pdu04.start_0 doing create fail-count-apc_snmp_node2_an-pdu04#start_0=INFINITY: Transient attribute change | cib=0.482.8 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 215 aborted by status-1-last-failure-apc_snmp_node2_an-pdu04.start_0 doing create last-failure-apc_snmp_node2_an-pdu04#start_0=1627085158: Transient attribute change | cib=0.482.9 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 17:05:44 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 17:05:44 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 17:05:44 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 17:05:44 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_0 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 17:05:44 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 17:05:44 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_0 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu04 on an-a02n01 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu04 ( an-a02n01 ) due to node availability Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node2_an-pdu03 ( an-a02n01 ) due to node availability Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node2_an-pdu04 ( an-a02n01 ) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 216, saving inputs in /var/lib/pacemaker/pengine/pe-input-236.bz2 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627085158-837 is obsolete Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 17:05:44 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 17:05:44 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 17:05:44 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 17:05:44 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_0 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 17:05:44 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 17:05:44 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_0 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu04 ( an-a02n01 ) due to node availability Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node2_an-pdu03 ( an-a02n01 ) due to node availability Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node2_an-pdu04 ( an-a02n01 ) due to node availability Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 217, saving inputs in /var/lib/pacemaker/pengine/pe-input-237.bz2 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 217 (ref=pe_calc-dc-1627085158-838) derived from /var/lib/pacemaker/pengine/pe-input-237.bz2 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu03_monitor_0 on an-a02n02 | action 14 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu04_stop_0 locally on an-a02n01 | action 9 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node1_an-pdu04 on an-a02n01 | transition_key=9:217:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu04_stop_0 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu04 action:stop call_id:293 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node2_an-pdu03_stop_0 locally on an-a02n01 | action 10 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node2_an-pdu03 on an-a02n01 | transition_key=10:217:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu03_stop_0 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1629) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1630) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu04 stop (call 293) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node2_an-pdu03 action:stop call_id:294 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node2_an-pdu04_stop_0 locally on an-a02n01 | action 11 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node2_an-pdu04 on an-a02n01 | transition_key=11:217:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu04_stop_0 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node2_an-pdu03 stop (call 294) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1631) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node2_an-pdu04 action:stop call_id:295 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node2_an-pdu04 stop (call 295) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node1_an-pdu04 on an-a02n01: ok | rc=0 call=293 key=apc_snmp_node1_an-pdu04_stop_0 confirmed=true cib-update=1632 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node2_an-pdu03 on an-a02n01: ok | rc=0 call=294 key=apc_snmp_node2_an-pdu03_stop_0 confirmed=true cib-update=1633 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.482.9 2 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.482.10 (null) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_stop_0, @operation=stop, @transition-key=9:217:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;9:217:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1627 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node2_an-pdu04 on an-a02n01: ok | rc=0 call=295 key=apc_snmp_node2_an-pdu04_stop_0 confirmed=true cib-update=1634 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1629, version=0.482.10) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.482.10 2 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.482.11 (null) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @operation_key=apc_snmp_node2_an-pdu03_stop_0, @operation=stop, @transition-key=10:217:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;10:217:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1630, version=0.482.11) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.482.11 2 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.482.12 (null) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @operation_key=apc_snmp_node2_an-pdu04_stop_0, @operation=stop, @transition-key=11:217:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;11:217:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1631, version=0.482.12) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1632) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1633) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1634) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.482.12 2 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.482.13 (null) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:0;9:217:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=293, @rc-code=0, @op-status=0 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1632, version=0.482.13) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 217 action 9 (apc_snmp_node1_an-pdu04_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=293 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.482.13 2 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.482.14 (null) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:0;10:217:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=294, @rc-code=0, @op-status=0 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1633, version=0.482.14) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 217 action 10 (apc_snmp_node2_an-pdu03_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=294 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.482.14 2 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.482.15 (null) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=15 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/465, version=0.482.15) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.482.15 2 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.482.16 (null) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=16 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:7;14:217:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=314, @rc-code=7, @op-status=0 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/466, version=0.482.16) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 217 action 14 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=314 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.482.16 2 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.482.17 (null) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=17 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:0;11:217:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=295, @rc-code=0, @op-status=0 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1634, version=0.482.17) Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 217 action 11 (apc_snmp_node2_an-pdu04_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=295 Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 217 (Complete=4, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-237.bz2): Complete Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:05:58 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:06:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 6f01b2697ba96ce063279ec607a6983a for 0.482.17 (0x56197608b520 0) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] to all (origin=local/crmd/1635) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 501 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]: 1627085142 -> (unset) | from an-a02n01 Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 502 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node1_an-pdu04#start_0' delayed: update 501 in progress Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]: 1627085156 -> (unset) | from an-a02n01 Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node1_an-pdu04#start_0' delayed: update 502 in progress Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.482.17 2 Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.482.18 (null) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=18 Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n01/crmd/1635, version=0.482.17) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-controld [121633] (delete_resource) info: Removing resource apc_snmp_node1_an-pdu04 from executor for ed043306-1e6e-4553-8509-f71e147ebc23 as root Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_resource_history) info: Clearing resource history for apc_snmp_node1_an-pdu04 on an-a02n01 (via CIB call 1636) | xpath=//node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-controld [121633] (notify_deleted) info: Notifying ed043306-1e6e-4553-8509-f71e147ebc23 on an-a02n01 that apc_snmp_node1_an-pdu04 was deleted Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/501) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/502) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] to all (origin=local/crmd/1636) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1638) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.482.17 2 Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.482.18 (null) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu04.start_0'] Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=18 Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/501, version=0.482.18) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 217 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.482.18 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu04.start_0'] complete=true Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 501 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 503 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.482.18 2 Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.482.19 (null) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu04.start_0'] Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=19 Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/502, version=0.482.19) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 217 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.482.19 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu04.start_0'] complete=true Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 502 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=1627085156 Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 504 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.482.19 2 Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.482.20 (null) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=20 Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n01/crmd/1636, version=0.482.20) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 217 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu04']: Resource state removal | cib=0.482.20 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] complete=true Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.482.20 2 Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.482.21 (null) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=21 Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n02/crmd/467, version=0.482.20) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.482.20 2 Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.483.0 (null) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=483, @num_updates=0 Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']: @value=1627085164 Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1638, version=0.483.0) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 217 aborted by cib-bootstrap-options-last-lrm-refresh doing modify last-lrm-refresh=1627085164: Configuration change | cib=0.483.0 source=te_update_diff_v2:474 path=/cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh'] complete=true Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/503) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/504) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.483.0 2 Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.483.1 (null) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n02/crmd/468, version=0.483.1) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.483.1 2 Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.483.2 (null) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu04.start_0'] Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/503, version=0.483.2) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 503 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.483.2 2 Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.483.3 (null) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu04.start_0'] Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/504, version=0.483.3) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 504 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n02/crmd/470, version=0.483.3) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-3.raw Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.483.0 of the CIB to disk (digest: c10212a544cb4c68eae49f3ac16e9ef2) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.T5pNQV (digest: /var/lib/pacemaker/cib/cib.BPmwvT) Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 217 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu04']: Resource state removal | cib=0.483.1 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] complete=true Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 217 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.483.2 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu04.start_0'] complete=true Jul 23 17:06:04 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 217 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.483.3 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu04.start_0'] complete=true Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 17:05:44 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 17:05:44 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n02 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu04 ( an-a02n02 ) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 218, saving inputs in /var/lib/pacemaker/pengine/pe-input-238.bz2 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 218 (ref=pe_calc-dc-1627085165-844) derived from /var/lib/pacemaker/pengine/pe-input-238.bz2 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu04_monitor_0 on an-a02n02 | action 12 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu04_monitor_0 locally on an-a02n01 | action 11 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_get_rsc_info) info: Agent information for 'apc_snmp_node1_an-pdu04' not in cache Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_rsc_register) info: Cached agent information for 'apc_snmp_node1_an-pdu04' Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for apc_snmp_node1_an-pdu04 on an-a02n01 | transition_key=11:218:7:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu04_monitor_0 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of probe operation for apc_snmp_node1_an-pdu04 on an-a02n01: not running | rc=7 call=300 key=apc_snmp_node1_an-pdu04_monitor_0 confirmed=true cib-update=1648 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1647) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1648) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.483.3 2 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.483.4 (null) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1647, version=0.483.4) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.483.4 2 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.483.5 (null) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:7;11:218:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=300, @rc-code=7, @op-status=0 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1648, version=0.483.5) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 218 action 11 (apc_snmp_node1_an-pdu04_monitor_0 on an-a02n01) confirmed: not running | rc=7 call-id=300 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.483.5 2 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.483.6 (null) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/472, version=0.483.6) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.483.6 2 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.483.7 (null) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:7;12:218:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=319, @rc-code=7, @op-status=0 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/473, version=0.483.7) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 218 action 12 (apc_snmp_node1_an-pdu04_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=319 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu04_start_0 on an-a02n02 | action 33 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.483.7 2 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.483.8 (null) Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_start_0, @operation=start, @transition-key=33:218:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;33:218:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1 Jul 23 17:06:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/474, version=0.483.8) Jul 23 17:06:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:06:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.483.8 2 Jul 23 17:06:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.484.0 (null) Jul 23 17:06:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=484, @num_updates=0 Jul 23 17:06:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu04']: Jul 23 17:06:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:06:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:06:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.484.0) Jul 23 17:06:09 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 218 aborted by meta_attributes.apc_snmp_node1_an-pdu04-meta_attributes 'create': Configuration change | cib=0.484.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu04'] complete=false Jul 23 17:06:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create primitive[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:06:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.484.0 Jul 23 17:06:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu03 has been disabled Jul 23 17:06:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu04 has been disabled Jul 23 17:06:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-4.raw Jul 23 17:06:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.484.0 of the CIB to disk (digest: 7ce5ab71b0469ea8b302ee1a3f2297d4) Jul 23 17:06:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.VRlMB2 (digest: /var/lib/pacemaker/cib/cib.r9gfg2) Jul 23 17:06:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: c4b0ac9d28c3ddd8e01b86bd3210ef52 for 0.484.0 (0x561975c683c0 0) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.484.0 2 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.484.1 (null) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:1;33:218:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=320, @rc-code=1, @op-status=0, @exec-time=13324 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/505, version=0.484.2) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 505 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.484.2 2 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.484.3 (null) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/506, version=0.484.3) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 506 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627085178 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 218 aborted by status-2-fail-count-apc_snmp_node1_an-pdu04.start_0 doing create fail-count-apc_snmp_node1_an-pdu04#start_0=INFINITY: Transient attribute change | cib=0.484.2 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 218 aborted by status-2-last-failure-apc_snmp_node1_an-pdu04.start_0 doing create last-failure-apc_snmp_node1_an-pdu04#start_0=1627085178: Transient attribute change | cib=0.484.3 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 17:05:44 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 17:05:44 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:06:05 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:06:05 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu04 ( an-a02n02 ) due to node availability Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 219, saving inputs in /var/lib/pacemaker/pengine/pe-input-239.bz2 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627085178-848 is obsolete Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 17:05:44 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 17:05:44 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:06:05 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:06:05 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu04 ( an-a02n02 ) due to node availability Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 220, saving inputs in /var/lib/pacemaker/pengine/pe-input-240.bz2 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 220 (ref=pe_calc-dc-1627085178-849) derived from /var/lib/pacemaker/pengine/pe-input-240.bz2 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu04_stop_0 on an-a02n02 | action 11 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.484.3 2 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.484.4 (null) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_stop_0, @operation=stop, @transition-key=11:220:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;11:220:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/476, version=0.484.4) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.484.4 2 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.484.5 (null) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:0;11:220:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=321, @rc-code=0, @op-status=0 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/477, version=0.484.5) Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 220 action 11 (apc_snmp_node1_an-pdu04_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=321 Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 220 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-240.bz2): Complete Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:06:18 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:06:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: b2f337d55839a2906fa9ba17dbc527a2 for 0.484.5 (0x561975c683c0 0) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 507 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/507) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]: 1627085178 -> (unset) | from an-a02n01 Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 508 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/508) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.484.5 2 Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.484.6 (null) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu04.start_0'] Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/507, version=0.484.6) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 507 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 220 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.484.6 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu04.start_0'] complete=true Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.484.6 2 Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.484.7 (null) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu04.start_0'] Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/508, version=0.484.7) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 220 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.484.7 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu04.start_0'] complete=true Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 508 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.484.7 2 Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.484.8 465cbaa65c8525ec017860576992e775 Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n02/crmd/478, version=0.484.7) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.484.7 2 Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.484.8 (null) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n02/crmd/479, version=0.484.8) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.484.8 2 Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.485.0 (null) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=485, @num_updates=0 Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']: @value=1627085185 Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n02/crmd/481, version=0.485.0) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-5.raw Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.485.0 of the CIB to disk (digest: ad046f30ce5c47f75b5fb854c55af988) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.OInaS9 (digest: /var/lib/pacemaker/cib/cib.B9nqm9) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 220 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu04']: Resource state removal | cib=0.484.8 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] complete=true Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 220 aborted by cib-bootstrap-options-last-lrm-refresh doing modify last-lrm-refresh=1627085185: Configuration change | cib=0.485.0 source=te_update_diff_v2:474 path=/cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh'] complete=true Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 17:05:44 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 17:05:44 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:06:05 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:06:25 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 221, saving inputs in /var/lib/pacemaker/pengine/pe-input-241.bz2 Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627085185-851 is obsolete Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 17:05:44 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 17:05:44 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 222, saving inputs in /var/lib/pacemaker/pengine/pe-input-242.bz2 Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 222 (ref=pe_calc-dc-1627085186-852) derived from /var/lib/pacemaker/pengine/pe-input-242.bz2 Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu04_monitor_0 on an-a02n02 | action 11 Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.485.0 2 Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.485.1 (null) Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/483, version=0.485.1) Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.485.1 2 Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.485.2 (null) Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:7;11:222:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=326, @rc-code=7, @op-status=0 Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/484, version=0.485.2) Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 222 action 11 (apc_snmp_node1_an-pdu04_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=326 Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 222 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-242.bz2): Complete Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:06:26 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:06:31 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 2c7f58bfbe832a1d3f6f3b1d3344f8db for 0.485.2 (0x5619762ab8c0 0) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] to all (origin=local/crmd/1658) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]: 1627085143 -> (unset) | from an-a02n01 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 509 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 510 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]: 1627085157 -> (unset) | from an-a02n01 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node2_an-pdu03#start_0' delayed: update 509 in progress Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node2_an-pdu03#start_0' delayed: update 510 in progress Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.485.2 2 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.485.3 (null) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/1658, version=0.485.2) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-controld [121633] (delete_resource) info: Removing resource apc_snmp_node2_an-pdu03 from executor for 2d157856-173c-4935-b14a-a940ee40711f as root Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_resource_history) info: Clearing resource history for apc_snmp_node2_an-pdu03 on an-a02n01 (via CIB call 1659) | xpath=//node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-controld [121633] (notify_deleted) info: Notifying 2d157856-173c-4935-b14a-a940ee40711f on an-a02n01 that apc_snmp_node2_an-pdu03 was deleted Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.485.2 2 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.485.3 (null) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/485, version=0.485.2) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] to all (origin=local/crmd/1659) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/509) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/510) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.485.2 2 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.485.3 (null) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/1659, version=0.485.3) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 222 aborted by deletion of lrm_resource[@id='apc_snmp_node2_an-pdu03']: Resource state removal | cib=0.485.3 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] complete=true Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.485.3 2 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.485.4 (null) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu03.start_0'] Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/509, version=0.485.4) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 222 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu03.start_0']: Transient attribute change | cib=0.485.4 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu03.start_0'] complete=true Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 509 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=1627085157 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 511 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.485.4 2 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.485.5 (null) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu03.start_0'] Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/510, version=0.485.5) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 222 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu03.start_0']: Transient attribute change | cib=0.485.5 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu03.start_0'] complete=true Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1661) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 510 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 512 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/511) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/512) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.485.5 2 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.486.0 (null) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=486, @num_updates=0 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']: @value=1627085206 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1661, version=0.486.0) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.486.0 2 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.486.1 (null) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu03.start_0'] Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/511, version=0.486.1) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 511 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.486.1 2 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.486.2 (null) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu03.start_0'] Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/512, version=0.486.2) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 512 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.486.2 2 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.486.3 (null) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-6.raw Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/486, version=0.486.3) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.486.0 of the CIB to disk (digest: 3df05f999519a78d179fcc801edcfe3c) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.ljsePm (digest: /var/lib/pacemaker/cib/cib.I1hhVk) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n02/crmd/488, version=0.486.3) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 222 aborted by cib-bootstrap-options-last-lrm-refresh doing modify last-lrm-refresh=1627085206: Configuration change | cib=0.486.0 source=te_update_diff_v2:474 path=/cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh'] complete=true Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 222 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu03.start_0']: Transient attribute change | cib=0.486.1 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu03.start_0'] complete=true Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 222 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu03.start_0']: Transient attribute change | cib=0.486.2 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu03.start_0'] complete=true Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 222 aborted by deletion of lrm_resource[@id='apc_snmp_node2_an-pdu03']: Resource state removal | cib=0.486.3 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] complete=true Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 17:05:44 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu03 on an-a02n02 Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node2_an-pdu03 ( an-a02n02 ) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:06:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 223, saving inputs in /var/lib/pacemaker/pengine/pe-input-243.bz2 Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627085206-855 is obsolete Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 17:05:44 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu03 on an-a02n02 Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node2_an-pdu03 ( an-a02n02 ) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 224, saving inputs in /var/lib/pacemaker/pengine/pe-input-244.bz2 Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 224 (ref=pe_calc-dc-1627085207-856) derived from /var/lib/pacemaker/pengine/pe-input-244.bz2 Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node2_an-pdu03_monitor_0 on an-a02n02 | action 12 Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node2_an-pdu03_monitor_0 locally on an-a02n01 | action 11 Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_get_rsc_info) info: Agent information for 'apc_snmp_node2_an-pdu03' not in cache Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_rsc_register) info: Cached agent information for 'apc_snmp_node2_an-pdu03' Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for apc_snmp_node2_an-pdu03 on an-a02n01 | transition_key=11:224:7:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu03_monitor_0 Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of probe operation for apc_snmp_node2_an-pdu03 on an-a02n01: not running | rc=7 call=305 key=apc_snmp_node2_an-pdu03_monitor_0 confirmed=true cib-update=1672 Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1671) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1672) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.486.3 2 Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.486.4 (null) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1671, version=0.486.4) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.486.4 2 Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.486.5 (null) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:7;11:224:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=305, @rc-code=7, @op-status=0 Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1672, version=0.486.5) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 224 action 11 (apc_snmp_node2_an-pdu03_monitor_0 on an-a02n01) confirmed: not running | rc=7 call-id=305 Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.486.5 2 Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.486.6 (null) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/490, version=0.486.6) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.486.6 2 Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.486.7 (null) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:7;12:224:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=331, @rc-code=7, @op-status=0 Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/491, version=0.486.7) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 224 action 12 (apc_snmp_node2_an-pdu03_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=331 Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node2_an-pdu03_start_0 on an-a02n02 | action 33 Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.486.7 2 Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.486.8 (null) Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @operation_key=apc_snmp_node2_an-pdu03_start_0, @operation=start, @transition-key=33:224:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;33:224:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1 Jul 23 17:06:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/492, version=0.486.8) Jul 23 17:06:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 0a70e33ea020df5c32d0f62329202a17 for 0.486.8 (0x5619763f5c00 0) Jul 23 17:06:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:06:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.486.8 2 Jul 23 17:06:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.487.0 (null) Jul 23 17:06:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=487, @num_updates=0 Jul 23 17:06:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu03']: Jul 23 17:06:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:06:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:06:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.487.0) Jul 23 17:06:53 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 224 aborted by meta_attributes.apc_snmp_node2_an-pdu03-meta_attributes 'create': Configuration change | cib=0.487.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu03'] complete=false Jul 23 17:06:53 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create primitive[@id='apc_snmp_node2_an-pdu03'] Jul 23 17:06:53 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.487.0 Jul 23 17:06:53 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu03 has been disabled Jul 23 17:06:53 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu04 has been disabled Jul 23 17:06:53 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 17:06:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-7.raw Jul 23 17:06:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.487.0 of the CIB to disk (digest: 6f0754f942fba115f53032f5739ffe00) Jul 23 17:06:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.ubioz5 (digest: /var/lib/pacemaker/cib/cib.IOsFI3) Jul 23 17:06:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 85c7a194d47e6792862bf547c9478bef for 0.487.0 (0x5619762ed290 0) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.487.0 2 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.487.1 (null) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:1;33:224:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=332, @rc-code=1, @op-status=0, @exec-time=13319 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/513, version=0.487.2) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 513 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.487.2 2 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.487.3 (null) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/514, version=0.487.3) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 514 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=1627085220 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 224 aborted by status-2-fail-count-apc_snmp_node2_an-pdu03.start_0 doing create fail-count-apc_snmp_node2_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.487.2 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 224 aborted by status-2-last-failure-apc_snmp_node2_an-pdu03.start_0 doing create last-failure-apc_snmp_node2_an-pdu03#start_0=1627085220: Transient attribute change | cib=0.487.3 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 17:05:44 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 17:06:47 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 17:06:47 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_0 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node2_an-pdu03 ( an-a02n02 ) due to node availability Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 225, saving inputs in /var/lib/pacemaker/pengine/pe-input-245.bz2 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627085220-860 is obsolete Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 17:05:44 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 17:06:47 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 17:06:47 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_0 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node2_an-pdu03 ( an-a02n02 ) due to node availability Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 226, saving inputs in /var/lib/pacemaker/pengine/pe-input-246.bz2 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 226 (ref=pe_calc-dc-1627085220-861) derived from /var/lib/pacemaker/pengine/pe-input-246.bz2 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node2_an-pdu03_stop_0 on an-a02n02 | action 11 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.487.3 2 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.487.4 (null) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @operation_key=apc_snmp_node2_an-pdu03_stop_0, @operation=stop, @transition-key=11:226:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;11:226:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/494, version=0.487.4) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.487.4 2 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.487.5 (null) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:0;11:226:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=333, @rc-code=0, @op-status=0 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/495, version=0.487.5) Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 226 action 11 (apc_snmp_node2_an-pdu03_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=333 Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 226 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-246.bz2): Complete Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:07:00 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]: 1627085220 -> (unset) | from an-a02n01 Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 515 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/515) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 516 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/516) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.487.5 2 Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.487.6 (null) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu03.start_0'] Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/515, version=0.487.6) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 515 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 226 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu03.start_0']: Transient attribute change | cib=0.487.6 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu03.start_0'] complete=true Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.487.6 2 Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.487.7 (null) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu03.start_0'] Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/516, version=0.487.7) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 226 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu03.start_0']: Transient attribute change | cib=0.487.7 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu03.start_0'] complete=true Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 516 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.487.7 2 Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.487.8 (null) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/496, version=0.487.7) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.487.7 2 Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.487.8 (null) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/497, version=0.487.8) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.487.8 2 Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.488.0 (null) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=488, @num_updates=0 Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']: @value=1627085223 Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n02/crmd/499, version=0.488.0) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-8.raw Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.488.0 of the CIB to disk (digest: 03bc3010bdbe1be836d9ded1ef9da3a4) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.9USXVQ (digest: /var/lib/pacemaker/cib/cib.rksxEU) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 226 aborted by deletion of lrm_resource[@id='apc_snmp_node2_an-pdu03']: Resource state removal | cib=0.487.8 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03'] complete=true Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 226 aborted by cib-bootstrap-options-last-lrm-refresh doing modify last-lrm-refresh=1627085223: Configuration change | cib=0.488.0 source=te_update_diff_v2:474 path=/cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh'] complete=true Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 17:05:44 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 17:06:47 2021 | rc=1 id=apc_snmp_node2_an-pdu03_last_failure_0 Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:07:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 227, saving inputs in /var/lib/pacemaker/pengine/pe-input-247.bz2 Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627085223-863 is obsolete Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 17:05:44 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 228, saving inputs in /var/lib/pacemaker/pengine/pe-input-248.bz2 Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 228 (ref=pe_calc-dc-1627085224-864) derived from /var/lib/pacemaker/pengine/pe-input-248.bz2 Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node2_an-pdu03_monitor_0 on an-a02n02 | action 11 Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.488.0 2 Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.488.1 (null) Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/501, version=0.488.1) Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.488.1 2 Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.488.2 (null) Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu03_last_0']: @transition-magic=0:7;11:228:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=338, @rc-code=7, @op-status=0 Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/502, version=0.488.2) Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 228 action 11 (apc_snmp_node2_an-pdu03_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=338 Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 228 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-248.bz2): Complete Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:07:04 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.488.2 2 Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.489.0 (null) Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=489, @num_updates=0 Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu04']: Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.489.0) Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 228 aborted by meta_attributes.apc_snmp_node2_an-pdu04-meta_attributes 'create': Configuration change | cib=0.489.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu04'] complete=true Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create primitive[@id='apc_snmp_node2_an-pdu04'] Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.489.0 Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu03 has been disabled Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu04 has been disabled Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-9.raw Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.489.0 of the CIB to disk (digest: 907145ed64b77afa65a6d7376766dc1d) Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.lAWd7K (digest: /var/lib/pacemaker/cib/cib.VieREH) Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 17:05:44 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 17:05:29 2021 | rc=1 id=apc_snmp_node2_an-pdu04_last_failure_0 Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node2_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 229, saving inputs in /var/lib/pacemaker/pengine/pe-input-249.bz2 Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 229 (ref=pe_calc-dc-1627085227-866) derived from /var/lib/pacemaker/pengine/pe-input-249.bz2 Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 229 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-249.bz2): Complete Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:07:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:07:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: e2a3f3ec58e28497b2ef0c583370d1a9 for 0.489.0 (0x5619762ed370 0) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] to all (origin=local/crmd/1683) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 517 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]: 1627085144 -> (unset) | from an-a02n01 Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 518 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node2_an-pdu04#start_0' delayed: update 517 in progress Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]: 1627085158 -> (unset) | from an-a02n01 Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node2_an-pdu04#start_0' delayed: update 518 in progress Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.489.0 2 Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.489.1 (null) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: OK (rc=0, origin=an-a02n01/crmd/1683, version=0.489.0) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-controld [121633] (delete_resource) info: Removing resource apc_snmp_node2_an-pdu04 from executor for db598432-1208-439c-8e97-2ab36ce0d38d as root Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/517) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_resource_history) info: Clearing resource history for apc_snmp_node2_an-pdu04 on an-a02n01 (via CIB call 1684) | xpath=//node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/518) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-controld [121633] (notify_deleted) info: Notifying db598432-1208-439c-8e97-2ab36ce0d38d on an-a02n01 that apc_snmp_node2_an-pdu04 was deleted Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] to all (origin=local/crmd/1684) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.489.0 2 Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.489.1 (null) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu04.start_0'] Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/517, version=0.489.1) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 229 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu04.start_0']: Transient attribute change | cib=0.489.1 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node2_an-pdu04.start_0'] complete=true Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 517 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 519 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.489.1 2 Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.489.2 (null) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu04.start_0'] Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/518, version=0.489.2) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 229 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu04.start_0']: Transient attribute change | cib=0.489.2 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node2_an-pdu04.start_0'] complete=true Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 518 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=1627085158 Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 520 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.489.2 2 Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.489.3 (null) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: OK (rc=0, origin=an-a02n01/crmd/1684, version=0.489.3) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 229 aborted by deletion of lrm_resource[@id='apc_snmp_node2_an-pdu04']: Resource state removal | cib=0.489.3 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] complete=true Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1686) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/519) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/520) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.489.3 2 Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.490.0 (null) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=490, @num_updates=0 Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']: @value=1627085233 Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1686, version=0.490.0) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.490.0 2 Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.490.1 (null) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu04.start_0'] Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/519, version=0.490.1) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 519 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.490.1 2 Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.490.2 (null) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: OK (rc=0, origin=an-a02n02/crmd/503, version=0.490.1) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.490.1 2 Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.490.2 (null) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu04.start_0'] Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/520, version=0.490.2) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 520 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.490.2 2 Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.490.3 (null) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']: OK (rc=0, origin=an-a02n02/crmd/504, version=0.490.3) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-10.raw Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n02/crmd/506, version=0.490.3) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.490.0 of the CIB to disk (digest: 0f0d36c79e0c3454d1de139434126bc9) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.aCC7IP (digest: /var/lib/pacemaker/cib/cib.YNNieP) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 229 aborted by cib-bootstrap-options-last-lrm-refresh doing modify last-lrm-refresh=1627085233: Configuration change | cib=0.490.0 source=te_update_diff_v2:474 path=/cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh'] complete=true Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 229 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu04.start_0']: Transient attribute change | cib=0.490.1 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node2_an-pdu04.start_0'] complete=true Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 229 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu04.start_0']: Transient attribute change | cib=0.490.2 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node2_an-pdu04.start_0'] complete=true Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 229 aborted by deletion of lrm_resource[@id='apc_snmp_node2_an-pdu04']: Resource state removal | cib=0.490.3 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04'] complete=true Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:07:13 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 230, saving inputs in /var/lib/pacemaker/pengine/pe-input-250.bz2 Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627085233-868 is obsolete Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 231, saving inputs in /var/lib/pacemaker/pengine/pe-input-250.bz2 Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (handle_pecalc_op) info: Input has not changed since last time, not saving to disk Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 231 (ref=pe_calc-dc-1627085234-869) derived from /var/lib/pacemaker/pengine/pe-input-250.bz2 Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node2_an-pdu04_monitor_0 on an-a02n02 | action 12 Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node2_an-pdu04_monitor_0 locally on an-a02n01 | action 11 Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_get_rsc_info) info: Agent information for 'apc_snmp_node2_an-pdu04' not in cache Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_rsc_register) info: Cached agent information for 'apc_snmp_node2_an-pdu04' Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for apc_snmp_node2_an-pdu04 on an-a02n01 | transition_key=11:231:7:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node2_an-pdu04_monitor_0 Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of probe operation for apc_snmp_node2_an-pdu04 on an-a02n01: not running | rc=7 call=310 key=apc_snmp_node2_an-pdu04_monitor_0 confirmed=true cib-update=1697 Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1696) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1697) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.490.3 2 Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.490.4 (null) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1696, version=0.490.4) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.490.4 2 Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.490.5 (null) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:7;11:231:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=310, @rc-code=7, @op-status=0 Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1697, version=0.490.5) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 231 action 11 (apc_snmp_node2_an-pdu04_monitor_0 on an-a02n01) confirmed: not running | rc=7 call-id=310 Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.490.5 2 Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.490.6 (null) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/508, version=0.490.6) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.490.6 2 Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.490.7 (null) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node2_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node2_an-pdu04_last_0']: @transition-magic=0:7;12:231:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=343, @rc-code=7, @op-status=0 Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/509, version=0.490.7) Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 231 action 12 (apc_snmp_node2_an-pdu04_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=343 Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 231 (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-250.bz2): Complete Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:07:14 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:07:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: d2e465c33457a4d867045575ef3efca5 for 0.490.7 (0x561976403250 0) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.490.7 2 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.491.0 a0fddb4ef7b7cb782805fbdd45f1b843 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=491, @num_updates=0 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03']/instance_attributes[@id='apc_snmp_node1_an-pdu03-instance_attributes']/nvpair[@id='apc_snmp_node1_an-pdu03-instance_attributes-ip']: @value=10.201.2.3 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03']/instance_attributes[@id='apc_snmp_node1_an-pdu03-instance_attributes']/nvpair[@id='apc_snmp_node1_an-pdu03-instance_attributes-port']: @value=3 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.490.7 -> 0.491.0 from an-a02n01 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.491.0) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 231 aborted by apc_snmp_node1_an-pdu03-instance_attributes-ip doing modify ip=10.201.2.3: Configuration change | cib=0.491.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03']/instance_attributes[@id='apc_snmp_node1_an-pdu03-instance_attributes']/nvpair[@id='apc_snmp_node1_an-pdu03-instance_attributes-ip'] complete=true Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1700) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 521 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1701) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 522 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 523 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 524 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 525 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 526 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 527 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 528 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/521) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/522) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/523) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/524) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/525) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: modify nvpair[@id='apc_snmp_node1_an-pdu03-instance_attributes-ip'] Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.491.0 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1700, version=0.491.0) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu03 has been disabled Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu04 has been disabled Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.491.0 2 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.491.1 (null) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1701, version=0.491.1) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.491.1 2 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.491.2 (null) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/521, version=0.491.2) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 521 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.491.2 2 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.491.3 (null) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/522, version=0.491.3) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 522 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.491.3 2 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.491.4 (null) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/523, version=0.491.4) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 523 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.491.4 2 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.491.5 (null) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/524, version=0.491.5) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 524 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.491.5 2 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.491.6 (null) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/525, version=0.491.6) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 525 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/526) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/527) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/528) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.491.6 2 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.491.7 (null) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/526, version=0.491.7) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 526 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.491.7 2 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.491.8 (null) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/527, version=0.491.8) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 527 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.491.8 2 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.491.9 (null) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/528, version=0.491.9) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 528 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-11.raw Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.491.0 of the CIB to disk (digest: ee8a02315ddd18f0d979e3c6207fe0c7) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.rqz8It (digest: /var/lib/pacemaker/cib/cib.i0wAJt) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/1703, version=0.491.9) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1704) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1706) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1708) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1710) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-35 offer to an-a02n02 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-35 offer to an-a02n01 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-35 requests from 2 outstanding nodes Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1704, version=0.491.9) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1706, version=0.491.9) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1708, version=0.491.9) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1710, version=0.491.9) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: 06d7ebeeffe0b39a32a560967e1d6992 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.491.9 with 0.491.9 from an-a02n01 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/1713, version=0.491.9) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1714) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1715) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-12.raw Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1714, version=0.491.9) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.491.0 of the CIB to disk (digest: ee8a02315ddd18f0d979e3c6207fe0c7) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.0IXhld (digest: /var/lib/pacemaker/cib/cib.O0MONb) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1715, version=0.491.9) Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[100110] exited with status 0 Jul 23 17:08:34 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[100113] exited with status 0 Jul 23 17:08:35 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[100123] exited with status 0 Jul 23 17:08:35 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[100136] exited with status 0 Jul 23 17:08:35 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[100142] exited with status 0 Jul 23 17:08:35 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[100146] exited with status 0 Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[100149] exited with status 0 Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[100171] exited with status 0 Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 1716) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/1716) Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1717) Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 1718) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.491.9 2 Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.491.10 (null) Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/1716, version=0.491.10) Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.491.10 2 Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.491.11 (null) Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1717, version=0.491.11) Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/1718) Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.491.11 2 Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.491.12 (null) Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/1718, version=0.491.12) Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[100196] exited with status 0 Jul 23 17:08:36 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[100204] exited with status 0 Jul 23 17:08:37 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[100326] exited with status 0 Jul 23 17:08:37 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[100337] exited with status 0 Jul 23 17:08:37 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[100363] exited with status 0 Jul 23 17:08:37 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[100365] exited with status 0 Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[100366] exited with status 0 Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[100520] exited with status 0 Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1719) Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 231 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1722) Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1723) Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1724) Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.491.12 2 Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.491.13 (null) Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1719, version=0.491.13) Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1722, version=0.491.13) Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.491.13 2 Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.491.14 (null) Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1723, version=0.491.14) Jul 23 17:08:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1724, version=0.491.14) Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 231 aborted: History refresh | source=process_resource_updates:268 complete=true Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 232, saving inputs in /var/lib/pacemaker/pengine/pe-input-251.bz2 Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 232 (ref=pe_calc-dc-1627085319-879) derived from /var/lib/pacemaker/pengine/pe-input-251.bz2 Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 232 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-251.bz2): Complete Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:08:39 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:08:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 534261041948a19c1b4c5fbc36f1119d for 0.491.14 (0x561975c42d10 0) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.491.14 2 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.492.0 (null) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=492, @num_updates=0 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.491.14 -> 0.492.0 from an-a02n02 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.492.0) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 232 aborted by deletion of fencing-topology: Configuration change | cib=0.492.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology complete=true Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 530 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level delete operation 0.492.0 for /cib/configuration/fencing-topology Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 531 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 532 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 533 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 534 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 535 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 536 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 537 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1729) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1730) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/530) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/531) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/532) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/533) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/534) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-13.raw Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.492.0 of the CIB to disk (digest: dec7769d1b6009a7d78958be79cd9f40) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.2u61G6 (digest: /var/lib/pacemaker/cib/cib.OEbUg6) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1729, version=0.492.0) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.492.0 2 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.492.1 (null) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1730, version=0.492.1) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.492.1 2 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.492.2 (null) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/530, version=0.492.2) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 530 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.492.2 2 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.492.3 (null) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/531, version=0.492.3) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 531 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.492.3 2 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.492.4 (null) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/532, version=0.492.4) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 532 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.492.4 2 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.492.5 (null) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/533, version=0.492.5) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 533 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.492.5 2 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.492.6 (null) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/534, version=0.492.6) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 534 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/535) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/536) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/537) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.492.6 2 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.492.7 (null) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/535, version=0.492.7) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 535 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.492.7 2 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.492.8 (null) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/536, version=0.492.8) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 536 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.492.8 2 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.492.9 (null) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/537, version=0.492.9) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 537 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/1732, version=0.492.9) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1733) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1735) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1733, version=0.492.9) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1735, version=0.492.9) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1737) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1739) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-36 offer to an-a02n02 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-36 offer to an-a02n01 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-36 requests from 2 outstanding nodes Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1737, version=0.492.9) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1739, version=0.492.9) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: faa4f3305be84926192f9421d9049786 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.492.9 with 0.492.9 from an-a02n01 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/1742, version=0.492.9) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1743) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1744) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-14.raw Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1743, version=0.492.9) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.492.0 of the CIB to disk (digest: dec7769d1b6009a7d78958be79cd9f40) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.BLRq2c (digest: /var/lib/pacemaker/cib/cib.6FiTRd) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1744, version=0.492.9) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[103034] exited with status 0 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.492.9 2 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.493.0 (null) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=493, @num_updates=0 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.493.0) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Device 'apc_snmp_node1_an-pdu03' not found (4 active devices) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-15.raw Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.493.0 of the CIB to disk (digest: a015d726d36a7d3a3a4280e73e2bc8d0) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.2sZH4m (digest: /var/lib/pacemaker/cib/cib.ZBZqTm) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[103036] exited with status 0 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.493.0 2 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.494.0 (null) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=494 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources: Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.494.0) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create resources Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.494.0 Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu04 has been disabled Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node1_an-pdu03' to device list (5 active devices) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-16.raw Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.494.0 of the CIB to disk (digest: 100f0f1313f3beefbf6392564207d255) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.1k1iLx (digest: /var/lib/pacemaker/cib/cib.S3KKQu) Jul 23 17:09:12 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[103038] exited with status 0 Jul 23 17:09:13 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[103040] exited with status 0 Jul 23 17:09:13 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[103058] exited with status 0 Jul 23 17:09:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.494.0 2 Jul 23 17:09:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.495.0 (null) Jul 23 17:09:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=495 Jul 23 17:09:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration: Jul 23 17:09:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.495.0) Jul 23 17:09:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.495.0 for /cib/configuration. Jul 23 17:09:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 17:09:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:09:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-17.raw Jul 23 17:09:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.495.0 of the CIB to disk (digest: 289cb636dc6e25ab37221ea6b1d3cb4a) Jul 23 17:09:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.o79ORy (digest: /var/lib/pacemaker/cib/cib.eoUJ5w) Jul 23 17:09:13 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[103060] exited with status 0 Jul 23 17:09:13 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[103105] exited with status 0 Jul 23 17:09:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.495.0 2 Jul 23 17:09:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.496.0 (null) Jul 23 17:09:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=496 Jul 23 17:09:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 17:09:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.496.0) Jul 23 17:09:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.496.0 for /cib/configuration/fencing-topology Jul 23 17:09:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 17:09:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:09:13 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 17:09:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-18.raw Jul 23 17:09:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.496.0 of the CIB to disk (digest: 2ca280818c70ecf80835b56d2da73337) Jul 23 17:09:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.hyGOKL (digest: /var/lib/pacemaker/cib/cib.tImEFJ) Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[103248] exited with status 0 Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 1745) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/1745) Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1746) Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 1747) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.496.0 2 Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.496.1 (null) Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/1745, version=0.496.1) Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.496.1 2 Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.496.2 (null) Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1746, version=0.496.2) Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/1747) Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.496.2 2 Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.496.3 (null) Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/1747, version=0.496.3) Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[103385] exited with status 0 Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.496.3 2 Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.497.0 (null) Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=497, @num_updates=0 Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n02/cibadmin/2, version=0.497.0) Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.497.0 for /cib/configuration/fencing-topology Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-19.raw Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.497.0 of the CIB to disk (digest: 220230069c1c620e4a29f928af762e0e) Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.0pAqQh (digest: /var/lib/pacemaker/cib/cib.B6H7Lj) Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[103389] exited with status 0 Jul 23 17:09:14 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[103531] exited with status 0 Jul 23 17:09:15 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[103546] exited with status 0 Jul 23 17:09:15 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[103552] exited with status 0 Jul 23 17:09:15 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[103554] exited with status 0 Jul 23 17:09:15 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[103705] exited with status 0 Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[103717] exited with status 0 Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1748) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 232 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.497.0 2 Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.497.1 (null) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1748, version=0.497.1) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 232 aborted: History refresh | source=process_resource_updates:268 complete=true Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1751) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1752) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1753) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1751, version=0.497.1) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.497.1 2 Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.497.2 (null) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1752, version=0.497.2) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1753, version=0.497.2) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 233, saving inputs in /var/lib/pacemaker/pengine/pe-input-252.bz2 Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 233 (ref=pe_calc-dc-1627085356-887) derived from /var/lib/pacemaker/pengine/pe-input-252.bz2 Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 on an-a02n02 | action 31 Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.497.2 2 Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.497.3 (null) Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=31:233:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;31:233:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 17:09:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/514, version=0.497.3) Jul 23 17:09:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 87068aa55a8521d27183f15c8f61997a for 0.497.3 (0x56197615c490 0) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.497.3 2 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.497.4 (null) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:1;31:233:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=344, @rc-code=1, @op-status=0, @exec-time=13316 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/539, version=0.497.5) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 539 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.497.5 2 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.497.6 (null) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/540, version=0.497.6) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 540 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627085369 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 233 aborted by status-2-fail-count-apc_snmp_node1_an-pdu03.start_0 doing create fail-count-apc_snmp_node1_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.497.5 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 233 aborted by status-2-last-failure-apc_snmp_node1_an-pdu03.start_0 doing create last-failure-apc_snmp_node1_an-pdu03#start_0=1627085369: Transient attribute change | cib=0.497.6 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:09:16 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:09:16 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 234, saving inputs in /var/lib/pacemaker/pengine/pe-input-253.bz2 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627085369-889 is obsolete Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:09:16 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:09:16 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n01 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n02 -> an-a02n01 ) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 235, saving inputs in /var/lib/pacemaker/pengine/pe-input-254.bz2 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 235 (ref=pe_calc-dc-1627085369-890) derived from /var/lib/pacemaker/pengine/pe-input-254.bz2 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 on an-a02n02 | action 10 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.497.6 2 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.497.7 (null) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @transition-key=10:235:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;10:235:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/516, version=0.497.7) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.497.7 2 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.497.8 (null) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;10:235:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=345, @rc-code=0, @op-status=0 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/517, version=0.497.8) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 235 action 10 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=345 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 locally on an-a02n01 | action 32 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=32:235:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_start_0 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1759) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu03 action:start call_id:311 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.497.8 2 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.497.9 (null) Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=32:235:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;32:235:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-cod Jul 23 17:09:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1759, version=0.497.9) Jul 23 17:09:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 5989cb194d2a0fe343a1a558e5d45b99 for 0.497.9 (0x56197615c490 0) Jul 23 17:09:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[104045] error output [ 2021-07-23 17:09:35,727 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:09:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[104045] error output [ ] Jul 23 17:09:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[104045] error output [ ] Jul 23 17:09:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[104045] error output [ 2021-07-23 17:09:35,728 ERROR: Please use '-h' for usage ] Jul 23 17:09:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[104045] error output [ ] Jul 23 17:09:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[104045] stderr: [ 2021-07-23 17:09:35,727 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:09:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[104045] stderr: [ ] Jul 23 17:09:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[104045] stderr: [ ] Jul 23 17:09:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[104045] stderr: [ 2021-07-23 17:09:35,728 ERROR: Please use '-h' for usage ] Jul 23 17:09:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[104045] stderr: [ ] Jul 23 17:09:35 an-a02n01.alteeve.com pacemaker-fenced [121629] (internal_stonith_action_execute) info: Attempt 2 to execute fence_apc_snmp (monitor). remaining timeout is 14 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[104143] error output [ 2021-07-23 17:09:42,886 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[104143] error output [ ] Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[104143] error output [ ] Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[104143] error output [ 2021-07-23 17:09:42,886 ERROR: Please use '-h' for usage ] Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[104143] error output [ ] Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[104143] stderr: [ 2021-07-23 17:09:42,886 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[104143] stderr: [ ] Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[104143] stderr: [ ] Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[104143] stderr: [ 2021-07-23 17:09:42,886 ERROR: Please use '-h' for usage ] Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[104143] stderr: [ ] Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_remaining_timeout) info: Attempted to execute agent fence_apc_snmp (monitor) the maximum number of times (2) allowed Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_operation) notice: Operation 'monitor' [104143] using apc_snmp_node1_an-pdu03 returned -201 (Generic Pacemaker error) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu03 start (call 311) exited with status 1 (execution time 13317ms, queue time 0ms) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for apc_snmp_node1_an-pdu03 on an-a02n01: error | rc=1 call=311 key=apc_snmp_node1_an-pdu03_start_0 confirmed=true cib-update=1760 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1760) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.497.9 2 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.497.10 (null) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:1;32:235:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=311, @rc-code=1, @op-status=0, @exec-time=13317 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/541, version=0.497.11) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 541 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.497.11 2 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.497.12 (null) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/542, version=0.497.12) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 542 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=1627085382 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627085369 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 235 aborted by status-1-fail-count-apc_snmp_node1_an-pdu03.start_0 doing create fail-count-apc_snmp_node1_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.497.11 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 235 aborted by status-1-last-failure-apc_snmp_node1_an-pdu03.start_0 doing create last-failure-apc_snmp_node1_an-pdu03#start_0=1627085382: Transient attribute change | cib=0.497.12 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:09:29 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:09:29 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:09:16 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n01 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n01 ) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 236, saving inputs in /var/lib/pacemaker/pengine/pe-input-255.bz2 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627085382-893 is obsolete Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:09:29 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:09:29 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:09:16 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu03 ( an-a02n01 ) due to node availability Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 237, saving inputs in /var/lib/pacemaker/pengine/pe-input-256.bz2 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 237 (ref=pe_calc-dc-1627085382-894) derived from /var/lib/pacemaker/pengine/pe-input-256.bz2 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 locally on an-a02n01 | action 2 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=2:237:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_stop_0 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu03 action:stop call_id:312 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1764) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu03 stop (call 312) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01: ok | rc=0 call=312 key=apc_snmp_node1_an-pdu03_stop_0 confirmed=true cib-update=1765 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1765) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.497.12 2 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.497.13 (null) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @transition-key=2:237:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;2:237:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1627 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1764, version=0.497.13) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.497.13 2 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.497.14 (null) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;2:237:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=312, @rc-code=0, @op-status=0 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1765, version=0.497.14) Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 237 action 2 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=312 Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 237 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-256.bz2): Complete Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:09:42 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:09:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: ecb11dc440d18f2c07082fc34f6b5e9c for 0.497.14 (0x56197615c490 0) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.497.14 2 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.498.0 1f15c600da572f290b10b3a8ef259a95 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=498, @num_updates=0 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1']: @id=fl-an-a02n01-1, @index=1 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2']: @devices=apc_snmp_node1_an-pdu04, @id=fl-an-a02n01-2, @index=2 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3']: @id=fl-an-a02n01-3, @index=3 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.497.14 -> 0.498.0 from an-a02n01 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.498.0) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 237 aborted by fencing-level.fl-an-a02n01-1 'modify': Configuration change | cib=0.498.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] complete=true Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 543 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 544 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 545 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 546 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 547 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 548 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 549 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 550 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/543) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/544) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/545) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/546) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/547) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.498.0 2 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.498.1 (null) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/543, version=0.498.1) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 543 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.498.1 2 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.498.2 (null) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/544, version=0.498.2) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 544 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/545, version=0.498.2) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 545 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/546, version=0.498.2) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 546 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=1627085382 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627085369 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.498.2 2 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.498.3 (null) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 1 from fencing topology for an-a02n01 (2 active levels remaining) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 2 from fencing topology for an-a02n01 (2 active levels remaining) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/547, version=0.498.3) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 3 from fencing topology for an-a02n01 (2 active levels remaining) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1768) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1769) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 547 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/548) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/549) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/550) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1768, version=0.498.3) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.498.3 2 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.498.4 (null) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1769, version=0.498.4) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.498.4 2 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.498.5 (null) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/548, version=0.498.5) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 548 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.498.5 2 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.498.6 (null) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/549, version=0.498.6) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 549 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.498.6 2 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.498.7 (null) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/550, version=0.498.7) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 550 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-20.raw Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.498.0 of the CIB to disk (digest: 3e574acd234a7f23a29ff8e778b4a7f0) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.IPtIgE (digest: /var/lib/pacemaker/cib/cib.twfjwF) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/1771, version=0.498.7) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1772) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1774) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1772, version=0.498.7) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1774, version=0.498.7) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1776) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-37 offer to an-a02n02 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-37 offer to an-a02n01 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-37 requests from 2 outstanding nodes Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1776, version=0.498.7) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1778) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1778, version=0.498.7) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: 4947d99cbc083db127e892530ab5d41b Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.498.7 with 0.498.7 from an-a02n01 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/1781, version=0.498.7) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1782) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1783) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1782, version=0.498.7) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1783, version=0.498.7) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[104488] exited with status 0 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-21.raw Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.498.0 of the CIB to disk (digest: 3e574acd234a7f23a29ff8e778b4a7f0) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.ZKXaMl (digest: /var/lib/pacemaker/cib/cib.ZuE1Sl) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/cibadmin/2) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.498.7 2 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.499.0 (null) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=499, @num_updates=0 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.499.0) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Removed 'apc_snmp_node1_an-pdu03' from device list (4 active devices) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-22.raw Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.499.0 of the CIB to disk (digest: 85a623e46dd25f64d9b68bb7b8c8ce0c) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.3EmDm6 (digest: /var/lib/pacemaker/cib/cib.K1wEV2) Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[104492] exited with status 0 Jul 23 17:10:01 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[104638] exited with status 0 Jul 23 17:10:02 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[104777] exited with status 0 Jul 23 17:10:02 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[104846] exited with status 0 Jul 23 17:10:02 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[104856] exited with status 0 Jul 23 17:10:02 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[104864] exited with status 0 Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[104874] exited with status 0 Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 1784) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/1784) Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1785) Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 1786) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.499.0 2 Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.499.1 (null) Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/1784, version=0.499.1) Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.499.1 2 Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.499.2 (null) Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1785, version=0.499.2) Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/1786) Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.499.2 2 Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.499.3 (null) Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/1786, version=0.499.3) Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[104881] exited with status 0 Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[104882] exited with status 0 Jul 23 17:10:03 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[104883] exited with status 0 Jul 23 17:10:04 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[104890] exited with status 0 Jul 23 17:10:04 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[104905] exited with status 0 Jul 23 17:10:04 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[104912] exited with status 0 Jul 23 17:10:04 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[104920] exited with status 0 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[105043] exited with status 0 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1787) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 237 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.499.3 2 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.499.4 (null) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1787, version=0.499.4) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1790) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1791) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1792) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 237 aborted: History refresh | source=process_resource_updates:268 complete=true Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1790, version=0.499.4) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.499.4 2 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.499.5 (null) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1791, version=0.499.5) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1792, version=0.499.5) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n01 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now f2317cad3d54cec5d7d7aa7d0bf35cf8 (reload:3.10.2) 0:1;32:235:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n01 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:09:29 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n02 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now f2317cad3d54cec5d7d7aa7d0bf35cf8 (reload:3.10.2) 0:1;31:233:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:09:16 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node1_an-pdu03 from an-a02n01 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node1_an-pdu03 from an-a02n02 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n01 because it is orphaned | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02 because it is orphaned | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 238, saving inputs in /var/lib/pacemaker/pengine/pe-input-257.bz2 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 238 (ref=pe_calc-dc-1627085405-903) derived from /var/lib/pacemaker/pengine/pe-input-257.bz2 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating delete operation apc_snmp_node1_an-pdu03_delete_0 on an-a02n02 | action 14 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating delete operation apc_snmp_node1_an-pdu03_delete_0 locally on an-a02n01 | action 13 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/crmd/1795) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.499.5 2 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.499.6 (null) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/1795, version=0.499.5) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (delete_resource) info: Removing resource apc_snmp_node1_an-pdu03 from executor for tengine Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_resource_history) info: Clearing resource history for apc_snmp_node1_an-pdu03 on an-a02n01 (via CIB call 1796) | xpath=//node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (notify_deleted) info: Notifying tengine on localhost that apc_snmp_node1_an-pdu03 was deleted Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 238 action 13 (apc_snmp_node1_an-pdu03_delete_0 on an-a02n01) confirmed: ok | rc=0 call-id=0 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (te_crm_command) info: Executing crm-event (11) without waiting: clear_failcount on an-a02n02 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (te_crm_command) info: Executing crm-event (2) locally without waiting: clear_failcount on an-a02n01 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/crmd/1796) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (update_attrd_clear_failures) info: Asking pacemaker-attrd to clear failure of all operations for apc_snmp_node1_an-pdu03 on cluster node an-a02n01 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 552 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: 1627085382 -> (unset) | from an-a02n01 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 553 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.499.5 2 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.499.6 (null) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/1796, version=0.499.6) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 238 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu03']: Resource state removal | cib=0.499.6 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] complete=false Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.499.6 2 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.499.7 (null) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/520, version=0.499.6) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section /cib/status/node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0'] to all (origin=local/crmd/1797) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/552) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/553) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n02 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node1_an-pdu03#start_0' delayed: update 552 in progress Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 238 action 14 (apc_snmp_node1_an-pdu03_delete_0 on an-a02n02) confirmed: ok | rc=0 call-id=0 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: 1627085369 -> (unset) | from an-a02n02 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node1_an-pdu03#start_0' delayed: update 553 in progress Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 238 (Complete=4, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-257.bz2): Complete Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section /cib/status/node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0']: OK (rc=0, origin=an-a02n01/crmd/1797, version=0.499.6) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.499.6 2 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.499.7 (null) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/552, version=0.499.7) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 552 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 238 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.499.7 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 554 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.499.7 2 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.499.8 (null) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/553, version=0.499.8) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 553 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627085369 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 238 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.499.8 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 555 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.499.8 2 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.499.9 (null) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/521, version=0.499.9) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 238 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu03']: Resource state removal | cib=0.499.9 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] complete=true Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section /cib/status/node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0']: OK (rc=0, origin=an-a02n02/crmd/522, version=0.499.9) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/554) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/555) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.499.9 2 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.499.10 (null) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/554, version=0.499.10) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 554 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.499.10 2 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.499.11 (null) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/555, version=0.499.11) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 555 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 238 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.499.10 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 238 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.499.11 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 239, saving inputs in /var/lib/pacemaker/pengine/pe-input-258.bz2 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627085405-909 is obsolete Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 240, saving inputs in /var/lib/pacemaker/pengine/pe-input-259.bz2 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 240 (ref=pe_calc-dc-1627085405-910) derived from /var/lib/pacemaker/pengine/pe-input-259.bz2 Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 240 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-259.bz2): Complete Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:10:05 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.499.11 2 Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.500.0 (null) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=500, @num_updates=0 Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources: Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.500.0) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 240 aborted by primitive.apc_snmp_node1_an-pdu03 'create': Configuration change | cib=0.500.0 source=te_update_diff_v2:474 path=/cib/configuration/resources complete=true Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create resources Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.500.0 Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu04 has been disabled Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node1_an-pdu03' to device list (5 active devices) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-23.raw Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.500.0 of the CIB to disk (digest: 5e8d619d002046a917b0ac7928529f26) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.wXsva3 (digest: /var/lib/pacemaker/cib/cib.aCRLQ1) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 241, saving inputs in /var/lib/pacemaker/pengine/pe-input-260.bz2 Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 241 (ref=pe_calc-dc-1627085407-911) derived from /var/lib/pacemaker/pengine/pe-input-260.bz2 Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu03_monitor_0 on an-a02n02 | action 12 Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu03_monitor_0 locally on an-a02n01 | action 11 Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_get_rsc_info) info: Agent information for 'apc_snmp_node1_an-pdu03' not in cache Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_rsc_register) info: Cached agent information for 'apc_snmp_node1_an-pdu03' Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=11:241:7:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_monitor_0 Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1805) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of probe operation for apc_snmp_node1_an-pdu03 on an-a02n01: not running | rc=7 call=317 key=apc_snmp_node1_an-pdu03_monitor_0 confirmed=true cib-update=1806 Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.500.0 2 Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.500.1 (null) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1805, version=0.500.1) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1806) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.500.1 2 Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.500.2 (null) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:7;11:241:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=317, @rc-code=7, @op-status=0 Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1806, version=0.500.2) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 241 action 11 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n01) confirmed: not running | rc=7 call-id=317 Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.500.2 2 Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.500.3 (null) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/523, version=0.500.3) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.500.3 2 Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.500.4 (null) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:7;12:241:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=350, @rc-code=7, @op-status=0 Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/524, version=0.500.4) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 241 action 12 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=350 Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 on an-a02n02 | action 33 Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.500.4 2 Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.500.5 (null) Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @transition-key=33:241:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;33:241:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1 Jul 23 17:10:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/525, version=0.500.5) Jul 23 17:10:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:10:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.500.5 2 Jul 23 17:10:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.501.0 (null) Jul 23 17:10:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] Jul 23 17:10:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=501, @num_updates=0 Jul 23 17:10:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.501.0) Jul 23 17:10:08 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 241 aborted by deletion of fencing-level[@id='fl-an-a02n01-1']: Configuration change | cib=0.501.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] complete=false Jul 23 17:10:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.501.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] Jul 23 17:10:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 17:10:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:10:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 17:10:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-24.raw Jul 23 17:10:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.501.0 of the CIB to disk (digest: 431c1bb443683c69d31b5d7db402e9bc) Jul 23 17:10:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.ryoxVX (digest: /var/lib/pacemaker/cib/cib.mrqwqZ) Jul 23 17:10:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:10:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.501.0 2 Jul 23 17:10:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.502.0 (null) Jul 23 17:10:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] Jul 23 17:10:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=502 Jul 23 17:10:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.502.0) Jul 23 17:10:08 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 241 aborted by deletion of fencing-level[@id='fl-an-a02n01-2']: Configuration change | cib=0.502.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] complete=false Jul 23 17:10:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.502.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] Jul 23 17:10:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 17:10:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:10:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-25.raw Jul 23 17:10:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.502.0 of the CIB to disk (digest: c63653382e2a30a65e41d3ed53d27370) Jul 23 17:10:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.ykjp4L (digest: /var/lib/pacemaker/cib/cib.uzX3CL) Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.502.0 2 Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.503.0 (null) Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3'] Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=503 Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.503.0) Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 241 aborted by deletion of fencing-level[@id='fl-an-a02n01-3']: Configuration change | cib=0.503.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3'] complete=false Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.503.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3'] Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-26.raw Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.503.0 of the CIB to disk (digest: 91e5659c28600ccf000e5f92c29f0a64) Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.nIYQoM (digest: /var/lib/pacemaker/cib/cib.n47KvM) Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.503.0 2 Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.504.0 (null) Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=504 Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.504.0) Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 241 aborted by fencing-level.fl-an-a02n01-1 'create': Configuration change | cib=0.504.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology complete=false Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.504.0 for /cib/configuration/fencing-topology Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-27.raw Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.504.0 of the CIB to disk (digest: e0e33395fc60289c7fce5c4c1c5f5cb0) Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.6ej5ms (digest: /var/lib/pacemaker/cib/cib.m9fKZs) Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.504.0 2 Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.505.0 (null) Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=505 Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.505.0) Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 241 aborted by fencing-level.fl-an-a02n01-2 'create': Configuration change | cib=0.505.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology complete=false Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.505.0 for /cib/configuration/fencing-topology Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-28.raw Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.505.0 of the CIB to disk (digest: 78f11d39bf40453a88d9d81ff0ddc290) Jul 23 17:10:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.IsYtJ9 (digest: /var/lib/pacemaker/cib/cib.rizDW8) Jul 23 17:10:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:10:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.505.0 2 Jul 23 17:10:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.506.0 (null) Jul 23 17:10:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=506 Jul 23 17:10:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 17:10:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.506.0) Jul 23 17:10:10 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 241 aborted by fencing-level.fl-an-a02n01-3 'create': Configuration change | cib=0.506.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology complete=false Jul 23 17:10:10 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.506.0 for /cib/configuration/fencing-topology Jul 23 17:10:10 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 17:10:10 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:10:10 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 17:10:10 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 17:10:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-29.raw Jul 23 17:10:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.506.0 of the CIB to disk (digest: 9ace4f69e1ea5af2b106694df8da9aa6) Jul 23 17:10:10 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.J8QOLC (digest: /var/lib/pacemaker/cib/cib.ulZJKC) Jul 23 17:10:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: cc4958c462fac571c9356198e3b49c3c for 0.506.0 (0x5619762f08c0 0) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.506.0 2 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.506.1 (null) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:1;33:241:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=351, @rc-code=1, @op-status=0, @exec-time=13318 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/556, version=0.506.2) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 556 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.506.2 2 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.506.3 (null) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/557, version=0.506.3) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 557 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627085420 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 241 aborted by status-2-fail-count-apc_snmp_node1_an-pdu03.start_0 doing create fail-count-apc_snmp_node1_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.506.2 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 241 aborted by status-2-last-failure-apc_snmp_node1_an-pdu03.start_0 doing create last-failure-apc_snmp_node1_an-pdu03#start_0=1627085420: Transient attribute change | cib=0.506.3 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:10:07 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:10:07 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 242, saving inputs in /var/lib/pacemaker/pengine/pe-input-261.bz2 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627085420-915 is obsolete Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:10:07 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:10:07 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n01 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n02 -> an-a02n01 ) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 243, saving inputs in /var/lib/pacemaker/pengine/pe-input-262.bz2 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 243 (ref=pe_calc-dc-1627085420-916) derived from /var/lib/pacemaker/pengine/pe-input-262.bz2 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 on an-a02n02 | action 11 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.506.3 2 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.506.4 (null) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @transition-key=11:243:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;11:243:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/527, version=0.506.4) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.506.4 2 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.506.5 (null) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;11:243:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=352, @rc-code=0, @op-status=0 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/528, version=0.506.5) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 243 action 11 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=352 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 locally on an-a02n01 | action 32 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=32:243:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_start_0 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1810) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu03 action:start call_id:318 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.506.5 2 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.506.6 (null) Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @transition-key=32:243:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;32:243:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change= Jul 23 17:10:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1810, version=0.506.6) Jul 23 17:10:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 0cb975d291d42773a17cf6c928f7f84e for 0.506.6 (0x5619762f08c0 0) Jul 23 17:10:27 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[107042] error output [ 2021-07-23 17:10:27,093 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:10:27 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[107042] error output [ ] Jul 23 17:10:27 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[107042] error output [ ] Jul 23 17:10:27 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[107042] error output [ 2021-07-23 17:10:27,093 ERROR: Please use '-h' for usage ] Jul 23 17:10:27 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[107042] error output [ ] Jul 23 17:10:27 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[107042] stderr: [ 2021-07-23 17:10:27,093 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:10:27 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[107042] stderr: [ ] Jul 23 17:10:27 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[107042] stderr: [ ] Jul 23 17:10:27 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[107042] stderr: [ 2021-07-23 17:10:27,093 ERROR: Please use '-h' for usage ] Jul 23 17:10:27 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[107042] stderr: [ ] Jul 23 17:10:27 an-a02n01.alteeve.com pacemaker-fenced [121629] (internal_stonith_action_execute) info: Attempt 2 to execute fence_apc_snmp (monitor). remaining timeout is 13 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[107324] error output [ 2021-07-23 17:10:34,253 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[107324] error output [ ] Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[107324] error output [ ] Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[107324] error output [ 2021-07-23 17:10:34,253 ERROR: Please use '-h' for usage ] Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[107324] error output [ ] Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[107324] stderr: [ 2021-07-23 17:10:34,253 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[107324] stderr: [ ] Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[107324] stderr: [ ] Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[107324] stderr: [ 2021-07-23 17:10:34,253 ERROR: Please use '-h' for usage ] Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[107324] stderr: [ ] Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_remaining_timeout) info: Attempted to execute agent fence_apc_snmp (monitor) the maximum number of times (2) allowed Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_operation) notice: Operation 'monitor' [107324] using apc_snmp_node1_an-pdu03 returned -201 (Generic Pacemaker error) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu03 start (call 318) exited with status 1 (execution time 13320ms, queue time 0ms) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for apc_snmp_node1_an-pdu03 on an-a02n01: error | rc=1 call=318 key=apc_snmp_node1_an-pdu03_start_0 confirmed=true cib-update=1811 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1811) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.506.6 2 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.506.7 (null) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:1;32:243:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=318, @rc-code=1, @op-status=0, @exec-time=13320 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/558, version=0.506.8) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 558 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.506.8 2 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.506.9 (null) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/559, version=0.506.9) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 559 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=1627085434 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627085420 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 243 aborted by status-1-fail-count-apc_snmp_node1_an-pdu03.start_0 doing create fail-count-apc_snmp_node1_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.506.8 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 243 aborted by status-1-last-failure-apc_snmp_node1_an-pdu03.start_0 doing create last-failure-apc_snmp_node1_an-pdu03#start_0=1627085434: Transient attribute change | cib=0.506.9 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:10:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:10:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:10:07 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n01 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n01 ) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 244, saving inputs in /var/lib/pacemaker/pengine/pe-input-263.bz2 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627085434-919 is obsolete Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:10:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:10:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:10:07 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu03 ( an-a02n01 ) due to node availability Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 245, saving inputs in /var/lib/pacemaker/pengine/pe-input-264.bz2 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 245 (ref=pe_calc-dc-1627085434-920) derived from /var/lib/pacemaker/pengine/pe-input-264.bz2 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 locally on an-a02n01 | action 9 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=9:245:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_stop_0 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu03 action:stop call_id:319 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1815) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu03 stop (call 319) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1816) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01: ok | rc=0 call=319 key=apc_snmp_node1_an-pdu03_stop_0 confirmed=true cib-update=1816 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.506.9 2 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.506.10 (null) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @transition-key=9:245:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;9:245:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1627 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1815, version=0.506.10) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.506.10 2 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.506.11 (null) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;9:245:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=319, @rc-code=0, @op-status=0 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1816, version=0.506.11) Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 245 action 9 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=319 Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 245 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-264.bz2): Complete Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:10:34 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:10:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 7fb86f61e898d5ff3fb8f00036b3a2d1 for 0.506.11 (0x5619762f08c0 0) Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.506.11 2 Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.507.0 4a7c9bde644f4815ad3b19c2c10bece6 Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=507, @num_updates=0 Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.507.0) Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 245 aborted by deletion of fencing-level[@id='fl-an-a02n01-2']: Configuration change | cib=0.507.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] complete=true Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.507.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-30.raw Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.507.0 of the CIB to disk (digest: bcd8503886b703ae87bcdf0a530fa386) Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.yhQPA8 (digest: /var/lib/pacemaker/cib/cib.nirZDa) Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:10:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:10:07 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 246, saving inputs in /var/lib/pacemaker/pengine/pe-input-265.bz2 Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 246 (ref=pe_calc-dc-1627085560-922) derived from /var/lib/pacemaker/pengine/pe-input-265.bz2 Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 246 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-265.bz2): Complete Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:12:40 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.507.0 2 Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.508.0 (null) Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3'] Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=508 Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.508.0) Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 246 aborted by deletion of fencing-level[@id='fl-an-a02n01-3']: Configuration change | cib=0.508.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3'] complete=true Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.508.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3'] Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-31.raw Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.508.0 of the CIB to disk (digest: 76d0be3fb050db3ada266e2da1583e47) Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.aS7kv9 (digest: /var/lib/pacemaker/cib/cib.HgtK54) Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:10:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:10:07 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 247, saving inputs in /var/lib/pacemaker/pengine/pe-input-266.bz2 Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 247 (ref=pe_calc-dc-1627085564-923) derived from /var/lib/pacemaker/pengine/pe-input-266.bz2 Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 247 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-266.bz2): Complete Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:12:44 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.508.0 2 Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.509.0 (null) Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=509 Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.509.0) Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 247 aborted by deletion of fencing-level[@id='fl-an-a02n01-1']: Configuration change | cib=0.509.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] complete=true Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.509.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-32.raw Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.509.0 of the CIB to disk (digest: aff749e783f73b7f7f91e3e8d68c164d) Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.5JkCKA (digest: /var/lib/pacemaker/cib/cib.S5KUKy) Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:10:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:10:07 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 248, saving inputs in /var/lib/pacemaker/pengine/pe-input-267.bz2 Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 248 (ref=pe_calc-dc-1627085566-924) derived from /var/lib/pacemaker/pengine/pe-input-267.bz2 Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 248 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-267.bz2): Complete Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:12:46 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:12:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 782e002187c35fe80b4240f5970a25a7 for 0.509.0 (0x5619763278e0 0) Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.509.0 2 Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.510.0 (null) Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=510 Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03']: Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.510.0) Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 248 aborted by meta_attributes.apc_snmp_node1_an-pdu03-meta_attributes 'create': Configuration change | cib=0.510.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03'] complete=true Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create primitive[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.510.0 Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu04 has been disabled Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu03 has been disabled Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-33.raw Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.510.0 of the CIB to disk (digest: 58c61f3e128d2c4b4e135402d18cf138) Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.lz0sCG (digest: /var/lib/pacemaker/cib/cib.Di9d0G) Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:10:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:10:07 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 249, saving inputs in /var/lib/pacemaker/pengine/pe-input-268.bz2 Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 249 (ref=pe_calc-dc-1627085617-925) derived from /var/lib/pacemaker/pengine/pe-input-268.bz2 Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 249 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-268.bz2): Complete Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:13:37 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: b0195f7d25c46f6a13f32c6ac0bd752c for 0.510.0 (0x56197614b780 0) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/crmd/1821) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 560 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/560) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: 1627085420 -> (unset) | from an-a02n01 Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 561 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'fail-count-apc_snmp_node1_an-pdu03#start_0' delayed: update 560 in progress Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: 1627085434 -> (unset) | from an-a02n01 Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Write out of 'last-failure-apc_snmp_node1_an-pdu03#start_0' delayed: update 561 in progress Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/561) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.510.0 2 Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.510.1 94d0d811093788b93ca2fcd51ccab8c5 Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/1821, version=0.510.0) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-controld [121633] (delete_resource) info: Removing resource apc_snmp_node1_an-pdu03 from executor for bd7d453f-4ded-48ad-9dfe-277c32e2a9de as root Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_resource_history) info: Clearing resource history for apc_snmp_node1_an-pdu03 on an-a02n01 (via CIB call 1822) | xpath=//node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-controld [121633] (notify_deleted) info: Notifying bd7d453f-4ded-48ad-9dfe-277c32e2a9de on an-a02n01 that apc_snmp_node1_an-pdu03 was deleted Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.510.0 2 Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.510.1 (null) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/560, version=0.510.1) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 560 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 562 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.510.1 2 Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.510.2 (null) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/561, version=0.510.2) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 561 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=1627085434 Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 563 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.510.2 2 Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.510.3 (null) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/529, version=0.510.2) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/crmd/1822) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/562) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/563) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 249 aborted by deletion of nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.510.1 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-fail-count-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1824) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 249 aborted by deletion of nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.510.2 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-last-failure-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.510.2 2 Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.510.3 (null) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/530, version=0.510.3) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 249 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu03']: Resource state removal | cib=0.510.3 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] complete=true Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.510.3 2 Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.511.0 (null) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=511, @num_updates=0 Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']: @value=1627085622 Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n02/crmd/532, version=0.511.0) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 249 aborted by cib-bootstrap-options-last-lrm-refresh doing modify last-lrm-refresh=1627085622: Configuration change | cib=0.511.0 source=te_update_diff_v2:474 path=/cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh'] complete=true Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.511.0 2 Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.511.1 (null) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/1822, version=0.511.1) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 249 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu03']: Resource state removal | cib=0.511.1 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] complete=true Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.511.1 2 Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.511.2 (null) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/562, version=0.511.2) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 249 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.511.2 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 562 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.511.2 2 Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.511.3 (null) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/563, version=0.511.3) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 249 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.511.3 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 563 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1824, version=0.511.3) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-34.raw Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.511.0 of the CIB to disk (digest: 2d0ee2105467ec04713fe76c02bbd204) Jul 23 17:13:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.8M5YWA (digest: /var/lib/pacemaker/cib/cib.tNZHly) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 250, saving inputs in /var/lib/pacemaker/pengine/pe-input-269.bz2 Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 250 (ref=pe_calc-dc-1627085623-927) derived from /var/lib/pacemaker/pengine/pe-input-269.bz2 Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu03_monitor_0 on an-a02n02 | action 12 Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu03_monitor_0 locally on an-a02n01 | action 11 Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_get_rsc_info) info: Agent information for 'apc_snmp_node1_an-pdu03' not in cache Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_rsc_register) info: Cached agent information for 'apc_snmp_node1_an-pdu03' Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=11:250:7:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_monitor_0 Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of probe operation for apc_snmp_node1_an-pdu03 on an-a02n01: not running | rc=7 call=324 key=apc_snmp_node1_an-pdu03_monitor_0 confirmed=true cib-update=1834 Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1833) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1834) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.511.3 2 Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.511.4 (null) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1833, version=0.511.4) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.511.4 2 Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.511.5 (null) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:7;11:250:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=324, @rc-code=7, @op-status=0 Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1834, version=0.511.5) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 250 action 11 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n01) confirmed: not running | rc=7 call-id=324 Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.511.5 2 Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.511.6 (null) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/534, version=0.511.6) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.511.6 2 Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.511.7 (null) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:7;12:250:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=357, @rc-code=7, @op-status=0 Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/535, version=0.511.7) Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 250 action 12 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=357 Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 250 (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-269.bz2): Complete Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:13:43 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:13:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 12b9fef1b94d5a1462e8c371556a65b4 for 0.511.7 (0x561976310580 0) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.511.7 2 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.512.0 3dfc7822ca6394fac41f59106a90dae3 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=512, @num_updates=0 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.511.7 -> 0.512.0 from an-a02n01 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.512.0) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 250 aborted by deletion of fencing-topology: Configuration change | cib=0.512.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology complete=true Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 564 with 2 changes for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 565 with 2 changes for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 566 with 2 changes for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 567 with 2 changes for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 568 with 2 changes for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 569 with 2 changes for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 570 with 2 changes for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 571 with 2 changes for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1837) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1838) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level delete operation 0.512.0 for /cib/configuration/fencing-topology Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-35.raw Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.512.0 of the CIB to disk (digest: 487737ff10e2b0222c748bbb62de478f) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.G1O5Xk (digest: /var/lib/pacemaker/cib/cib.n55Oym) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1837, version=0.512.0) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.512.0 2 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.512.1 (null) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1838, version=0.512.1) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/564) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/565) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/566) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/567) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/568) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/569) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/570) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/571) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.512.1 2 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.512.2 (null) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/564, version=0.512.2) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 564 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.512.2 2 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.512.3 (null) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/565, version=0.512.3) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 565 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.512.3 2 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.512.4 (null) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/566, version=0.512.4) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 566 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.512.4 2 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.512.5 (null) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/567, version=0.512.5) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 567 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=(null) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.512.5 2 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.512.6 (null) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/568, version=0.512.6) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 568 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.512.6 2 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.512.7 (null) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/569, version=0.512.7) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 569 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.512.7 2 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.512.8 (null) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/570, version=0.512.8) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 570 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.512.8 2 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.512.9 (null) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/571, version=0.512.9) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 571 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=(null) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/1840, version=0.512.9) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1841) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1843) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1845) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-38 offer to an-a02n02 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-38 offer to an-a02n01 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-38 requests from 2 outstanding nodes Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1841, version=0.512.9) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1843, version=0.512.9) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1845, version=0.512.9) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1847) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1847, version=0.512.9) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: b5b801b68bbe4845d578b8d20c4ca7ff Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.512.9 with 0.512.9 from an-a02n01 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/1850, version=0.512.9) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1851) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1852) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1851, version=0.512.9) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1852, version=0.512.9) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[122057] exited with status 0 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-36.raw Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.512.0 of the CIB to disk (digest: 487737ff10e2b0222c748bbb62de478f) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.KWGYYC (digest: /var/lib/pacemaker/cib/cib.Q2lX9F) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/cibadmin/2) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.512.9 2 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.513.0 (null) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=513, @num_updates=0 Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.513.0) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Device 'apc_snmp_node1_an-pdu03' not found (4 active devices) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-37.raw Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.513.0 of the CIB to disk (digest: 1791b11a168c284341adb804e966830c) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.hY7x7V (digest: /var/lib/pacemaker/cib/cib.gsdi2W) Jul 23 17:14:58 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[122060] exited with status 0 Jul 23 17:14:59 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[122063] exited with status 0 Jul 23 17:14:59 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[122064] exited with status 0 Jul 23 17:14:59 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[122065] exited with status 0 Jul 23 17:14:59 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[122067] exited with status 0 Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[122068] exited with status 0 Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[122069] exited with status 0 Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n02 (via CIB call 1853) | xpath=//node_state[@uname='an-a02n02']/lrm Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/lrm to all (origin=local/crmd/1853) Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 1855) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.513.0 2 Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.513.1 (null) Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2'] Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm: OK (rc=0, origin=an-a02n01/crmd/1853, version=0.513.1) Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1854) Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/1855) Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.513.1 2 Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.513.2 (null) Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']: Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1854, version=0.513.2) Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.513.2 2 Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.513.3 (null) Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/1855, version=0.513.3) Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[122074] exited with status 0 Jul 23 17:15:00 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[122075] exited with status 0 Jul 23 17:15:01 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[122076] exited with status 0 Jul 23 17:15:01 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[122081] exited with status 0 Jul 23 17:15:01 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[122082] exited with status 0 Jul 23 17:15:01 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[122084] exited with status 0 Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[122085] exited with status 0 Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[122086] exited with status 0 Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1856) Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 250 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.513.3 2 Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.513.4 (null) Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1856, version=0.513.4) Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 250 aborted: History refresh | source=process_resource_updates:268 complete=true Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1859) Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1860) Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1861) Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1859, version=0.513.4) Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.513.4 2 Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.513.5 (null) Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1860, version=0.513.5) Jul 23 17:15:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1861, version=0.513.5) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node1_an-pdu03 from an-a02n01 Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node1_an-pdu03 from an-a02n02 Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 251, saving inputs in /var/lib/pacemaker/pengine/pe-input-270.bz2 Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 251 (ref=pe_calc-dc-1627085703-937) derived from /var/lib/pacemaker/pengine/pe-input-270.bz2 Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating delete operation apc_snmp_node1_an-pdu03_delete_0 on an-a02n02 | action 12 Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating delete operation apc_snmp_node1_an-pdu03_delete_0 locally on an-a02n01 | action 11 Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/crmd/1865) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.513.5 2 Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.513.6 (null) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/1865, version=0.513.5) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-controld [121633] (delete_resource) info: Removing resource apc_snmp_node1_an-pdu03 from executor for tengine Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_resource_history) info: Clearing resource history for apc_snmp_node1_an-pdu03 on an-a02n01 (via CIB call 1866) | xpath=//node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-controld [121633] (notify_deleted) info: Notifying tengine on localhost that apc_snmp_node1_an-pdu03 was deleted Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 251 action 11 (apc_snmp_node1_an-pdu03_delete_0 on an-a02n01) confirmed: ok | rc=0 call-id=0 Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/crmd/1866) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.513.5 2 Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.513.6 (null) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/1866, version=0.513.6) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 251 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu03']: Resource state removal | cib=0.513.6 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] complete=false Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.513.6 2 Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.513.7 (null) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/538, version=0.513.6) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 251 action 12 (apc_snmp_node1_an-pdu03_delete_0 on an-a02n02) confirmed: ok | rc=0 call-id=0 Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 251 (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-270.bz2): Complete Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.513.6 2 Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.513.7 (null) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n02/crmd/539, version=0.513.7) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 251 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu03']: Resource state removal | cib=0.513.7 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] complete=true Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 252, saving inputs in /var/lib/pacemaker/pengine/pe-input-271.bz2 Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 252 (ref=pe_calc-dc-1627085703-941) derived from /var/lib/pacemaker/pengine/pe-input-271.bz2 Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 252 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-271.bz2): Complete Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:15:03 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:15:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: a5b142c21f4d19d1c6cacdd0b73985e4 for 0.513.7 (0x561976403250 0) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.513.7) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-38.raw Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.513.0 of the CIB to disk (digest: 8ec6a49757debf211ea72515e1d1b881) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.eVwmg1 (digest: /var/lib/pacemaker/cib/cib.81Aen1) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu04'] to all (origin=local/cibadmin/2) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.513.7 2 Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.514.0 (null) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=514, @num_updates=0 Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.514.0) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 252 aborted by deletion of primitive[@id='apc_snmp_node1_an-pdu04']: Configuration change | cib=0.514.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu04'] complete=true Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Device 'apc_snmp_node1_an-pdu04' not found (4 active devices) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-39.raw Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.514.0 of the CIB to disk (digest: 91c98c69ec9e3c36c6218fd0016612b9) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.5JNcoq (digest: /var/lib/pacemaker/cib/cib.kA0vor) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node1_an-pdu04 from an-a02n01 Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node1_an-pdu04 from an-a02n02 Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 253, saving inputs in /var/lib/pacemaker/pengine/pe-input-272.bz2 Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 253 (ref=pe_calc-dc-1627085763-942) derived from /var/lib/pacemaker/pengine/pe-input-272.bz2 Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating delete operation apc_snmp_node1_an-pdu04_delete_0 on an-a02n02 | action 12 Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating delete operation apc_snmp_node1_an-pdu04_delete_0 locally on an-a02n01 | action 11 Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] to all (origin=local/crmd/1870) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.514.0 2 Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.514.1 (null) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n01/crmd/1870, version=0.514.0) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-controld [121633] (delete_resource) info: Removing resource apc_snmp_node1_an-pdu04 from executor for tengine Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_resource_history) info: Clearing resource history for apc_snmp_node1_an-pdu04 on an-a02n01 (via CIB call 1871) | xpath=//node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-controld [121633] (notify_deleted) info: Notifying tengine on localhost that apc_snmp_node1_an-pdu04 was deleted Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 253 action 11 (apc_snmp_node1_an-pdu04_delete_0 on an-a02n01) confirmed: ok | rc=0 call-id=0 Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] to all (origin=local/crmd/1871) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.514.0 2 Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.514.1 (null) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n01/crmd/1871, version=0.514.1) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 253 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu04']: Resource state removal | cib=0.514.1 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] complete=false Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.514.1 2 Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.514.2 (null) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n02/crmd/540, version=0.514.1) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 253 action 12 (apc_snmp_node1_an-pdu04_delete_0 on an-a02n02) confirmed: ok | rc=0 call-id=0 Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 253 (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-272.bz2): Complete Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.514.1 2 Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.514.2 (null) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n02/crmd/541, version=0.514.2) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 253 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu04']: Resource state removal | cib=0.514.2 source=abort_unless_down:327 path=/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] complete=true Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node1_an-pdu04 from an-a02n02 Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 254, saving inputs in /var/lib/pacemaker/pengine/pe-input-273.bz2 Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627085763-946 is obsolete Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 255, saving inputs in /var/lib/pacemaker/pengine/pe-input-274.bz2 Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 255 (ref=pe_calc-dc-1627085763-947) derived from /var/lib/pacemaker/pengine/pe-input-274.bz2 Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 255 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-274.bz2): Complete Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:16:03 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:16:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 429f1d237fbeb79035736950930dc0bc for 0.514.2 (0x56197612f2c0 0) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.514.2 2 Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.515.0 (null) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=515, @num_updates=0 Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources: Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.515.0) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 255 aborted by primitive.apc_snmp_node1_an-pdu03 'create': Configuration change | cib=0.515.0 source=te_update_diff_v2:474 path=/cib/configuration/resources complete=true Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create resources Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.515.0 Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node1_an-pdu03' to device list (5 active devices) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-40.raw Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.515.0 of the CIB to disk (digest: a97379cd8b904b0b0324e56ec6b97986) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.KgLsDS (digest: /var/lib/pacemaker/cib/cib.OM5fIQ) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 256, saving inputs in /var/lib/pacemaker/pengine/pe-input-275.bz2 Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 256 (ref=pe_calc-dc-1627085780-948) derived from /var/lib/pacemaker/pengine/pe-input-275.bz2 Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu03_monitor_0 on an-a02n02 | action 12 Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu03_monitor_0 locally on an-a02n01 | action 11 Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_get_rsc_info) info: Agent information for 'apc_snmp_node1_an-pdu03' not in cache Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_rsc_register) info: Cached agent information for 'apc_snmp_node1_an-pdu03' Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=11:256:7:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_monitor_0 Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1875) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of probe operation for apc_snmp_node1_an-pdu03 on an-a02n01: not running | rc=7 call=330 key=apc_snmp_node1_an-pdu03_monitor_0 confirmed=true cib-update=1876 Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1876) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.515.0 2 Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.515.1 (null) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1875, version=0.515.1) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.515.1 2 Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.515.2 (null) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:7;11:256:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=330, @rc-code=7, @op-status=0 Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1876, version=0.515.2) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 256 action 11 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n01) confirmed: not running | rc=7 call-id=330 Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.515.2 2 Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.515.3 (null) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/542, version=0.515.3) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.515.3 2 Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.515.4 (null) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:7;12:256:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=363, @rc-code=7, @op-status=0 Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/543, version=0.515.4) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 256 action 12 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=363 Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 on an-a02n02 | action 33 Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.515.4 2 Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.515.5 (null) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @transition-key=33:256:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;33:256:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1 Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/544, version=0.515.5) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.515.5 2 Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.516.0 (null) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=516, @num_updates=0 Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources: Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.516.0) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 256 aborted by primitive.apc_snmp_node1_an-pdu04 'create': Configuration change | cib=0.516.0 source=te_update_diff_v2:474 path=/cib/configuration/resources complete=false Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create resources Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.516.0 Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node1_an-pdu04' to device list (6 active devices) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-41.raw Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.516.0 of the CIB to disk (digest: 47ce93b91bf891da9541879673c23152) Jul 23 17:16:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.3OrTnG (digest: /var/lib/pacemaker/cib/cib.ywNwuI) Jul 23 17:16:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:16:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.516.0 2 Jul 23 17:16:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.517.0 (null) Jul 23 17:16:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=517 Jul 23 17:16:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration: Jul 23 17:16:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:16:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:16:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.517.0) Jul 23 17:16:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 256 aborted by fencing-topology. 'create': Configuration change | cib=0.517.0 source=te_update_diff_v2:474 path=/cib/configuration complete=false Jul 23 17:16:21 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.517.0 for /cib/configuration. Jul 23 17:16:21 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 17:16:21 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:16:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-42.raw Jul 23 17:16:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.517.0 of the CIB to disk (digest: 31df70ebb13ec93ab6a7c3b5ae161db9) Jul 23 17:16:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.iFsawU (digest: /var/lib/pacemaker/cib/cib.IGgDwV) Jul 23 17:16:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:16:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.517.0 2 Jul 23 17:16:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.518.0 (null) Jul 23 17:16:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=518 Jul 23 17:16:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 17:16:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.518.0) Jul 23 17:16:21 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 256 aborted by fencing-level.fl-an-a02n01-2 'create': Configuration change | cib=0.518.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology complete=false Jul 23 17:16:21 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.518.0 for /cib/configuration/fencing-topology Jul 23 17:16:21 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 17:16:21 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:16:21 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 17:16:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-43.raw Jul 23 17:16:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.518.0 of the CIB to disk (digest: 758f2b9a65c6ec866162a4dfd732382f) Jul 23 17:16:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.KAONAI (digest: /var/lib/pacemaker/cib/cib.T42m9E) Jul 23 17:16:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:16:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.518.0 2 Jul 23 17:16:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.519.0 (null) Jul 23 17:16:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=519 Jul 23 17:16:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 17:16:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.519.0) Jul 23 17:16:22 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 256 aborted by fencing-level.fl-an-a02n01-3 'create': Configuration change | cib=0.519.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology complete=false Jul 23 17:16:22 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.519.0 for /cib/configuration/fencing-topology Jul 23 17:16:22 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 17:16:22 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:16:22 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 17:16:22 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 17:16:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-44.raw Jul 23 17:16:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.519.0 of the CIB to disk (digest: 6fb17a102adccc1229a720f239b0b811) Jul 23 17:16:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.PuDNJn (digest: /var/lib/pacemaker/cib/cib.aLsoGl) Jul 23 17:16:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 449f0c7ff46d2aa404664ec921db2dad for 0.519.0 (0x561975c6edd0 0) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.519.0 2 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.519.1 (null) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:1;33:256:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=364, @rc-code=1, @op-status=0, @exec-time=13316 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/573, version=0.519.2) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 573 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/574) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.519.2 2 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.519.3 (null) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/574, version=0.519.3) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 574 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627085793 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 256 aborted by status-2-fail-count-apc_snmp_node1_an-pdu03.start_0 doing create fail-count-apc_snmp_node1_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.519.2 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 256 aborted by status-2-last-failure-apc_snmp_node1_an-pdu03.start_0 doing create last-failure-apc_snmp_node1_an-pdu03#start_0=1627085793: Transient attribute change | cib=0.519.3 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n02 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu04 ( an-a02n02 ) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 257, saving inputs in /var/lib/pacemaker/pengine/pe-input-276.bz2 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627085793-952 is obsolete Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n01 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n02 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n02 -> an-a02n01 ) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu04 ( an-a02n02 ) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 258, saving inputs in /var/lib/pacemaker/pengine/pe-input-277.bz2 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 258 (ref=pe_calc-dc-1627085793-953) derived from /var/lib/pacemaker/pengine/pe-input-277.bz2 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 on an-a02n02 | action 11 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu04_monitor_0 on an-a02n02 | action 13 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu04_monitor_0 locally on an-a02n01 | action 12 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_get_rsc_info) info: Agent information for 'apc_snmp_node1_an-pdu04' not in cache Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_rsc_register) info: Cached agent information for 'apc_snmp_node1_an-pdu04' Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for apc_snmp_node1_an-pdu04 on an-a02n01 | transition_key=12:258:7:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu04_monitor_0 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of probe operation for apc_snmp_node1_an-pdu04 on an-a02n01: not running | rc=7 call=334 key=apc_snmp_node1_an-pdu04_monitor_0 confirmed=true cib-update=1881 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1880) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1881) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.519.3 2 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.519.4 (null) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1880, version=0.519.4) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.519.4 2 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.519.5 (null) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:7;12:258:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=334, @rc-code=7, @op-status=0 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1881, version=0.519.5) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 258 action 12 (apc_snmp_node1_an-pdu04_monitor_0 on an-a02n01) confirmed: not running | rc=7 call-id=334 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.519.5 2 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.519.6 (null) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @transition-key=11:258:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;11:258:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/546, version=0.519.6) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.519.6 2 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.519.7 (null) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/547, version=0.519.7) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.519.7 2 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.519.8 (null) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;11:258:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=365, @rc-code=0, @op-status=0 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/548, version=0.519.8) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 258 action 11 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=365 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 locally on an-a02n01 | action 34 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=34:258:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_start_0 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu03 action:start call_id:335 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.519.8 2 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.519.9 (null) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:7;13:258:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=369, @rc-code=7, @op-status=0 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/549, version=0.519.9) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 258 action 13 (apc_snmp_node1_an-pdu04_monitor_0 on an-a02n02) confirmed: not running | rc=7 call-id=369 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1882) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu04_start_0 on an-a02n02 | action 36 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.519.9 2 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.519.10 (null) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @transition-key=34:258:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;34:258:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change= Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1882, version=0.519.10) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.519.10 2 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.519.11 (null) Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_start_0, @operation=start, @transition-key=36:258:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;36:258:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1 Jul 23 17:16:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/550, version=0.519.11) Jul 23 17:16:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: d3c1ee95ba2b96d4a9046e29f7e51a5c for 0.519.11 (0x561975c6edd0 0) Jul 23 17:16:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[126609] error output [ 2021-07-23 17:16:39,645 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:16:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[126609] error output [ ] Jul 23 17:16:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[126609] error output [ ] Jul 23 17:16:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[126609] error output [ 2021-07-23 17:16:39,645 ERROR: Please use '-h' for usage ] Jul 23 17:16:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[126609] error output [ ] Jul 23 17:16:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[126609] stderr: [ 2021-07-23 17:16:39,645 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:16:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[126609] stderr: [ ] Jul 23 17:16:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[126609] stderr: [ ] Jul 23 17:16:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[126609] stderr: [ 2021-07-23 17:16:39,645 ERROR: Please use '-h' for usage ] Jul 23 17:16:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[126609] stderr: [ ] Jul 23 17:16:39 an-a02n01.alteeve.com pacemaker-fenced [121629] (internal_stonith_action_execute) info: Attempt 2 to execute fence_apc_snmp (monitor). remaining timeout is 14 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[126996] error output [ 2021-07-23 17:16:46,800 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[126996] error output [ ] Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[126996] error output [ ] Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[126996] error output [ 2021-07-23 17:16:46,801 ERROR: Please use '-h' for usage ] Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[126996] error output [ ] Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[126996] stderr: [ 2021-07-23 17:16:46,800 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[126996] stderr: [ ] Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[126996] stderr: [ ] Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[126996] stderr: [ 2021-07-23 17:16:46,801 ERROR: Please use '-h' for usage ] Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[126996] stderr: [ ] Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_remaining_timeout) info: Attempted to execute agent fence_apc_snmp (monitor) the maximum number of times (2) allowed Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_operation) notice: Operation 'monitor' [126996] using apc_snmp_node1_an-pdu03 returned -201 (Generic Pacemaker error) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu03 start (call 335) exited with status 1 (execution time 13318ms, queue time 0ms) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for apc_snmp_node1_an-pdu03 on an-a02n01: error | rc=1 call=335 key=apc_snmp_node1_an-pdu03_start_0 confirmed=true cib-update=1883 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1883) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.519.11 2 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.519.12 (null) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:1;36:258:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=370, @rc-code=1, @op-status=0, @exec-time=13316 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/575, version=0.519.14) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 575 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.519.14 2 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.519.15 (null) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=15 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/576, version=0.519.15) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/577) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 576 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627085806 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/578) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.519.15 2 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.519.16 (null) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=16 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/577, version=0.519.16) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 577 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=INFINITY Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.519.16 2 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.519.17 (null) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=17 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/578, version=0.519.17) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 578 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=1627085806 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]=1627085793 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 258 aborted by status-2-fail-count-apc_snmp_node1_an-pdu04.start_0 doing create fail-count-apc_snmp_node1_an-pdu04#start_0=INFINITY: Transient attribute change | cib=0.519.14 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 258 aborted by status-2-last-failure-apc_snmp_node1_an-pdu04.start_0 doing create last-failure-apc_snmp_node1_an-pdu04#start_0=1627085806: Transient attribute change | cib=0.519.15 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 258 aborted by status-1-fail-count-apc_snmp_node1_an-pdu03.start_0 doing create fail-count-apc_snmp_node1_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.519.16 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 258 aborted by status-1-last-failure-apc_snmp_node1_an-pdu03.start_0 doing create last-failure-apc_snmp_node1_an-pdu03#start_0=1627085806: Transient attribute change | cib=0.519.17 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n01 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n02 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n01 ) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu04 ( an-a02n02 ) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 259, saving inputs in /var/lib/pacemaker/pengine/pe-input-278.bz2 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627085806-959 is obsolete Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n01 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu03 ( an-a02n01 ) due to node availability Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu04 ( an-a02n02 -> an-a02n01 ) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 260, saving inputs in /var/lib/pacemaker/pengine/pe-input-279.bz2 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 260 (ref=pe_calc-dc-1627085806-960) derived from /var/lib/pacemaker/pengine/pe-input-279.bz2 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 locally on an-a02n01 | action 9 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=9:260:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_stop_0 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu03 action:stop call_id:336 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu04_stop_0 on an-a02n02 | action 12 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu03 stop (call 336) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1889) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1890) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01: ok | rc=0 call=336 key=apc_snmp_node1_an-pdu03_stop_0 confirmed=true cib-update=1890 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.519.17 2 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.519.18 (null) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=18 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @transition-key=9:260:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;9:260:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1627 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1889, version=0.519.18) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.519.18 2 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.519.19 (null) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=19 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;9:260:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=336, @rc-code=0, @op-status=0 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1890, version=0.519.19) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 260 action 9 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=336 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.519.19 2 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.519.20 (null) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=20 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_stop_0, @operation=stop, @transition-key=12:260:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;12:260:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/552, version=0.519.20) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.519.20 2 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.519.21 (null) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=21 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:0;12:260:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=371, @rc-code=0, @op-status=0 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/553, version=0.519.21) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 260 action 12 (apc_snmp_node1_an-pdu04_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=371 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu04_start_0 locally on an-a02n01 | action 33 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node1_an-pdu04 on an-a02n01 | transition_key=33:260:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu04_start_0 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1891) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu04 action:start call_id:337 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.519.21 2 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.519.22 (null) Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=22 Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_start_0, @operation=start, @transition-key=33:260:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;33:260:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change= Jul 23 17:16:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1891, version=0.519.22) Jul 23 17:16:51 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 0cc6975efa26d101a4b9729deeb776ba for 0.519.22 (0x561975c6edd0 0) Jul 23 17:16:53 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[127878] error output [ 2021-07-23 17:16:53,010 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 17:16:53 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[127878] error output [ ] Jul 23 17:16:53 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[127878] error output [ ] Jul 23 17:16:53 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[127878] error output [ 2021-07-23 17:16:53,010 ERROR: Please use '-h' for usage ] Jul 23 17:16:53 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[127878] error output [ ] Jul 23 17:16:53 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[127878] stderr: [ 2021-07-23 17:16:53,010 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 17:16:53 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[127878] stderr: [ ] Jul 23 17:16:53 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[127878] stderr: [ ] Jul 23 17:16:53 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[127878] stderr: [ 2021-07-23 17:16:53,010 ERROR: Please use '-h' for usage ] Jul 23 17:16:53 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[127878] stderr: [ ] Jul 23 17:16:53 an-a02n01.alteeve.com pacemaker-fenced [121629] (internal_stonith_action_execute) info: Attempt 2 to execute fence_apc_snmp (monitor). remaining timeout is 13 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[128316] error output [ 2021-07-23 17:17:00,169 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[128316] error output [ ] Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[128316] error output [ ] Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[128316] error output [ 2021-07-23 17:17:00,169 ERROR: Please use '-h' for usage ] Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[128316] error output [ ] Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[128316] stderr: [ 2021-07-23 17:17:00,169 ERROR: Returned 1: Timeout: No Response from 10.201.2.2:161 ] Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[128316] stderr: [ ] Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[128316] stderr: [ ] Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[128316] stderr: [ 2021-07-23 17:17:00,169 ERROR: Please use '-h' for usage ] Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[128316] stderr: [ ] Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_remaining_timeout) info: Attempted to execute agent fence_apc_snmp (monitor) the maximum number of times (2) allowed Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_operation) notice: Operation 'monitor' [128316] using apc_snmp_node1_an-pdu04 returned -201 (Generic Pacemaker error) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu04 start (call 337) exited with status 1 (execution time 13318ms, queue time 0ms) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for apc_snmp_node1_an-pdu04 on an-a02n01: error | rc=1 call=337 key=apc_snmp_node1_an-pdu04_start_0 confirmed=true cib-update=1892 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1892) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.519.22 2 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.519.23 (null) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=23 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:1;33:260:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=337, @rc-code=1, @op-status=0, @exec-time=13318 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/579, version=0.519.24) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 579 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=INFINITY Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=INFINITY Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.519.24 2 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.519.25 (null) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=25 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/580, version=0.519.25) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 580 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=1627085820 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n02]=1627085806 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 260 aborted by status-1-fail-count-apc_snmp_node1_an-pdu04.start_0 doing create fail-count-apc_snmp_node1_an-pdu04#start_0=INFINITY: Transient attribute change | cib=0.519.24 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 260 aborted by status-1-last-failure-apc_snmp_node1_an-pdu04.start_0 doing create last-failure-apc_snmp_node1_an-pdu04#start_0=1627085820: Transient attribute change | cib=0.519.25 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 17:16:46 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 17:16:46 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n01 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu04 ( an-a02n01 ) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 261, saving inputs in /var/lib/pacemaker/pengine/pe-input-280.bz2 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627085820-964 is obsolete Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 17:16:46 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 17:16:46 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_0 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu04 ( an-a02n01 ) due to node availability Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 262, saving inputs in /var/lib/pacemaker/pengine/pe-input-281.bz2 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 262 (ref=pe_calc-dc-1627085820-965) derived from /var/lib/pacemaker/pengine/pe-input-281.bz2 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu04_stop_0 locally on an-a02n01 | action 9 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node1_an-pdu04 on an-a02n01 | transition_key=9:262:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu04_stop_0 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu04 action:stop call_id:338 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu04 stop (call 338) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node1_an-pdu04 on an-a02n01: ok | rc=0 call=338 key=apc_snmp_node1_an-pdu04_stop_0 confirmed=true cib-update=1897 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1896) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1897) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.519.25 2 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.519.26 (null) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=26 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @operation_key=apc_snmp_node1_an-pdu04_stop_0, @operation=stop, @transition-key=9:262:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;9:262:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1627 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1896, version=0.519.26) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.519.26 2 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.519.27 (null) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=27 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:0;9:262:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=338, @rc-code=0, @op-status=0 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1897, version=0.519.27) Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 262 action 9 (apc_snmp_node1_an-pdu04_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=338 Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 262 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-281.bz2): Complete Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:17:00 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:17:05 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: ea2a7c9e84017a15af2ef032ba1b02a3 for 0.519.27 (0x561975c6edd0 0) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.519.27 2 Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.520.0 a21364f159aaccbbc48becb7db495d2d Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3'] Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=520, @num_updates=0 Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.520.0) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 262 aborted by deletion of fencing-level[@id='fl-an-a02n01-3']: Configuration change | cib=0.520.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3'] complete=true Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.520.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3'] Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-45.raw Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.520.0 of the CIB to disk (digest: 4af6f435f342566ed14f3ea79dc0c7cc) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.ATgmbL (digest: /var/lib/pacemaker/cib/cib.49XzrJ) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 17:16:46 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 263, saving inputs in /var/lib/pacemaker/pengine/pe-input-282.bz2 Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 263 (ref=pe_calc-dc-1627085854-967) derived from /var/lib/pacemaker/pengine/pe-input-282.bz2 Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 263 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-282.bz2): Complete Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:17:34 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.520.0 2 Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.521.0 (null) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=521 Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.521.0) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 263 aborted by deletion of fencing-level[@id='fl-an-a02n01-2']: Configuration change | cib=0.521.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] complete=true Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.521.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-46.raw Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.521.0 of the CIB to disk (digest: 96ba9bc47e7b3b8d96410f8ed483ac27) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.HoDAm8 (digest: /var/lib/pacemaker/cib/cib.dJ2nea) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 17:16:46 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 264, saving inputs in /var/lib/pacemaker/pengine/pe-input-283.bz2 Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 264 (ref=pe_calc-dc-1627085858-968) derived from /var/lib/pacemaker/pengine/pe-input-283.bz2 Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 264 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-283.bz2): Complete Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:17:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.521.0 2 Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.522.0 (null) Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=522 Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.522.0) Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 264 aborted by deletion of fencing-level[@id='fl-an-a02n01-1']: Configuration change | cib=0.522.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] complete=true Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.522.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-47.raw Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.522.0 of the CIB to disk (digest: 479e70e00e35d8f12ae8cb093306b650) Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.JpTpQb (digest: /var/lib/pacemaker/cib/cib.KKznoc) Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n02 is active Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is online Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 17:16:46 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n02) Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n02) Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 265, saving inputs in /var/lib/pacemaker/pengine/pe-input-284.bz2 Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 265 (ref=pe_calc-dc-1627085861-969) derived from /var/lib/pacemaker/pengine/pe-input-284.bz2 Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 265 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-284.bz2): Complete Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:17:41 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:17:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: d14ac994e9f415ad459554d6ee7b7b93 for 0.522.0 (0x5619762e6e20 0) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-controld [121633] (pcmk__update_peer_expected) info: handle_request: Node an-a02n02[2] - expected state is now down (was member) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_shutdown_request) info: Creating shutdown request for an-a02n02 (state=S_IDLE) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting shutdown[an-a02n02]: (unset) -> 1627085902 | from an-a02n01 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 581 with 1 change for shutdown (id n/a, set n/a) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/581) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.522.0 2 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.522.1 (null) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/581, version=0.522.1) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 581 result for shutdown: OK | rc=0 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * shutdown[an-a02n02]=1627085902 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 265 aborted by status-2-shutdown doing create shutdown=1627085902: Transient attribute change | cib=0.522.1 source=abort_unless_down:313 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2'] complete=true Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n02 is shutting down Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 17:16:46 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n02 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n02 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for delay_node2 on an-a02n01 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for ipmilan_node1 on an-a02n01 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (sched_shutdown_op) notice: Scheduling shutdown of node an-a02n02 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (node_action) notice: * Shutdown an-a02n02 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Move delay_node2 ( an-a02n02 -> an-a02n01 ) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Move ipmilan_node1 ( an-a02n02 -> an-a02n01 ) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 266, saving inputs in /var/lib/pacemaker/pengine/pe-input-285.bz2 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 266 (ref=pe_calc-dc-1627085902-970) derived from /var/lib/pacemaker/pengine/pe-input-285.bz2 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation delay_node2_stop_0 on an-a02n02 | action 13 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation ipmilan_node1_stop_0 on an-a02n02 | action 16 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.522.1 2 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.522.2 (null) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='delay_node2']/lrm_rsc_op[@id='delay_node2_last_0']: @operation_key=delay_node2_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=13:266:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;13:266:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change= Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/554, version=0.522.2) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.522.2 2 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.522.3 (null) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ipmilan_node1']/lrm_rsc_op[@id='ipmilan_node1_last_0']: @operation_key=ipmilan_node1_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=16:266:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;16:266:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-c Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/555, version=0.522.3) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.522.3 2 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.522.4 (null) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='delay_node2']/lrm_rsc_op[@id='delay_node2_last_0']: @transition-magic=0:0;13:266:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=373, @rc-code=0, @op-status=0 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/556, version=0.522.4) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 266 action 13 (delay_node2_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=373 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation delay_node2_start_0 locally on an-a02n01 | action 14 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for delay_node2 on an-a02n01 | transition_key=14:266:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=delay_node2_start_0 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.522.4 2 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.522.5 (null) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ipmilan_node1']/lrm_rsc_op[@id='ipmilan_node1_last_0']: @transition-magic=0:0;16:266:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=375, @rc-code=0, @op-status=0 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n02/crmd/557, version=0.522.5) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:delay_node2 action:start call_id:339 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1902) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 266 action 16 (ipmilan_node1_stop_0 on an-a02n02) confirmed: ok | rc=0 call-id=375 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation ipmilan_node1_start_0 locally on an-a02n01 | action 17 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for ipmilan_node1 on an-a02n01 | transition_key=17:266:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=ipmilan_node1_start_0 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1903) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.522.5 2 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.522.6 (null) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='delay_node2']/lrm_rsc_op[@id='delay_node2_last_0']: @operation_key=delay_node2_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=14:266:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;14:266:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-chang Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:ipmilan_node1 action:start call_id:340 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-controld [121633] (te_crm_command) info: Executing crm-event (33) without waiting: do_shutdown on an-a02n02 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1902, version=0.522.6) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.522.6 2 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.522.7 (null) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ipmilan_node1']/lrm_rsc_op[@id='ipmilan_node1_last_0']: @operation_key=ipmilan_node1_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=17:266:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;17:266:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1903, version=0.522.7) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-controld [121633] (pcmk_cpg_membership) info: Group crmd event 4: an-a02n02 (node 2 pid 3296860) left via cpg_leave Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-controld [121633] (crm_update_peer_proc) info: pcmk_cpg_membership: Node an-a02n02[2] - corosync-cpg is now offline Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-controld [121633] (peer_update_callback) info: Node an-a02n02 is no longer a peer | DC=true old=0x4000000 new=0x0000000 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting transient attributes for node an-a02n02 (via CIB call 1904) | xpath=//node_state[@uname='an-a02n02']/transient_attributes Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-controld [121633] (peer_update_callback) info: do_shutdown of peer an-a02n02 is in progress | action=33 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-controld [121633] (pcmk_cpg_membership) info: Group crmd event 4: an-a02n01 (node 1 pid 121633) is member Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n02']/transient_attributes to all (origin=local/crmd/1904) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1905) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.522.7 2 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.522.8 (null) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2'] Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n02']/transient_attributes: OK (rc=0, origin=an-a02n01/crmd/1904, version=0.522.8) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.522.8 2 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.522.9 (null) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crmd=offline, @crm-debug-origin=peer_update_callback, @join=down, @expected=down Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1905, version=0.522.9) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-attrd [121631] (pcmk_cpg_membership) info: Group attrd event 4: an-a02n02 (node 2 pid 3296858) left for unknown reason Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-attrd [121631] (crm_update_peer_proc) info: pcmk_cpg_membership: Node an-a02n02[2] - corosync-cpg is now offline Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-attrd [121631] (update_peer_state_iter) notice: Node an-a02n02 state is now lost | nodeid=2 previous=member source=crm_update_peer_proc Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_remove) notice: Removing all an-a02n02 attributes for peer loss Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-attrd [121631] (crm_reap_dead_member) info: Removing node with name an-a02n02 and id 2 from membership cache Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-attrd [121631] (reap_crm_member) notice: Purged 1 peer with id=2 and/or uname=an-a02n02 from the membership cache Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-attrd [121631] (pcmk_cpg_membership) info: Group attrd event 4: an-a02n01 (node 1 pid 121631) is member Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: delay_node2 start (call 339) exited with status 0 (execution time 9ms, queue time 0ms) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_shutdown_req) info: Peer an-a02n02 is requesting to shut down Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-fenced [121629] (pcmk_cpg_membership) info: Group stonith-ng event 4: an-a02n02 (node 2 pid 3296856) left for unknown reason Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-fenced [121629] (crm_update_peer_proc) info: pcmk_cpg_membership: Node an-a02n02[2] - corosync-cpg is now offline Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_peer_state_iter) notice: Node an-a02n02 state is now lost | nodeid=2 previous=member source=crm_update_peer_proc Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-fenced [121629] (crm_reap_dead_member) info: Removing node with name an-a02n02 and id 2 from membership cache Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-fenced [121629] (reap_crm_member) notice: Purged 1 peer with id=2 and/or uname=an-a02n02 from the membership cache Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-fenced [121629] (pcmk_cpg_membership) info: Group stonith-ng event 4: an-a02n01 (node 1 pid 121629) is member Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (pcmk_cpg_membership) info: Group cib event 4: an-a02n02 (node 2 pid 3296855) left via cpg_leave Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (crm_update_peer_proc) info: pcmk_cpg_membership: Node an-a02n02[2] - corosync-cpg is now offline Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (update_peer_state_iter) notice: Node an-a02n02 state is now lost | nodeid=2 previous=member source=crm_update_peer_proc Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (crm_reap_dead_member) info: Removing node with name an-a02n02 and id 2 from membership cache Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (reap_crm_member) notice: Purged 1 peer with id=2 and/or uname=an-a02n02 from the membership cache Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (pcmk_cpg_membership) info: Group cib event 4: an-a02n01 (node 1 pid 121628) is member Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[132885] exited with status 0 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for delay_node2 on an-a02n01: ok | rc=0 call=339 key=delay_node2_start_0 confirmed=true cib-update=1906 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1906) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.522.9 2 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.522.10 (null) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='delay_node2']/lrm_rsc_op[@id='delay_node2_last_0']: @transition-magic=0:0;14:266:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=339, @rc-code=0, @op-status=0, @exec-time=9 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1906, version=0.522.10) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 266 action 14 (delay_node2_start_0 on an-a02n01) confirmed: ok | rc=0 call-id=339 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation delay_node2_monitor_60000 locally on an-a02n01 | action 15 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of monitor operation for delay_node2 on an-a02n01 | transition_key=15:266:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=delay_node2_monitor_60000 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1907) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.522.10 2 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.522.11 (null) Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 17:18:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='delay_node2']: S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:18:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 30540a3cce7cdcc1b399987ca2f7443d for 0.522.16 (0x56197618e260 0) Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.522.16 2 Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.523.0 bb1681fdc3fde7f5332ab2b9362bc27f Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=523, @num_updates=0 Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03']: Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.523.0) Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 266 aborted by meta_attributes.apc_snmp_node1_an-pdu03-meta_attributes 'create': Configuration change | cib=0.523.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03'] complete=true Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create primitive[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.523.0 Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-48.raw Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.523.0 of the CIB to disk (digest: 25935eb437c7fbabb02939f3196fbb84) Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.HUgoyV (digest: /var/lib/pacemaker/cib/cib.GJRljV) Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu03 has been disabled Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 17:16:46 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n01) Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 267, saving inputs in /var/lib/pacemaker/pengine/pe-input-286.bz2 Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 267 (ref=pe_calc-dc-1627086080-979) derived from /var/lib/pacemaker/pengine/pe-input-286.bz2 Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 267 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-286.bz2): Complete Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:21:20 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:21:25 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: bb1681fdc3fde7f5332ab2b9362bc27f for 0.523.0 (0x561976131cd0 0) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/crmd/1918) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 582 with 1 change for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: 1627085806 -> (unset) | from an-a02n01 Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 583 with 1 change for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.523.0 2 Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.523.1 (null) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/1918, version=0.523.0) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-controld [121633] (delete_resource) info: Removing resource apc_snmp_node1_an-pdu03 from executor for e4302b71-5eb7-4995-baa3-da71eefff76d as root Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_resource_history) info: Clearing resource history for apc_snmp_node1_an-pdu03 on an-a02n01 (via CIB call 1919) | xpath=//node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/582) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-controld [121633] (notify_deleted) info: Notifying e4302b71-5eb7-4995-baa3-da71eefff76d on an-a02n01 that apc_snmp_node1_an-pdu03 was deleted Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/583) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.523.0 2 Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.523.1 (null) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/582, version=0.523.1) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 582 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.523.1 2 Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.523.2 (null) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0'] Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/583, version=0.523.2) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 583 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/crmd/1919) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 267 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.523.1 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 267 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.523.2 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.523.2 2 Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.523.3 (null) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/1919, version=0.523.3) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1921) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 267 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu03']: Resource state removal | cib=0.523.3 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] complete=true Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.523.3 2 Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.524.0 (null) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=524, @num_updates=0 Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']: @value=1627086086 Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1921, version=0.524.0) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-49.raw Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.524.0 of the CIB to disk (digest: 81ec77748949508005590e2e1fac14ae) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.X39NLS (digest: /var/lib/pacemaker/cib/cib.lmswVR) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 267 aborted by cib-bootstrap-options-last-lrm-refresh doing modify last-lrm-refresh=1627086086: Configuration change | cib=0.524.0 source=te_update_diff_v2:474 path=/cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh'] complete=true Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 17:16:46 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n01) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:21:26 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 268, saving inputs in /var/lib/pacemaker/pengine/pe-input-287.bz2 Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627086086-981 is obsolete Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 17:16:46 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n01) Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 269, saving inputs in /var/lib/pacemaker/pengine/pe-input-287.bz2 Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (handle_pecalc_op) info: Input has not changed since last time, not saving to disk Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 269 (ref=pe_calc-dc-1627086087-982) derived from /var/lib/pacemaker/pengine/pe-input-287.bz2 Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu03_monitor_0 locally on an-a02n01 | action 11 Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_get_rsc_info) info: Agent information for 'apc_snmp_node1_an-pdu03' not in cache Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_rsc_register) info: Cached agent information for 'apc_snmp_node1_an-pdu03' Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=11:269:7:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_monitor_0 Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of probe operation for apc_snmp_node1_an-pdu03 on an-a02n01: not running | rc=7 call=347 key=apc_snmp_node1_an-pdu03_monitor_0 confirmed=true cib-update=1929 Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1928) Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1929) Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.524.0 2 Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.524.1 (null) Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1928, version=0.524.1) Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.524.1 2 Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.524.2 (null) Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:7;11:269:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=347, @rc-code=7, @op-status=0 Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1929, version=0.524.2) Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 269 action 11 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n01) confirmed: not running | rc=7 call-id=347 Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 269 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-287.bz2): Complete Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:21:27 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:21:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: febe30be33e875214f99445624dca409 for 0.524.2 (0x5619760ad450 0) Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.524.2 2 Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.525.0 (null) Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=525, @num_updates=0 Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu04']: Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.525.0) Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 269 aborted by meta_attributes.apc_snmp_node1_an-pdu04-meta_attributes 'create': Configuration change | cib=0.525.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu04'] complete=true Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create primitive[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.525.0 Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu03 has been disabled Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu04 has been disabled Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-50.raw Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.525.0 of the CIB to disk (digest: e0b962ad9eb818a5a5ff938785e19a8c) Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.uZYk3F (digest: /var/lib/pacemaker/cib/cib.QItc0D) Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n01 at Jul 23 17:16:46 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu04 has failed INFINITY times on an-a02n01 Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n01) Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 270, saving inputs in /var/lib/pacemaker/pengine/pe-input-288.bz2 Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 270 (ref=pe_calc-dc-1627086098-984) derived from /var/lib/pacemaker/pengine/pe-input-288.bz2 Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 270 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-288.bz2): Complete Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:21:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 584 with 1 change for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]: 1627085820 -> (unset) | from an-a02n01 Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 585 with 1 change for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] to all (origin=local/crmd/1931) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/584) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/585) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.525.0 2 Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.525.1 (null) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n01/crmd/1931, version=0.525.0) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-controld [121633] (delete_resource) info: Removing resource apc_snmp_node1_an-pdu04 from executor for 0a440157-8c9f-4ce3-a9af-a332316078a0 as root Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_resource_history) info: Clearing resource history for apc_snmp_node1_an-pdu04 on an-a02n01 (via CIB call 1932) | xpath=//node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-controld [121633] (notify_deleted) info: Notifying 0a440157-8c9f-4ce3-a9af-a332316078a0 on an-a02n01 that apc_snmp_node1_an-pdu04 was deleted Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.525.0 2 Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.525.1 (null) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu04.start_0'] Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/584, version=0.525.1) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 584 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.525.1 2 Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.525.2 (null) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu04.start_0'] Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/585, version=0.525.2) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 585 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] to all (origin=local/crmd/1932) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 270 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.525.1 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu04.start_0'] complete=true Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 270 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu04.start_0']: Transient attribute change | cib=0.525.2 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu04.start_0'] complete=true Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.525.2 2 Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.525.3 (null) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n01/crmd/1932, version=0.525.3) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1934) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 270 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu04']: Resource state removal | cib=0.525.3 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04'] complete=true Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.525.3 2 Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.526.0 (null) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=526, @num_updates=0 Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']: @value=1627086101 Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1934, version=0.526.0) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-51.raw Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.526.0 of the CIB to disk (digest: 68783fc14e2d1315511c9a0972143ab0) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.Dm9kJa (digest: /var/lib/pacemaker/cib/cib.dvgouc) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 270 aborted by cib-bootstrap-options-last-lrm-refresh doing modify last-lrm-refresh=1627086101: Configuration change | cib=0.526.0 source=te_update_diff_v2:474 path=/cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh'] complete=true Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n01) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:21:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 271, saving inputs in /var/lib/pacemaker/pengine/pe-input-289.bz2 Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627086101-986 is obsolete Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n01) Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 272, saving inputs in /var/lib/pacemaker/pengine/pe-input-289.bz2 Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (handle_pecalc_op) info: Input has not changed since last time, not saving to disk Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 272 (ref=pe_calc-dc-1627086102-987) derived from /var/lib/pacemaker/pengine/pe-input-289.bz2 Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu04_monitor_0 locally on an-a02n01 | action 11 Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_get_rsc_info) info: Agent information for 'apc_snmp_node1_an-pdu04' not in cache Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_rsc_register) info: Cached agent information for 'apc_snmp_node1_an-pdu04' Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for apc_snmp_node1_an-pdu04 on an-a02n01 | transition_key=11:272:7:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu04_monitor_0 Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1941) Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of probe operation for apc_snmp_node1_an-pdu04 on an-a02n01: not running | rc=7 call=352 key=apc_snmp_node1_an-pdu04_monitor_0 confirmed=true cib-update=1942 Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1942) Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.526.0 2 Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.526.1 (null) Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1941, version=0.526.1) Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.526.1 2 Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.526.2 (null) Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu04']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu04_last_0']: @transition-magic=0:7;11:272:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=352, @rc-code=7, @op-status=0 Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1942, version=0.526.2) Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 272 action 11 (apc_snmp_node1_an-pdu04_monitor_0 on an-a02n01) confirmed: not running | rc=7 call-id=352 Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 272 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-289.bz2): Complete Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:21:42 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:21:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: a61482bedb51fc91342f80ee5abb4f8a for 0.526.2 (0x561975c274d0 0) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.526.2 2 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.527.0 92897d1d5efb33f68a8507538038a2af Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=527, @num_updates=0 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.526.2 -> 0.527.0 from an-a02n01 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.527.0) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 272 aborted by deletion of fencing-topology: Configuration change | cib=0.527.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology complete=true Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1945) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1946) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 586 with 1 change for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 587 with 1 change for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 1 private change for #attrd-protocol, id=n/a, set=n/a Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 588 with 1 change for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 589 with 1 change for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 590 with 1 change for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 591 with 1 change for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 592 with 1 change for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 593 with 1 change for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level delete operation 0.527.0 for /cib/configuration/fencing-topology Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-52.raw Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1945, version=0.527.0) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.527.0 2 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.527.1 (null) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.527.0 of the CIB to disk (digest: d8100577c74fc19ad8c1fea52e6a70b4) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1946, version=0.527.1) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.J6kvrc (digest: /var/lib/pacemaker/cib/cib.U7lfL8) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/586) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/587) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/588) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/589) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/590) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.527.1 2 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.527.2 (null) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/586, version=0.527.2) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 586 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.527.2 2 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.527.3 (null) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/587, version=0.527.3) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 587 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.527.3 2 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.527.4 (null) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/588, version=0.527.4) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 588 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.527.4 2 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.527.5 (null) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/589, version=0.527.5) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 589 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.527.5 2 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.527.6 (null) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/590, version=0.527.6) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 590 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/591) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/592) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/593) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.527.6 2 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.527.7 (null) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/591, version=0.527.7) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 591 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.527.7 2 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.527.8 (null) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/592, version=0.527.8) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 592 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.527.8 2 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.527.9 (null) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/593, version=0.527.9) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 593 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/1948, version=0.527.9) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1949) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1949, version=0.527.9) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1951) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1951, version=0.527.9) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1953) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Not making join-39 offer to inactive node an-a02n02 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1955) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Making join-39 offers based on membership event 263 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-39 offer to an-a02n01 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-39 requests from 1 outstanding node Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1953, version=0.527.9) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1955, version=0.527.9) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: 506cb2075ddb2b384b22fa8cf075c287 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.527.9 with 0.527.9 from an-a02n01 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/1958, version=0.527.9) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1959) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1959, version=0.527.9) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[146785] exited with status 0 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-53.raw Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.527.0 of the CIB to disk (digest: d8100577c74fc19ad8c1fea52e6a70b4) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.fHZJxr (digest: /var/lib/pacemaker/cib/cib.sMOUou) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[146787] exited with status 0 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/cibadmin/2) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.527.9 2 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.528.0 (null) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=528, @num_updates=0 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.528.0) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Device 'apc_snmp_node1_an-pdu03' not found (4 active devices) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-54.raw Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.528.0 of the CIB to disk (digest: ab58af40eda9af1c7f8ed03a760a9581) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.lPcyoQ (digest: /var/lib/pacemaker/cib/cib.Xl2dlS) Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[146790] exited with status 0 Jul 23 17:23:28 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[146792] exited with status 0 Jul 23 17:23:29 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[146793] exited with status 0 Jul 23 17:23:29 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[146794] exited with status 0 Jul 23 17:23:29 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[146796] exited with status 0 Jul 23 17:23:29 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[146798] exited with status 0 Jul 23 17:23:29 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[146799] exited with status 0 Jul 23 17:23:30 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[146800] exited with status 0 Jul 23 17:23:30 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 1960) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 17:23:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/1960) Jul 23 17:23:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.528.0 2 Jul 23 17:23:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.528.1 (null) Jul 23 17:23:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 17:23:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:23:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/1960, version=0.528.1) Jul 23 17:23:30 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[146804] exited with status 0 Jul 23 17:23:30 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[146806] exited with status 0 Jul 23 17:23:30 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[146807] exited with status 0 Jul 23 17:23:30 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[146823] exited with status 0 Jul 23 17:23:31 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[146824] exited with status 0 Jul 23 17:23:31 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[146825] exited with status 0 Jul 23 17:23:31 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[146827] exited with status 0 Jul 23 17:23:31 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[146829] exited with status 0 Jul 23 17:23:31 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[146831] exited with status 0 Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[146856] exited with status 0 Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1961) Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.528.1 2 Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.528.2 (null) Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1961, version=0.528.2) Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 272 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1964) Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1965) Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1966) Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/1964, version=0.528.2) Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.528.2 2 Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.528.3 (null) Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1965, version=0.528.3) Jul 23 17:23:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/1966, version=0.528.3) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n02 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now f2317cad3d54cec5d7d7aa7d0bf35cf8 (reload:3.10.2) 0:1;33:256:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (custom_action) warning: apc_snmp_node1_an-pdu03_clear_failcount_0 on an-a02n02 is unrunnable (node is offline) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node1_an-pdu03 from an-a02n01 Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n01) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 273, saving inputs in /var/lib/pacemaker/pengine/pe-input-290.bz2 Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 273 (ref=pe_calc-dc-1627086213-994) derived from /var/lib/pacemaker/pengine/pe-input-290.bz2 Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating delete operation apc_snmp_node1_an-pdu03_delete_0 locally on an-a02n01 | action 12 Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/crmd/1969) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.528.3 2 Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.528.4 (null) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/1969, version=0.528.3) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-controld [121633] (delete_resource) info: Removing resource apc_snmp_node1_an-pdu03 from executor for tengine Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_resource_history) info: Clearing resource history for apc_snmp_node1_an-pdu03 on an-a02n01 (via CIB call 1970) | xpath=//node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-controld [121633] (notify_deleted) info: Notifying tengine on localhost that apc_snmp_node1_an-pdu03 was deleted Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 273 action 12 (apc_snmp_node1_an-pdu03_delete_0 on an-a02n01) confirmed: ok | rc=0 call-id=0 Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 273 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-290.bz2): Complete Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/crmd/1970) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.528.3 2 Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.528.4 (null) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/1970, version=0.528.4) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 273 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu03']: Resource state removal | cib=0.528.4 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] complete=true Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n02 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now f2317cad3d54cec5d7d7aa7d0bf35cf8 (reload:3.10.2) 0:1;33:256:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (custom_action) warning: apc_snmp_node1_an-pdu03_clear_failcount_0 on an-a02n02 is unrunnable (node is offline) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n01) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 274, saving inputs in /var/lib/pacemaker/pengine/pe-input-291.bz2 Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 274 (ref=pe_calc-dc-1627086213-997) derived from /var/lib/pacemaker/pengine/pe-input-291.bz2 Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 274 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-291.bz2): Complete Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:23:33 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:23:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: b0aa1bf327f9614fb36b55027ccc8e5b for 0.528.4 (0x56197608b400 0) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.528.4 2 Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.529.0 (null) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=529, @num_updates=0 Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources: Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.529.0) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 274 aborted by primitive.apc_snmp_node1_an-pdu03 'create': Configuration change | cib=0.529.0 source=te_update_diff_v2:474 path=/cib/configuration/resources complete=true Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create resources Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.529.0 Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu04 has been disabled Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node1_an-pdu03' to device list (5 active devices) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-55.raw Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.529.0 of the CIB to disk (digest: f7e3f313d486d99e2dddb3641b59e72c) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.30pIJT (digest: /var/lib/pacemaker/cib/cib.M2nHMS) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n01 Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n01) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu03 ( an-a02n01 ) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 275, saving inputs in /var/lib/pacemaker/pengine/pe-input-292.bz2 Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 275 (ref=pe_calc-dc-1627086242-998) derived from /var/lib/pacemaker/pengine/pe-input-292.bz2 Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu03_monitor_0 locally on an-a02n01 | action 11 Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_get_rsc_info) info: Agent information for 'apc_snmp_node1_an-pdu03' not in cache Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_rsc_register) info: Cached agent information for 'apc_snmp_node1_an-pdu03' Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=11:275:7:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_monitor_0 Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1973) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of probe operation for apc_snmp_node1_an-pdu03 on an-a02n01: not running | rc=7 call=357 key=apc_snmp_node1_an-pdu03_monitor_0 confirmed=true cib-update=1974 Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1974) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.529.0 2 Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.529.1 (null) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1973, version=0.529.1) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.529.1 2 Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.529.2 (null) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:7;11:275:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=357, @rc-code=7, @op-status=0 Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1974, version=0.529.2) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 275 action 11 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n01) confirmed: not running | rc=7 call-id=357 Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 locally on an-a02n01 | action 32 Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of start operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=32:275:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_start_0 Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1975) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu03 action:start call_id:358 Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.529.2 2 Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.529.3 (null) Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_start_0, @operation=start, @transition-key=32:275:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;32:275:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1 Jul 23 17:24:02 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1975, version=0.529.3) Jul 23 17:24:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:24:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.529.3 2 Jul 23 17:24:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.530.0 (null) Jul 23 17:24:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=530, @num_updates=0 Jul 23 17:24:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration: Jul 23 17:24:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:24:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 17:24:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.530.0) Jul 23 17:24:03 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.530.0 for /cib/configuration. Jul 23 17:24:03 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 17:24:03 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:24:03 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) notice: Transition 275 aborted by fencing-topology. 'create': Configuration change | cib=0.530.0 source=te_update_diff_v2:474 path=/cib/configuration complete=false Jul 23 17:24:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-56.raw Jul 23 17:24:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.530.0 of the CIB to disk (digest: 0730f84ed91dd54f66e060e8f8db74d7) Jul 23 17:24:03 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.2hOxwY (digest: /var/lib/pacemaker/cib/cib.86ZeNX) Jul 23 17:24:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:24:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.530.0 2 Jul 23 17:24:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.531.0 (null) Jul 23 17:24:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=531 Jul 23 17:24:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 17:24:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.531.0) Jul 23 17:24:04 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 275 aborted by fencing-level.fl-an-a02n01-2 'create': Configuration change | cib=0.531.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology complete=false Jul 23 17:24:04 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.531.0 for /cib/configuration/fencing-topology Jul 23 17:24:04 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 17:24:04 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:24:04 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 17:24:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-57.raw Jul 23 17:24:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.531.0 of the CIB to disk (digest: 1883af2085fae12db34e35b847a576ab) Jul 23 17:24:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.HPH8GF (digest: /var/lib/pacemaker/cib/cib.TwWz4E) Jul 23 17:24:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 17:24:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.531.0 2 Jul 23 17:24:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.532.0 (null) Jul 23 17:24:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=532 Jul 23 17:24:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 17:24:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.532.0) Jul 23 17:24:04 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 275 aborted by fencing-level.fl-an-a02n01-3 'create': Configuration change | cib=0.532.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology complete=false Jul 23 17:24:04 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.532.0 for /cib/configuration/fencing-topology Jul 23 17:24:04 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 17:24:04 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 17:24:04 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 17:24:04 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 17:24:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-58.raw Jul 23 17:24:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.532.0 of the CIB to disk (digest: 84717d04de436a38c5af9f7566ba9c24) Jul 23 17:24:04 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.ZCvWnm (digest: /var/lib/pacemaker/cib/cib.JIuubm) Jul 23 17:24:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[148697] error output [ 2021-07-23 17:24:09,022 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:24:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[148697] error output [ ] Jul 23 17:24:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[148697] error output [ ] Jul 23 17:24:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[148697] error output [ 2021-07-23 17:24:09,023 ERROR: Please use '-h' for usage ] Jul 23 17:24:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_1[148697] error output [ ] Jul 23 17:24:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[148697] stderr: [ 2021-07-23 17:24:09,022 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:24:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[148697] stderr: [ ] Jul 23 17:24:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[148697] stderr: [ ] Jul 23 17:24:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[148697] stderr: [ 2021-07-23 17:24:09,023 ERROR: Please use '-h' for usage ] Jul 23 17:24:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[148697] stderr: [ ] Jul 23 17:24:09 an-a02n01.alteeve.com pacemaker-fenced [121629] (internal_stonith_action_execute) info: Attempt 2 to execute fence_apc_snmp (monitor). remaining timeout is 13 Jul 23 17:24:09 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 97eae33f3261aa6bcae6c31712f9dbcc for 0.532.0 (0x5619764caa40 0) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[149393] error output [ 2021-07-23 17:24:16,184 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[149393] error output [ ] Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[149393] error output [ ] Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[149393] error output [ 2021-07-23 17:24:16,184 ERROR: Please use '-h' for usage ] Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_op_output) notice: fence_apc_snmp_monitor_2[149393] error output [ ] Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[149393] stderr: [ 2021-07-23 17:24:16,184 ERROR: Returned 1: Timeout: No Response from 10.201.2.1:161 ] Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[149393] stderr: [ ] Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[149393] stderr: [ ] Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[149393] stderr: [ 2021-07-23 17:24:16,184 ERROR: Please use '-h' for usage ] Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_action) warning: fence_apc_snmp[149393] stderr: [ ] Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_remaining_timeout) info: Attempted to execute agent fence_apc_snmp (monitor) the maximum number of times (2) allowed Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-fenced [121629] (log_operation) notice: Operation 'monitor' [149393] using apc_snmp_node1_an-pdu03 returned -201 (Generic Pacemaker error) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu03 start (call 358) exited with status 1 (execution time 13323ms, queue time 0ms) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of start operation for apc_snmp_node1_an-pdu03 on an-a02n01: error | rc=1 call=358 key=apc_snmp_node1_an-pdu03_start_0 confirmed=true cib-update=1976 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1976) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.532.0 2 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.532.1 (null) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:1;32:275:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=358, @rc-code=1, @op-status=0, @exec-time=13323 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/595, version=0.532.2) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 595 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=INFINITY Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.532.2 2 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.532.3 (null) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/596, version=0.532.3) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 596 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=1627086256 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 275 aborted by status-1-fail-count-apc_snmp_node1_an-pdu03.start_0 doing create fail-count-apc_snmp_node1_an-pdu03#start_0=INFINITY: Transient attribute change | cib=0.532.2 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 275 aborted by status-1-last-failure-apc_snmp_node1_an-pdu03.start_0 doing create last-failure-apc_snmp_node1_an-pdu03#start_0=1627086256: Transient attribute change | cib=0.532.3 source=abort_unless_down:313 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:24:02 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:24:02 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n01 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n01) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n01 ) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 276, saving inputs in /var/lib/pacemaker/pengine/pe-input-293.bz2 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627086256-1001 is obsolete Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:24:02 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:24:02 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_0 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n01) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Stop apc_snmp_node1_an-pdu03 ( an-a02n01 ) due to node availability Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 277, saving inputs in /var/lib/pacemaker/pengine/pe-input-294.bz2 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 277 (ref=pe_calc-dc-1627086256-1002) derived from /var/lib/pacemaker/pengine/pe-input-294.bz2 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 locally on an-a02n01 | action 11 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=11:277:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_stop_0 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-execd [121630] (log_execute) info: executing - rsc:apc_snmp_node1_an-pdu03 action:stop call_id:359 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1980) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-execd [121630] (log_finished) info: apc_snmp_node1_an-pdu03 stop (call 359) exited with status 0 (execution time 0ms, queue time 0ms) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01: ok | rc=0 call=359 key=apc_snmp_node1_an-pdu03_stop_0 confirmed=true cib-update=1981 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.532.3 2 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.532.4 (null) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @operation_key=apc_snmp_node1_an-pdu03_stop_0, @operation=stop, @transition-key=11:277:0:789131c0-5164-4a52-9822-30c5026c3827, @transition-magic=-1:193;11:277:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=16 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1980, version=0.532.4) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/1981) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.532.4 2 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.532.5 (null) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:0;11:277:0:789131c0-5164-4a52-9822-30c5026c3827, @call-id=359, @rc-code=0, @op-status=0 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/1981, version=0.532.5) Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 277 action 11 (apc_snmp_node1_an-pdu03_stop_0 on an-a02n01) confirmed: ok | rc=0 call-id=359 Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 277 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-294.bz2): Complete Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:24:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:24:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 4144694d4d88e721664b275d4ba4364b for 0.532.5 (0x5619764caa40 0) Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-controld [121633] (crm_timer_popped) info: Cluster Recheck Timer just popped | input=I_PE_CALC time=900000ms Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:24:02 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n01) Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 278, saving inputs in /var/lib/pacemaker/pengine/pe-input-295.bz2 Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 278 (ref=pe_calc-dc-1627087156-1004) derived from /var/lib/pacemaker/pengine/pe-input-295.bz2 Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 278 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-295.bz2): Complete Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:39:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-controld [121633] (crm_timer_popped) info: Cluster Recheck Timer just popped | input=I_PE_CALC time=900000ms Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:24:02 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n01) Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 279, saving inputs in /var/lib/pacemaker/pengine/pe-input-295.bz2 Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (handle_pecalc_op) info: Input has not changed since last time, not saving to disk Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 279 (ref=pe_calc-dc-1627088056-1005) derived from /var/lib/pacemaker/pengine/pe-input-295.bz2 Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 279 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-295.bz2): Complete Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 17:54:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-controld [121633] (crm_timer_popped) info: Cluster Recheck Timer just popped | input=I_PE_CALC time=900000ms Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:24:02 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n01) Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 280, saving inputs in /var/lib/pacemaker/pengine/pe-input-295.bz2 Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (handle_pecalc_op) info: Input has not changed since last time, not saving to disk Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 280 (ref=pe_calc-dc-1627088956-1006) derived from /var/lib/pacemaker/pengine/pe-input-295.bz2 Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 280 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-295.bz2): Complete Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 18:09:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-controld [121633] (crm_timer_popped) info: Cluster Recheck Timer just popped | input=I_PE_CALC time=900000ms Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:24:02 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n01) Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 281, saving inputs in /var/lib/pacemaker/pengine/pe-input-295.bz2 Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (handle_pecalc_op) info: Input has not changed since last time, not saving to disk Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 281 (ref=pe_calc-dc-1627089856-1007) derived from /var/lib/pacemaker/pengine/pe-input-295.bz2 Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 281 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-295.bz2): Complete Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 18:24:16 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.532.5 2 Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.533.0 14b0fa264c2f98cae6f019ee0d2d9e0e Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3'] Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=533, @num_updates=0 Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.533.0) Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 281 aborted by deletion of fencing-level[@id='fl-an-a02n01-3']: Configuration change | cib=0.533.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3'] complete=true Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.533.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3'] Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-59.raw Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.533.0 of the CIB to disk (digest: 19813812307c73175eeac19785c93a36) Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.ase8bb (digest: /var/lib/pacemaker/cib/cib.Nk4VKa) Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:24:02 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n01) Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 282, saving inputs in /var/lib/pacemaker/pengine/pe-input-296.bz2 Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 282 (ref=pe_calc-dc-1627090167-1008) derived from /var/lib/pacemaker/pengine/pe-input-296.bz2 Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 282 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-296.bz2): Complete Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 18:29:27 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.533.0 2 Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.534.0 (null) Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=534 Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.534.0) Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 282 aborted by deletion of fencing-level[@id='fl-an-a02n01-2']: Configuration change | cib=0.534.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] complete=true Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.534.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2'] Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-60.raw Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.534.0 of the CIB to disk (digest: 54a2d95ffcb07ce66d98a03ab2b2c6d1) Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.URPhv3 (digest: /var/lib/pacemaker/cib/cib.pCwVO1) Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:24:02 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n01) Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 283, saving inputs in /var/lib/pacemaker/pengine/pe-input-297.bz2 Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 283 (ref=pe_calc-dc-1627090170-1009) derived from /var/lib/pacemaker/pengine/pe-input-297.bz2 Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 283 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-297.bz2): Complete Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 18:29:30 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.534.0 2 Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.535.0 (null) Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=535 Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.535.0) Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 283 aborted by deletion of fencing-level[@id='fl-an-a02n01-1']: Configuration change | cib=0.535.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] complete=true Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after delete operation 0.535.0 for /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1'] Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-61.raw Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.535.0 of the CIB to disk (digest: c562593126aa95521248f2e66106a22c) Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.BJPLfC (digest: /var/lib/pacemaker/cib/cib.lAbWOC) Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:24:02 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n01) Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 284, saving inputs in /var/lib/pacemaker/pengine/pe-input-298.bz2 Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 284 (ref=pe_calc-dc-1627090172-1010) derived from /var/lib/pacemaker/pengine/pe-input-298.bz2 Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 284 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-298.bz2): Complete Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 18:29:32 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 18:29:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: efc4eacab7e4ac8ba1ffde9363da1582 for 0.535.0 (0x5619762d3e40 0) Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.535.0 2 Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.536.0 (null) Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=536 Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03']: Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.536.0) Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 284 aborted by meta_attributes.apc_snmp_node1_an-pdu03-meta_attributes 'create': Configuration change | cib=0.536.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03'] complete=true Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create primitive[@id='apc_snmp_node1_an-pdu03'] Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.536.0 Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu03 has been disabled Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node2_an-pdu04 has been disabled Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu04 has been disabled Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_device_update) info: Device apc_snmp_node1_an-pdu03 has been disabled Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-62.raw Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.536.0 of the CIB to disk (digest: 4ec34566808bbc19b8f48d44eca665ea) Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.TGB10u (digest: /var/lib/pacemaker/cib/cib.zpp9tu) Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 17:24:02 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe_get_failcount) info: apc_snmp_node1_an-pdu03 has failed INFINITY times on an-a02n01 Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (check_migration_threshold) warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n01) Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 285, saving inputs in /var/lib/pacemaker/pengine/pe-input-299.bz2 Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 285 (ref=pe_calc-dc-1627090183-1011) derived from /var/lib/pacemaker/pengine/pe-input-299.bz2 Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 285 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-299.bz2): Complete Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 18:29:43 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 18:29:48 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: e25999215308da1427ed701849379a25 for 0.536.0 (0x56197618a9e0 0) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/crmd/1990) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: INFINITY -> (unset) | from an-a02n01 Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 597 with 1 change for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_peer_update) notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: 1627086256 -> (unset) | from an-a02n01 Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 598 with 1 change for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.536.0 2 Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.536.1 (null) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/1990, version=0.536.0) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-controld [121633] (delete_resource) info: Removing resource apc_snmp_node1_an-pdu03 from executor for c290c1ad-446f-4b54-a7b4-8de4612464a3 as root Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/597) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_resource_history) info: Clearing resource history for apc_snmp_node1_an-pdu03 on an-a02n01 (via CIB call 1991) | xpath=//node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-controld [121633] (notify_deleted) info: Notifying c290c1ad-446f-4b54-a7b4-8de4612464a3 on an-a02n01 that apc_snmp_node1_an-pdu03 was deleted Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/598) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.536.0 2 Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.536.1 (null) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0'] Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/597, version=0.536.1) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 597 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.536.1 2 Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.536.2 (null) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0'] Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/598, version=0.536.2) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 598 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/crmd/1991) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 285 aborted by deletion of nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.536.1 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 285 aborted by deletion of nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0']: Transient attribute change | cib=0.536.2 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apc_snmp_node1_an-pdu03.start_0'] complete=true Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.536.2 2 Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.536.3 (null) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/crmd/1991, version=0.536.3) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 285 aborted by deletion of lrm_resource[@id='apc_snmp_node1_an-pdu03']: Resource state removal | cib=0.536.3 source=abort_unless_down:327 path=/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03'] complete=true Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1993) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.536.3 2 Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.537.0 (null) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=537, @num_updates=0 Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']: @value=1627090193 Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/1993, version=0.537.0) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-63.raw Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.537.0 of the CIB to disk (digest: 6e7ffdf9f6269f413c9f542ab655dfcb) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.UKhl1l (digest: /var/lib/pacemaker/cib/cib.GdYuKj) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 285 aborted by cib-bootstrap-options-last-lrm-refresh doing modify last-lrm-refresh=1627090193: Configuration change | cib=0.537.0 source=te_update_diff_v2:474 path=/cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh'] complete=true Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n01) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 18:29:53 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 286, saving inputs in /var/lib/pacemaker/pengine/pe-input-300.bz2 Jul 23 18:29:54 an-a02n01.alteeve.com pacemaker-controld [121633] (handle_response) info: pe_calc calculation pe_calc-dc-1627090193-1013 is obsolete Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu03 cannot run anywhere Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node2_an-pdu04 cannot run anywhere Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu04 cannot run anywhere Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource apc_snmp_node1_an-pdu03 cannot run anywhere Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n01) Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped) Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped) Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped) Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped) Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 287, saving inputs in /var/lib/pacemaker/pengine/pe-input-300.bz2 Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (handle_pecalc_op) info: Input has not changed since last time, not saving to disk Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 287 (ref=pe_calc-dc-1627090195-1014) derived from /var/lib/pacemaker/pengine/pe-input-300.bz2 Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation apc_snmp_node1_an-pdu03_monitor_0 locally on an-a02n01 | action 11 Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_get_rsc_info) info: Agent information for 'apc_snmp_node1_an-pdu03' not in cache Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-execd [121630] (process_lrmd_rsc_register) info: Cached agent information for 'apc_snmp_node1_an-pdu03' Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of probe operation for apc_snmp_node1_an-pdu03 on an-a02n01 | transition_key=11:287:7:789131c0-5164-4a52-9822-30c5026c3827 op_key=apc_snmp_node1_an-pdu03_monitor_0 Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) notice: Result of probe operation for apc_snmp_node1_an-pdu03 on an-a02n01: not running | rc=7 call=364 key=apc_snmp_node1_an-pdu03_monitor_0 confirmed=true cib-update=2001 Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/2000) Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/2001) Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.537.0 2 Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.537.1 (null) Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/2000, version=0.537.1) Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.537.1 2 Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.537.2 (null) Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='apc_snmp_node1_an-pdu03']/lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_0']: @transition-magic=0:7;11:287:7:789131c0-5164-4a52-9822-30c5026c3827, @call-id=364, @rc-code=7, @op-status=0 Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/2001, version=0.537.2) Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-controld [121633] (process_graph_event) info: Transition 287 action 11 (apc_snmp_node1_an-pdu03_monitor_0 on an-a02n01) confirmed: not running | rc=7 call-id=364 Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 287 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-300.bz2): Complete Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 18:29:55 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 18:30:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: b7ccbb94d42e035ddc90480c040f96b1 for 0.537.2 (0x5619760a2940 0) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.537.2 2 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.538.0 39a86ac0b5bb45068b3bec3971446175 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=538, @num_updates=0 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']: Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.537.2 -> 0.538.0 from an-a02n01 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.538.0) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 287 aborted by cib-bootstrap-options-maintenance-mode doing create maintenance-mode=true: Configuration change | cib=0.538.0 source=te_update_diff_v2:474 path=/cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options'] complete=true Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 599 with 1 change for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 600 with 1 change for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 1 private change for #attrd-protocol, id=n/a, set=n/a Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 601 with 1 change for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 602 with 1 change for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 603 with 1 change for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 604 with 1 change for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 605 with 1 change for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 606 with 1 change for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/599) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/600) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/601) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/602) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/603) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.538.0 2 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.538.1 (null) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/599, version=0.538.1) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 599 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.538.1 2 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.538.2 (null) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/600, version=0.538.2) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 600 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.538.2 2 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.538.3 (null) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/601, version=0.538.3) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 601 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.538.3 2 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.538.4 (null) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/602, version=0.538.4) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 602 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.538.4 2 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.538.5 (null) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/603, version=0.538.5) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 603 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/2004) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/2005) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/604) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/605) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/606) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/2004, version=0.538.5) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.538.5 2 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.538.6 (null) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/2005, version=0.538.6) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.538.6 2 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.538.7 (null) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/604, version=0.538.7) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 604 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.538.7 2 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.538.8 (null) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/605, version=0.538.8) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 605 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.538.8 2 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.538.9 (null) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/606, version=0.538.9) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 606 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-64.raw Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.538.0 of the CIB to disk (digest: 054ad196c69266e1ab0d1d14e93db0ed) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.IApoxa (digest: /var/lib/pacemaker/cib/cib.qNAaoc) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/2008, version=0.538.9) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/2009) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/2011) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/2009, version=0.538.9) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/2011, version=0.538.9) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/2013) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/2015) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Not making join-40 offer to inactive node an-a02n02 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-40 offer to an-a02n01 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-40 requests from 1 outstanding node Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/2013, version=0.538.9) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/2015, version=0.538.9) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: 65d19f0562a09193ecf4e5ca0e97af5b Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.538.9 with 0.538.9 from an-a02n01 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/2018, version=0.538.9) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/2019) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-65.raw Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/2019, version=0.538.9) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.538.0 of the CIB to disk (digest: 054ad196c69266e1ab0d1d14e93db0ed) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.sXa8kk (digest: /var/lib/pacemaker/cib/cib.1YRjEm) Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[348700] exited with status 0 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[348702] exited with status 0 Jul 23 18:33:37 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[348703] exited with status 0 Jul 23 18:33:38 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[348706] exited with status 0 Jul 23 18:33:38 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[348726] exited with status 0 Jul 23 18:33:38 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[348734] exited with status 0 Jul 23 18:33:38 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[348736] exited with status 0 Jul 23 18:33:39 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[348738] exited with status 0 Jul 23 18:33:39 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[348739] exited with status 0 Jul 23 18:33:39 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[348740] exited with status 0 Jul 23 18:33:39 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 2020) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 18:33:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/2020) Jul 23 18:33:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.538.9 2 Jul 23 18:33:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.538.10 (null) Jul 23 18:33:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 18:33:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 18:33:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/2020, version=0.538.10) Jul 23 18:33:39 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[348772] exited with status 0 Jul 23 18:33:39 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[348773] exited with status 0 Jul 23 18:33:40 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[348774] exited with status 0 Jul 23 18:33:40 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[348775] exited with status 0 Jul 23 18:33:40 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[348778] exited with status 0 Jul 23 18:33:40 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[348781] exited with status 0 Jul 23 18:33:40 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[348783] exited with status 0 Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[348785] exited with status 0 Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[348786] exited with status 0 Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[348787] exited with status 0 Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/2021) Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.538.10 2 Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.538.11 (null) Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/2021, version=0.538.11) Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 287 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/2024) Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/2025) Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/2026) Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/2024, version=0.538.11) Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.538.11 2 Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.538.12 (null) Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/2025, version=0.538.12) Jul 23 18:33:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/2026, version=0.538.12) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node1 isn't managed Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node2 isn't managed Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv22-machineinterface isn't managed Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv24-inventory isn't managed Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv21-tickets isn't managed Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node1 isn't managed Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node2 isn't managed Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv19-uls isn't managed Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv20-fileprocessor isn't managed Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv23-orderprocessing isn't managed Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (CancelXmlOp) info: Action delay_node1_monitor_60000 on an-a02n01 will be stopped: maintenance mode Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (CancelXmlOp) info: Action delay_node2_monitor_60000 on an-a02n01 will be stopped: maintenance mode Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (CancelXmlOp) info: Action srv22-machineinterface_monitor_60000 on an-a02n01 will be stopped: maintenance mode Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (CancelXmlOp) info: Action srv24-inventory_monitor_60000 on an-a02n01 will be stopped: maintenance mode Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (CancelXmlOp) info: Action srv21-tickets_monitor_60000 on an-a02n01 will be stopped: maintenance mode Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (CancelXmlOp) info: Action ipmilan_node1_monitor_60000 on an-a02n01 will be stopped: maintenance mode Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (CancelXmlOp) info: Action ipmilan_node2_monitor_60000 on an-a02n01 will be stopped: maintenance mode Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (CancelXmlOp) info: Action srv19-uls_monitor_60000 on an-a02n01 will be stopped: maintenance mode Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (CancelXmlOp) info: Action srv20-fileprocessor_monitor_60000 on an-a02n01 will be stopped: maintenance mode Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (CancelXmlOp) info: Action srv23-orderprocessing_monitor_60000 on an-a02n01 will be stopped: maintenance mode Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node2 allocated to an-a02n01: active Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node2 allocated to an-a02n01: active Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node1 allocated to an-a02n01: active Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node1 allocated to an-a02n01: active Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv13-psql allocated to no node: inactive Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv14-file-processor allocated to no node: inactive Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv15-tickets allocated to no node: inactive Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv16-machine-interface allocated to no node: inactive Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv17-order-processing allocated to no node: inactive Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv18-inventory allocated to no node: inactive Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv19-uls allocated to an-a02n01: active Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv20-fileprocessor allocated to an-a02n01: active Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv21-tickets allocated to an-a02n01: active Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv22-machineinterface allocated to an-a02n01: active Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv23-orderprocessing allocated to an-a02n01: active Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv24-inventory allocated to an-a02n01: active Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu03 allocated to no node: inactive Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu04 allocated to no node: inactive Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu04 allocated to no node: inactive Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu03 allocated to no node: inactive Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (stage6) notice: Delaying fencing operations until there are resources to manage Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started unmanaged) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started unmanaged) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started unmanaged) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started unmanaged) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped unmanaged) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped unmanaged) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped unmanaged) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped unmanaged) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped unmanaged) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped unmanaged) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started unmanaged) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started unmanaged) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started unmanaged) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started unmanaged) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started unmanaged) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started unmanaged) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped unmanaged) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped unmanaged) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped unmanaged) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped unmanaged) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 288, saving inputs in /var/lib/pacemaker/pengine/pe-input-301.bz2 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 288 (ref=pe_calc-dc-1627090422-1021) derived from /var/lib/pacemaker/pengine/pe-input-301.bz2 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating cancel operation ipmilan_node2_monitor_60000 locally on an-a02n01 | action 7 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating cancel operation delay_node2_monitor_60000 locally on an-a02n01 | action 2 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating cancel operation ipmilan_node1_monitor_60000 locally on an-a02n01 | action 6 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating cancel operation delay_node1_monitor_60000 locally on an-a02n01 | action 1 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating cancel operation srv19-uls_monitor_60000 locally on an-a02n01 | action 8 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-execd [121630] (cancel_recurring_action) info: Cancelling ocf operation srv19-uls_monitor_60000 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating cancel operation srv20-fileprocessor_monitor_60000 locally on an-a02n01 | action 9 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-execd [121630] (cancel_recurring_action) info: Cancelling ocf operation srv20-fileprocessor_monitor_60000 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating cancel operation srv21-tickets_monitor_60000 locally on an-a02n01 | action 5 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-execd [121630] (cancel_recurring_action) info: Cancelling ocf operation srv21-tickets_monitor_60000 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating cancel operation srv22-machineinterface_monitor_60000 locally on an-a02n01 | action 3 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-execd [121630] (cancel_recurring_action) info: Cancelling ocf operation srv22-machineinterface_monitor_60000 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating cancel operation srv23-orderprocessing_monitor_60000 locally on an-a02n01 | action 10 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-execd [121630] (cancel_recurring_action) info: Cancelling ocf operation srv23-orderprocessing_monitor_60000 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating cancel operation srv24-inventory_monitor_60000 locally on an-a02n01 | action 4 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-execd [121630] (cancel_recurring_action) info: Cancelling ocf operation srv24-inventory_monitor_60000 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) info: Result of monitor operation for ipmilan_node2 on an-a02n01: Cancelled | call=161 key=ipmilan_node2_monitor_60000 confirmed=true Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) info: Result of monitor operation for delay_node2 on an-a02n01: Cancelled | call=341 key=delay_node2_monitor_60000 confirmed=true Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) info: Result of monitor operation for ipmilan_node1 on an-a02n01: Cancelled | call=342 key=ipmilan_node1_monitor_60000 confirmed=true Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) info: Result of monitor operation for delay_node1 on an-a02n01: Cancelled | call=165 key=delay_node1_monitor_60000 confirmed=true Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section status to all (origin=local/crmd/2029) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) info: Result of monitor operation for srv19-uls on an-a02n01: Cancelled | call=95 key=srv19-uls_monitor_60000 confirmed=true Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) info: Result of monitor operation for srv20-fileprocessor on an-a02n01: Cancelled | call=97 key=srv20-fileprocessor_monitor_60000 confirmed=true Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section status to all (origin=local/crmd/2030) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section status to all (origin=local/crmd/2031) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) info: Result of monitor operation for srv21-tickets on an-a02n01: Cancelled | call=99 key=srv21-tickets_monitor_60000 confirmed=true Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section status to all (origin=local/crmd/2032) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section status to all (origin=local/crmd/2033) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) info: Result of monitor operation for srv22-machineinterface on an-a02n01: Cancelled | call=101 key=srv22-machineinterface_monitor_60000 confirmed=true Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section status to all (origin=local/crmd/2034) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) info: Result of monitor operation for srv23-orderprocessing on an-a02n01: Cancelled | call=103 key=srv23-orderprocessing_monitor_60000 confirmed=true Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section status to all (origin=local/crmd/2035) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section status to all (origin=local/crmd/2036) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (process_lrm_event) info: Result of monitor operation for srv24-inventory on an-a02n01: Cancelled | call=105 key=srv24-inventory_monitor_60000 confirmed=true Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.538.12 2 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.538.13 (null) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ipmilan_node2']/lrm_rsc_op[@id='ipmilan_node2_monitor_60000'] Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section status: OK (rc=0, origin=an-a02n01/crmd/2029, version=0.538.13) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (confirm_cancel_action) info: Cancellation of ipmilan_node2_monitor_60000 on an-a02n01 confirmed (action 7) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.538.13 2 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.538.14 (null) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='delay_node2']/lrm_rsc_op[@id='delay_node2_monitor_60000'] Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=14 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section status: OK (rc=0, origin=an-a02n01/crmd/2030, version=0.538.14) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (confirm_cancel_action) info: Cancellation of delay_node2_monitor_60000 on an-a02n01 confirmed (action 2) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.538.14 2 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.538.15 (null) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ipmilan_node1']/lrm_rsc_op[@id='ipmilan_node1_monitor_60000'] Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=15 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section status: OK (rc=0, origin=an-a02n01/crmd/2031, version=0.538.15) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (confirm_cancel_action) info: Cancellation of ipmilan_node1_monitor_60000 on an-a02n01 confirmed (action 6) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.538.15 2 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.538.16 (null) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='delay_node1']/lrm_rsc_op[@id='delay_node1_monitor_60000'] Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=16 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section status: OK (rc=0, origin=an-a02n01/crmd/2032, version=0.538.16) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (confirm_cancel_action) info: Cancellation of delay_node1_monitor_60000 on an-a02n01 confirmed (action 1) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.538.16 2 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.538.17 (null) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='srv19-uls']/lrm_rsc_op[@id='srv19-uls_monitor_60000'] Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=17 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section status: OK (rc=0, origin=an-a02n01/crmd/2033, version=0.538.17) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (confirm_cancel_action) info: Cancellation of srv19-uls_monitor_60000 on an-a02n01 confirmed (action 8) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.538.17 2 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.538.18 (null) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='srv20-fileprocessor']/lrm_rsc_op[@id='srv20-fileprocessor_monitor_60000'] Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=18 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section status: OK (rc=0, origin=an-a02n01/crmd/2034, version=0.538.18) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (confirm_cancel_action) info: Cancellation of srv20-fileprocessor_monitor_60000 on an-a02n01 confirmed (action 9) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.538.18 2 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.538.19 (null) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='srv21-tickets']/lrm_rsc_op[@id='srv21-tickets_monitor_60000'] Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=19 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section status: OK (rc=0, origin=an-a02n01/crmd/2035, version=0.538.19) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (confirm_cancel_action) info: Cancellation of srv21-tickets_monitor_60000 on an-a02n01 confirmed (action 5) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.538.19 2 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.538.20 (null) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='srv22-machineinterface']/lrm_rsc_op[@id='srv22-machineinterface_monitor_60000'] Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=20 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section status: OK (rc=0, origin=an-a02n01/crmd/2036, version=0.538.20) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (confirm_cancel_action) info: Cancellation of srv22-machineinterface_monitor_60000 on an-a02n01 confirmed (action 3) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section status to all (origin=local/crmd/2037) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section status to all (origin=local/crmd/2038) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.538.20 2 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.538.21 (null) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='srv23-orderprocessing']/lrm_rsc_op[@id='srv23-orderprocessing_monitor_60000'] Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=21 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section status: OK (rc=0, origin=an-a02n01/crmd/2037, version=0.538.21) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (confirm_cancel_action) info: Cancellation of srv23-orderprocessing_monitor_60000 on an-a02n01 confirmed (action 10) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.538.21 2 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.538.22 (null) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='srv24-inventory']/lrm_rsc_op[@id='srv24-inventory_monitor_60000'] Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=22 Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section status: OK (rc=0, origin=an-a02n01/crmd/2038, version=0.538.22) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (confirm_cancel_action) info: Cancellation of srv24-inventory_monitor_60000 on an-a02n01 confirmed (action 4) Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 288 (Complete=10, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-301.bz2): Complete Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 18:33:42 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 18:33:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: e1fd3e1d2300d6b06b49d727038d9bd2 for 0.538.22 (0x561976166450 0) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.538.22 2 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.539.0 (null) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/fencing-topology Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=539, @num_updates=0 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Replaced: 0.538.22 -> 0.539.0 from an-a02n01 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.539.0) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 288 aborted by deletion of fencing-topology: Configuration change | cib=0.539.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology complete=true Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 608 with 1 change for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 609 with 1 change for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 1 private change for #attrd-protocol, id=n/a, set=n/a Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 610 with 1 change for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 611 with 1 change for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 612 with 1 change for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 613 with 1 change for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 614 with 1 change for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 615 with 1 change for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/608) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/609) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/610) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/611) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/612) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level delete operation 0.539.0 for /cib/configuration/fencing-topology Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.539.0 2 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.539.1 (null) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/608, version=0.539.1) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 608 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.539.1 2 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.539.2 (null) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/609, version=0.539.2) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 609 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.539.2 2 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.539.3 (null) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/610, version=0.539.3) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 610 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.539.3 2 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.539.4 (null) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/611, version=0.539.4) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 611 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.539.4 2 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.539.5 (null) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/612, version=0.539.5) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 612 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/2041) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/2042) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/613) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/614) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/615) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/2041, version=0.539.5) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.539.5 2 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.539.6 (null) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/2042, version=0.539.6) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.539.6 2 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.539.7 (null) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/613, version=0.539.7) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 613 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.539.7 2 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.539.8 (null) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/614, version=0.539.8) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 614 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.539.8 2 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.539.9 (null) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/615, version=0.539.9) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 615 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-66.raw Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/2044, version=0.539.9) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/2045) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.539.0 of the CIB to disk (digest: 38d7be5a87db3ff311a8f53de4703b4e) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.0UWUjO (digest: /var/lib/pacemaker/cib/cib.B8hPCS) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/2045, version=0.539.9) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/2047) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/2047, version=0.539.9) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/2049) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Not making join-41 offer to inactive node an-a02n02 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-41 offer to an-a02n01 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-41 requests from 1 outstanding node Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/2049, version=0.539.9) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/2051) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/2051, version=0.539.9) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: 64a238dd9c9f59d42e2bfabaa5a2a644 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.539.9 with 0.539.9 from an-a02n01 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/2054, version=0.539.9) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/2055) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-67.raw Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/2055, version=0.539.9) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.539.0 of the CIB to disk (digest: 38d7be5a87db3ff311a8f53de4703b4e) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.MIsUA2 (digest: /var/lib/pacemaker/cib/cib.0HFVQ5) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[350019] exited with status 0 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[350022] exited with status 0 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu03'] to all (origin=local/cibadmin/2) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.539.9 2 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.540.0 (null) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu03'] Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=540, @num_updates=0 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu03']: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.540.0) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Device 'apc_snmp_node1_an-pdu03' not found (4 active devices) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-68.raw Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.540.0 of the CIB to disk (digest: 5c7e21ff1b3e0b8791c61b19966c978c) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.3VMfat (digest: /var/lib/pacemaker/cib/cib.AI6g4p) Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[350024] exited with status 0 Jul 23 18:34:19 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[350026] exited with status 0 Jul 23 18:34:20 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[350039] exited with status 0 Jul 23 18:34:20 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[350042] exited with status 0 Jul 23 18:34:20 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[350044] exited with status 0 Jul 23 18:34:20 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[350046] exited with status 0 Jul 23 18:34:21 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[350049] exited with status 0 Jul 23 18:34:21 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[350050] exited with status 0 Jul 23 18:34:21 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 2056) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 18:34:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/2056) Jul 23 18:34:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.540.0 2 Jul 23 18:34:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.540.1 (null) Jul 23 18:34:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 18:34:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 18:34:21 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/2056, version=0.540.1) Jul 23 18:34:21 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[350051] exited with status 0 Jul 23 18:34:21 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[350052] exited with status 0 Jul 23 18:34:21 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[350053] exited with status 0 Jul 23 18:34:22 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[350054] exited with status 0 Jul 23 18:34:22 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[350055] exited with status 0 Jul 23 18:34:22 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[350072] exited with status 0 Jul 23 18:34:22 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[350074] exited with status 0 Jul 23 18:34:22 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[350076] exited with status 0 Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[350079] exited with status 0 Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[350082] exited with status 0 Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/2057) Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.540.1 2 Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.540.2 (null) Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/2057, version=0.540.2) Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 288 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/2060) Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/2061) Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/2062) Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/2060, version=0.540.2) Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.540.2 2 Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.540.3 (null) Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/2061, version=0.540.3) Jul 23 18:34:23 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/2062, version=0.540.3) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node1 isn't managed Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node2 isn't managed Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv22-machineinterface isn't managed Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv24-inventory isn't managed Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv21-tickets isn't managed Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node1 isn't managed Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node2 isn't managed Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv19-uls isn't managed Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv20-fileprocessor isn't managed Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv23-orderprocessing isn't managed Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n02 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now f2317cad3d54cec5d7d7aa7d0bf35cf8 (reload:3.10.2) 0:1;33:256:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node1_an-pdu03 from an-a02n01 Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node2 allocated to an-a02n01: active Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node2 allocated to an-a02n01: active Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node1 allocated to an-a02n01: active Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node1 allocated to an-a02n01: active Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv13-psql allocated to no node: inactive Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv14-file-processor allocated to no node: inactive Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv15-tickets allocated to no node: inactive Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv16-machine-interface allocated to no node: inactive Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv17-order-processing allocated to no node: inactive Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv18-inventory allocated to no node: inactive Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv19-uls allocated to an-a02n01: active Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv20-fileprocessor allocated to an-a02n01: active Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv21-tickets allocated to an-a02n01: active Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv22-machineinterface allocated to an-a02n01: active Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv23-orderprocessing allocated to an-a02n01: active Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv24-inventory allocated to an-a02n01: active Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu03 allocated to no node: inactive Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu04 allocated to no node: inactive Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu04 allocated to no node: inactive Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu03 allocated to no node: inactive Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (stage6) notice: Delaying fencing operations until there are resources to manage Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 289, saving inputs in /var/lib/pacemaker/pengine/pe-input-302.bz2 Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 289 (ref=pe_calc-dc-1627090464-1037) derived from /var/lib/pacemaker/pengine/pe-input-302.bz2 Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 289 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-302.bz2): Complete Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.540.3) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-69.raw Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.540.0 of the CIB to disk (digest: b432e3b4306ce6754de73f7d783e1685) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.bxTecs (digest: /var/lib/pacemaker/cib/cib.o08kRq) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu04'] to all (origin=local/cibadmin/2) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.540.3 2 Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.541.0 (null) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu04'] Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=541, @num_updates=0 Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //primitive[@id='apc_snmp_node1_an-pdu04']: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.541.0) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 289 aborted by deletion of primitive[@id='apc_snmp_node1_an-pdu04']: Configuration change | cib=0.541.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node1_an-pdu04'] complete=true Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Device 'apc_snmp_node1_an-pdu04' not found (4 active devices) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-70.raw Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.541.0 of the CIB to disk (digest: 18cbf1361129c4204a69ed39d911b7e6) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.xNIiWM (digest: /var/lib/pacemaker/cib/cib.dBgoDP) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node1 isn't managed Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node2 isn't managed Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv22-machineinterface isn't managed Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv24-inventory isn't managed Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv21-tickets isn't managed Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node1 isn't managed Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node2 isn't managed Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv19-uls isn't managed Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv20-fileprocessor isn't managed Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv23-orderprocessing isn't managed Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n02 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now f2317cad3d54cec5d7d7aa7d0bf35cf8 (reload:3.10.2) 0:1;33:256:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu04 on an-a02n02 changed: hash was dcfe282eb9c3d7f7189c538ed5e16421 vs. now f2317cad3d54cec5d7d7aa7d0bf35cf8 (reload:3.10.2) 0:1;36:258:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu04 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu04_clear_failcount_0 Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node1_an-pdu04 from an-a02n01 Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node1_an-pdu03 from an-a02n01 Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node2 allocated to an-a02n01: active Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node2 allocated to an-a02n01: active Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node1 allocated to an-a02n01: active Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node1 allocated to an-a02n01: active Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv13-psql allocated to no node: inactive Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv14-file-processor allocated to no node: inactive Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv15-tickets allocated to no node: inactive Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv16-machine-interface allocated to no node: inactive Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv17-order-processing allocated to no node: inactive Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv18-inventory allocated to no node: inactive Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv19-uls allocated to an-a02n01: active Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv20-fileprocessor allocated to an-a02n01: active Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv21-tickets allocated to an-a02n01: active Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv22-machineinterface allocated to an-a02n01: active Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv23-orderprocessing allocated to an-a02n01: active Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv24-inventory allocated to an-a02n01: active Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu03 allocated to no node: inactive Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu04 allocated to no node: inactive Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu04 allocated to no node: inactive Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu03 allocated to no node: inactive Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (stage6) notice: Delaying fencing operations until there are resources to manage Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped unmanaged) Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 290, saving inputs in /var/lib/pacemaker/pengine/pe-input-303.bz2 Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 290 (ref=pe_calc-dc-1627090464-1038) derived from /var/lib/pacemaker/pengine/pe-input-303.bz2 Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 290 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-303.bz2): Complete Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 18:34:24 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 18:34:29 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 1ce7f33360603428be851723b9587bd3 for 0.541.0 (0x5619760ad450 0) Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.541.0) Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-71.raw Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.541.0 of the CIB to disk (digest: d651bb6d264e682e112095874e182dc0) Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.aPMF1f (digest: /var/lib/pacemaker/cib/cib.dTZUSd) Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //primitive[@id='apc_snmp_node2_an-pdu03'] to all (origin=local/cibadmin/2) Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.541.0 2 Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.542.0 (null) Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu03'] Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=542 Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //primitive[@id='apc_snmp_node2_an-pdu03']: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.542.0) Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 290 aborted by deletion of primitive[@id='apc_snmp_node2_an-pdu03']: Configuration change | cib=0.542.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu03'] complete=true Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Device 'apc_snmp_node2_an-pdu03' not found (4 active devices) Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-72.raw Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.542.0 of the CIB to disk (digest: b31c2077bbf36554d9570ba6de12b4ea) Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.UOu8a8 (digest: /var/lib/pacemaker/cib/cib.CpST39) Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node1 isn't managed Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node2 isn't managed Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv22-machineinterface isn't managed Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv24-inventory isn't managed Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv21-tickets isn't managed Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node1 isn't managed Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node2 isn't managed Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv19-uls isn't managed Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv20-fileprocessor isn't managed Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv23-orderprocessing isn't managed Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n02 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now f2317cad3d54cec5d7d7aa7d0bf35cf8 (reload:3.10.2) 0:1;33:256:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu04 on an-a02n02 changed: hash was dcfe282eb9c3d7f7189c538ed5e16421 vs. now f2317cad3d54cec5d7d7aa7d0bf35cf8 (reload:3.10.2) 0:1;36:258:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu04 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu04_clear_failcount_0 Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node1_an-pdu04 from an-a02n01 Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node1_an-pdu03 from an-a02n01 Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node2_an-pdu03 from an-a02n01 Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node2 allocated to an-a02n01: active Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node2 allocated to an-a02n01: active Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node1 allocated to an-a02n01: active Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node1 allocated to an-a02n01: active Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv13-psql allocated to no node: inactive Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv14-file-processor allocated to no node: inactive Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv15-tickets allocated to no node: inactive Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv16-machine-interface allocated to no node: inactive Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv17-order-processing allocated to no node: inactive Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv18-inventory allocated to no node: inactive Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv19-uls allocated to an-a02n01: active Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv20-fileprocessor allocated to an-a02n01: active Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv21-tickets allocated to an-a02n01: active Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv22-machineinterface allocated to an-a02n01: active Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv23-orderprocessing allocated to an-a02n01: active Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv24-inventory allocated to an-a02n01: active Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu04 allocated to no node: inactive Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu04 allocated to no node: inactive Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu03 allocated to no node: inactive Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu03 allocated to no node: inactive Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (stage6) notice: Delaying fencing operations until there are resources to manage Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started unmanaged) Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started unmanaged) Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started unmanaged) Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started unmanaged) Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped unmanaged) Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped unmanaged) Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped unmanaged) Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped unmanaged) Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped unmanaged) Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped unmanaged) Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started unmanaged) Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started unmanaged) Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started unmanaged) Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started unmanaged) Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started unmanaged) Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started unmanaged) Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped unmanaged) Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 291, saving inputs in /var/lib/pacemaker/pengine/pe-input-304.bz2 Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 291 (ref=pe_calc-dc-1627090478-1039) derived from /var/lib/pacemaker/pengine/pe-input-304.bz2 Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 291 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-304.bz2): Complete Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 18:34:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.542.0) Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-73.raw Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.542.0 of the CIB to disk (digest: c66b9e145378a358685683f876358e62) Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.ZX7o1r (digest: /var/lib/pacemaker/cib/cib.AZCP3u) Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //primitive[@id='apc_snmp_node2_an-pdu04'] to all (origin=local/cibadmin/2) Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.542.0 2 Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.543.0 (null) Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu04'] Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=543 Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //primitive[@id='apc_snmp_node2_an-pdu04']: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.543.0) Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 291 aborted by deletion of primitive[@id='apc_snmp_node2_an-pdu04']: Configuration change | cib=0.543.0 source=te_update_diff_v2:474 path=/cib/configuration/resources/primitive[@id='apc_snmp_node2_an-pdu04'] complete=true Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_remove) info: Device 'apc_snmp_node2_an-pdu04' not found (4 active devices) Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-74.raw Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.543.0 of the CIB to disk (digest: 18da157816f3dd264f839dae2f8ce08f) Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.FSszER (digest: /var/lib/pacemaker/cib/cib.xsOoJP) Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node1 isn't managed Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node2 isn't managed Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv22-machineinterface isn't managed Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv24-inventory isn't managed Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv21-tickets isn't managed Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node1 isn't managed Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node2 isn't managed Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv19-uls isn't managed Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv20-fileprocessor isn't managed Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv23-orderprocessing isn't managed Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n02 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now f2317cad3d54cec5d7d7aa7d0bf35cf8 (reload:3.10.2) 0:1;33:256:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu04 on an-a02n02 changed: hash was dcfe282eb9c3d7f7189c538ed5e16421 vs. now f2317cad3d54cec5d7d7aa7d0bf35cf8 (reload:3.10.2) 0:1;36:258:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu04 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu04_clear_failcount_0 Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node1_an-pdu04 from an-a02n01 Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node1_an-pdu03 from an-a02n01 Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node2_an-pdu03 from an-a02n01 Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node2_an-pdu04 from an-a02n01 Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node2 allocated to an-a02n01: active Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node2 allocated to an-a02n01: active Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node1 allocated to an-a02n01: active Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node1 allocated to an-a02n01: active Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv13-psql allocated to no node: inactive Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv14-file-processor allocated to no node: inactive Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv15-tickets allocated to no node: inactive Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv16-machine-interface allocated to no node: inactive Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv17-order-processing allocated to no node: inactive Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv18-inventory allocated to no node: inactive Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv19-uls allocated to an-a02n01: active Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv20-fileprocessor allocated to an-a02n01: active Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv21-tickets allocated to an-a02n01: active Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv22-machineinterface allocated to an-a02n01: active Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv23-orderprocessing allocated to an-a02n01: active Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv24-inventory allocated to an-a02n01: active Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu04 allocated to no node: inactive Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu03 allocated to no node: inactive Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu03 allocated to no node: inactive Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu04 allocated to no node: inactive Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (stage6) notice: Delaying fencing operations until there are resources to manage Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started unmanaged) Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started unmanaged) Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started unmanaged) Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started unmanaged) Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped unmanaged) Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped unmanaged) Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped unmanaged) Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped unmanaged) Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped unmanaged) Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped unmanaged) Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started unmanaged) Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started unmanaged) Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started unmanaged) Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started unmanaged) Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started unmanaged) Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started unmanaged) Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 292, saving inputs in /var/lib/pacemaker/pengine/pe-input-305.bz2 Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 292 (ref=pe_calc-dc-1627090481-1040) derived from /var/lib/pacemaker/pengine/pe-input-305.bz2 Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 292 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-305.bz2): Complete Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 18:34:41 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 18:34:46 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: b564dfc1e65d5900013936a06e65df15 for 0.543.0 (0x5619762ad490 0) Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.543.0 2 Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.544.0 a87d5693c9c95d7865697817dab5228d Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=544 Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources: Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.544.0) Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 292 aborted by primitive.apc_snmp_node1_an-pdu03 'create': Configuration change | cib=0.544.0 source=te_update_diff_v2:474 path=/cib/configuration/resources complete=true Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create resources Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.544.0 Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-75.raw Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node1_an-pdu03' to device list (5 active devices) Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.544.0 of the CIB to disk (digest: 9d9c6de9273687eb24face4beb38016f) Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.3oRxIc (digest: /var/lib/pacemaker/cib/cib.GJjkpc) Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node1 isn't managed Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node2 isn't managed Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv22-machineinterface isn't managed Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv24-inventory isn't managed Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv21-tickets isn't managed Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node1 isn't managed Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node2 isn't managed Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv19-uls isn't managed Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv20-fileprocessor isn't managed Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv23-orderprocessing isn't managed Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n02 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now cef2c24c7e11f1292c46d7137f454a30 (reload:3.10.2) 0:1;33:256:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu04 on an-a02n02 changed: hash was dcfe282eb9c3d7f7189c538ed5e16421 vs. now f2317cad3d54cec5d7d7aa7d0bf35cf8 (reload:3.10.2) 0:1;36:258:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu04 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu04_clear_failcount_0 Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node1_an-pdu04 from an-a02n01 Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node2_an-pdu03 from an-a02n01 Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node2_an-pdu04 from an-a02n01 Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node2 allocated to an-a02n01: active Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node2 allocated to an-a02n01: active Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node1 allocated to an-a02n01: active Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node1 allocated to an-a02n01: active Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv13-psql allocated to no node: inactive Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv14-file-processor allocated to no node: inactive Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv15-tickets allocated to no node: inactive Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv16-machine-interface allocated to no node: inactive Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv17-order-processing allocated to no node: inactive Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv18-inventory allocated to no node: inactive Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv19-uls allocated to an-a02n01: active Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv20-fileprocessor allocated to an-a02n01: active Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv21-tickets allocated to an-a02n01: active Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv22-machineinterface allocated to an-a02n01: active Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv23-orderprocessing allocated to an-a02n01: active Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv24-inventory allocated to an-a02n01: active Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu03 allocated to no node: inactive Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu04 allocated to no node: inactive Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu03 allocated to no node: inactive Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu04 allocated to no node: inactive Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (stage6) notice: Delaying fencing operations until there are resources to manage Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started unmanaged) Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started unmanaged) Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started unmanaged) Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started unmanaged) Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped unmanaged) Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped unmanaged) Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped unmanaged) Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped unmanaged) Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped unmanaged) Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped unmanaged) Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started unmanaged) Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started unmanaged) Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started unmanaged) Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started unmanaged) Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started unmanaged) Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started unmanaged) Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped unmanaged) Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 293, saving inputs in /var/lib/pacemaker/pengine/pe-input-306.bz2 Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 293 (ref=pe_calc-dc-1627090615-1041) derived from /var/lib/pacemaker/pengine/pe-input-306.bz2 Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 293 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-306.bz2): Complete Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 18:36:55 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 18:37:00 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: a87d5693c9c95d7865697817dab5228d for 0.544.0 (0x5619762ac720 0) Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.544.0 2 Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.545.0 (null) Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=545 Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources: Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.545.0) Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 293 aborted by primitive.apc_snmp_node1_an-pdu04 'create': Configuration change | cib=0.545.0 source=te_update_diff_v2:474 path=/cib/configuration/resources complete=true Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create resources Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.545.0 Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node1_an-pdu04' to device list (6 active devices) Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-76.raw Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.545.0 of the CIB to disk (digest: 05f3bcaf85f2026885a23d5a056ace1d) Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.H9eFLC (digest: /var/lib/pacemaker/cib/cib.pxiiJC) Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node1 isn't managed Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node2 isn't managed Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv22-machineinterface isn't managed Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv24-inventory isn't managed Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv21-tickets isn't managed Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node1 isn't managed Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node2 isn't managed Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv19-uls isn't managed Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv20-fileprocessor isn't managed Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv23-orderprocessing isn't managed Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n02 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now cef2c24c7e11f1292c46d7137f454a30 (reload:3.10.2) 0:1;33:256:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu04 on an-a02n02 changed: hash was dcfe282eb9c3d7f7189c538ed5e16421 vs. now 6b32ecd9197e2963fbde8b23a095287a (reload:3.10.2) 0:1;36:258:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu04 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu04_clear_failcount_0 Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node2_an-pdu03 from an-a02n01 Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node2_an-pdu04 from an-a02n01 Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node2 allocated to an-a02n01: active Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node2 allocated to an-a02n01: active Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node1 allocated to an-a02n01: active Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node1 allocated to an-a02n01: active Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv13-psql allocated to no node: inactive Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv14-file-processor allocated to no node: inactive Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv15-tickets allocated to no node: inactive Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv16-machine-interface allocated to no node: inactive Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv17-order-processing allocated to no node: inactive Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv18-inventory allocated to no node: inactive Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv19-uls allocated to an-a02n01: active Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv20-fileprocessor allocated to an-a02n01: active Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv21-tickets allocated to an-a02n01: active Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv22-machineinterface allocated to an-a02n01: active Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv23-orderprocessing allocated to an-a02n01: active Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv24-inventory allocated to an-a02n01: active Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu03 allocated to no node: inactive Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu04 allocated to no node: inactive Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu03 allocated to no node: inactive Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu04 allocated to no node: inactive Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (stage6) notice: Delaying fencing operations until there are resources to manage Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started unmanaged) Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started unmanaged) Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started unmanaged) Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started unmanaged) Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped unmanaged) Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped unmanaged) Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped unmanaged) Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped unmanaged) Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped unmanaged) Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped unmanaged) Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started unmanaged) Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started unmanaged) Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started unmanaged) Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started unmanaged) Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started unmanaged) Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started unmanaged) Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped unmanaged) Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped unmanaged) Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 294, saving inputs in /var/lib/pacemaker/pengine/pe-input-307.bz2 Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 294 (ref=pe_calc-dc-1627090628-1042) derived from /var/lib/pacemaker/pengine/pe-input-307.bz2 Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 294 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-307.bz2): Complete Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 18:37:08 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 18:37:13 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 7f8b1f377ca6565ae9f872bf0014cc3b for 0.545.0 (0x561975c6ee70 0) Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.545.0 2 Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.546.0 (null) Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=546 Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources: Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.546.0) Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 294 aborted by primitive.apc_snmp_node2_an-pdu03 'create': Configuration change | cib=0.546.0 source=te_update_diff_v2:474 path=/cib/configuration/resources complete=true Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create resources Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.546.0 Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node2_an-pdu03' to device list (7 active devices) Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-77.raw Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.546.0 of the CIB to disk (digest: 8e5796b359da0940a964cb60128d88ab) Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.gWWTVM (digest: /var/lib/pacemaker/cib/cib.Z8liYP) Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node1 isn't managed Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node2 isn't managed Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv22-machineinterface isn't managed Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv24-inventory isn't managed Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv21-tickets isn't managed Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node1 isn't managed Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node2 isn't managed Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv19-uls isn't managed Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv20-fileprocessor isn't managed Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv23-orderprocessing isn't managed Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n02 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now cef2c24c7e11f1292c46d7137f454a30 (reload:3.10.2) 0:1;33:256:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu04 on an-a02n02 changed: hash was dcfe282eb9c3d7f7189c538ed5e16421 vs. now 6b32ecd9197e2963fbde8b23a095287a (reload:3.10.2) 0:1;36:258:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu04 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu04_clear_failcount_0 Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (DeleteRsc) notice: Removing apc_snmp_node2_an-pdu04 from an-a02n01 Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node2 allocated to an-a02n01: active Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node2 allocated to an-a02n01: active Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node1 allocated to an-a02n01: active Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node1 allocated to an-a02n01: active Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv13-psql allocated to no node: inactive Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv14-file-processor allocated to no node: inactive Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv15-tickets allocated to no node: inactive Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv16-machine-interface allocated to no node: inactive Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv17-order-processing allocated to no node: inactive Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv18-inventory allocated to no node: inactive Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv19-uls allocated to an-a02n01: active Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv20-fileprocessor allocated to an-a02n01: active Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv21-tickets allocated to an-a02n01: active Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv22-machineinterface allocated to an-a02n01: active Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv23-orderprocessing allocated to an-a02n01: active Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv24-inventory allocated to an-a02n01: active Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu03 allocated to no node: inactive Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu04 allocated to no node: inactive Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu03 allocated to no node: inactive Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu04 allocated to no node: inactive Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (stage6) notice: Delaying fencing operations until there are resources to manage Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started unmanaged) Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started unmanaged) Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started unmanaged) Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started unmanaged) Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped unmanaged) Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped unmanaged) Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped unmanaged) Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped unmanaged) Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped unmanaged) Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped unmanaged) Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started unmanaged) Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started unmanaged) Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started unmanaged) Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started unmanaged) Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started unmanaged) Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started unmanaged) Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped unmanaged) Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped unmanaged) Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped unmanaged) Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 295, saving inputs in /var/lib/pacemaker/pengine/pe-input-308.bz2 Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 295 (ref=pe_calc-dc-1627090654-1043) derived from /var/lib/pacemaker/pengine/pe-input-308.bz2 Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 295 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-308.bz2): Complete Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 18:37:34 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 18:37:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: ee42b77604b321475d15dcf5894ddf6f for 0.546.0 (0x56197608d6a0 0) Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.546.0 2 Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.547.0 (null) Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=547 Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/resources: Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.547.0) Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 295 aborted by primitive.apc_snmp_node2_an-pdu04 'create': Configuration change | cib=0.547.0 source=te_update_diff_v2:474 path=/cib/configuration/resources complete=true Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_cib_stonith_devices_v2) info: Updating device list from CIB: create resources Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-fenced [121629] (cib_devices_update) info: Updating devices to version 0.547.0 Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-78.raw Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_device_register) notice: Added 'apc_snmp_node2_an-pdu04' to device list (8 active devices) Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.547.0 of the CIB to disk (digest: b00d6e10f4bb15b86f2a37870187cf46) Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.EES7zs (digest: /var/lib/pacemaker/cib/cib.v9oeqv) Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node1 isn't managed Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node2 isn't managed Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv22-machineinterface isn't managed Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv24-inventory isn't managed Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv21-tickets isn't managed Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node1 isn't managed Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node2 isn't managed Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv19-uls isn't managed Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv20-fileprocessor isn't managed Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv23-orderprocessing isn't managed Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n02 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now cef2c24c7e11f1292c46d7137f454a30 (reload:3.10.2) 0:1;33:256:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu04 on an-a02n02 changed: hash was dcfe282eb9c3d7f7189c538ed5e16421 vs. now 6b32ecd9197e2963fbde8b23a095287a (reload:3.10.2) 0:1;36:258:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu04 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu04_clear_failcount_0 Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node2 allocated to an-a02n01: active Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node2 allocated to an-a02n01: active Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node1 allocated to an-a02n01: active Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node1 allocated to an-a02n01: active Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv13-psql allocated to no node: inactive Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv14-file-processor allocated to no node: inactive Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv15-tickets allocated to no node: inactive Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv16-machine-interface allocated to no node: inactive Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv17-order-processing allocated to no node: inactive Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv18-inventory allocated to no node: inactive Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv19-uls allocated to an-a02n01: active Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv20-fileprocessor allocated to an-a02n01: active Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv21-tickets allocated to an-a02n01: active Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv22-machineinterface allocated to an-a02n01: active Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv23-orderprocessing allocated to an-a02n01: active Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv24-inventory allocated to an-a02n01: active Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu03 allocated to no node: inactive Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu04 allocated to no node: inactive Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu03 allocated to no node: inactive Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu04 allocated to no node: inactive Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (stage6) notice: Delaying fencing operations until there are resources to manage Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started unmanaged) Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started unmanaged) Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started unmanaged) Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started unmanaged) Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped unmanaged) Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped unmanaged) Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped unmanaged) Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped unmanaged) Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped unmanaged) Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped unmanaged) Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started unmanaged) Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started unmanaged) Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started unmanaged) Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started unmanaged) Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started unmanaged) Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started unmanaged) Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped unmanaged) Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped unmanaged) Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped unmanaged) Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped unmanaged) Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 296, saving inputs in /var/lib/pacemaker/pengine/pe-input-309.bz2 Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 296 (ref=pe_calc-dc-1627090667-1044) derived from /var/lib/pacemaker/pengine/pe-input-309.bz2 Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 296 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-309.bz2): Complete Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 18:37:47 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 18:37:52 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: a7de2fbc07e5612cb2889809d901c147 for 0.547.0 (0x561975c7a5d0 0) Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.547.0 2 Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.548.0 e7f7552cd4f8bd371e2e1c6eccd4cace Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=548 Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration: Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.548.0) Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 296 aborted by fencing-topology. 'create': Configuration change | cib=0.548.0 source=te_update_diff_v2:474 path=/cib/configuration complete=true Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.548.0 for /cib/configuration. Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-79.raw Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.548.0 of the CIB to disk (digest: db428e2be0221b25b8e653ceb0ea21fe) Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.iMKlH4 (digest: /var/lib/pacemaker/cib/cib.51Jd32) Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node1 isn't managed Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node2 isn't managed Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv22-machineinterface isn't managed Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv24-inventory isn't managed Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv21-tickets isn't managed Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node1 isn't managed Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node2 isn't managed Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv19-uls isn't managed Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv20-fileprocessor isn't managed Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv23-orderprocessing isn't managed Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n02 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now cef2c24c7e11f1292c46d7137f454a30 (reload:3.10.2) 0:1;33:256:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu04 on an-a02n02 changed: hash was dcfe282eb9c3d7f7189c538ed5e16421 vs. now 6b32ecd9197e2963fbde8b23a095287a (reload:3.10.2) 0:1;36:258:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu04 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu04_clear_failcount_0 Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node2 allocated to an-a02n01: active Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node2 allocated to an-a02n01: active Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node1 allocated to an-a02n01: active Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node1 allocated to an-a02n01: active Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv13-psql allocated to no node: inactive Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv14-file-processor allocated to no node: inactive Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv15-tickets allocated to no node: inactive Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv16-machine-interface allocated to no node: inactive Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv17-order-processing allocated to no node: inactive Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv18-inventory allocated to no node: inactive Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv19-uls allocated to an-a02n01: active Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv20-fileprocessor allocated to an-a02n01: active Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv21-tickets allocated to an-a02n01: active Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv22-machineinterface allocated to an-a02n01: active Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv23-orderprocessing allocated to an-a02n01: active Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv24-inventory allocated to an-a02n01: active Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu03 allocated to no node: inactive Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu04 allocated to no node: inactive Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu03 allocated to no node: inactive Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu04 allocated to no node: inactive Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (stage6) notice: Delaying fencing operations until there are resources to manage Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started unmanaged) Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started unmanaged) Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started unmanaged) Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started unmanaged) Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped unmanaged) Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped unmanaged) Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped unmanaged) Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped unmanaged) Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped unmanaged) Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped unmanaged) Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started unmanaged) Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started unmanaged) Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started unmanaged) Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started unmanaged) Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started unmanaged) Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started unmanaged) Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped unmanaged) Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped unmanaged) Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped unmanaged) Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped unmanaged) Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 297, saving inputs in /var/lib/pacemaker/pengine/pe-input-310.bz2 Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 297 (ref=pe_calc-dc-1627090747-1045) derived from /var/lib/pacemaker/pengine/pe-input-310.bz2 Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 297 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-310.bz2): Complete Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 18:39:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 18:39:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: e7f7552cd4f8bd371e2e1c6eccd4cace for 0.548.0 (0x561975c2c790 0) Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.548.0 2 Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.549.0 (null) Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=549 Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.549.0) Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 297 aborted by fencing-level.fl-an-a02n02-1 'create': Configuration change | cib=0.549.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology complete=true Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.549.0 for /cib/configuration/fencing-topology Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (1 active entry) Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-80.raw Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.549.0 of the CIB to disk (digest: 01f4300a8b5cac5dc261b40c5b9c5895) Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.Mjkfld (digest: /var/lib/pacemaker/cib/cib.XoYUMe) Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node1 isn't managed Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node2 isn't managed Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv22-machineinterface isn't managed Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv24-inventory isn't managed Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv21-tickets isn't managed Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node1 isn't managed Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node2 isn't managed Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv19-uls isn't managed Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv20-fileprocessor isn't managed Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv23-orderprocessing isn't managed Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n02 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now cef2c24c7e11f1292c46d7137f454a30 (reload:3.10.2) 0:1;33:256:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu04 on an-a02n02 changed: hash was dcfe282eb9c3d7f7189c538ed5e16421 vs. now 6b32ecd9197e2963fbde8b23a095287a (reload:3.10.2) 0:1;36:258:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu04 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu04_clear_failcount_0 Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node2 allocated to an-a02n01: active Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node2 allocated to an-a02n01: active Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node1 allocated to an-a02n01: active Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node1 allocated to an-a02n01: active Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv13-psql allocated to no node: inactive Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv14-file-processor allocated to no node: inactive Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv15-tickets allocated to no node: inactive Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv16-machine-interface allocated to no node: inactive Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv17-order-processing allocated to no node: inactive Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv18-inventory allocated to no node: inactive Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv19-uls allocated to an-a02n01: active Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv20-fileprocessor allocated to an-a02n01: active Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv21-tickets allocated to an-a02n01: active Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv22-machineinterface allocated to an-a02n01: active Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv23-orderprocessing allocated to an-a02n01: active Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv24-inventory allocated to an-a02n01: active Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu03 allocated to no node: inactive Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu04 allocated to no node: inactive Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu03 allocated to no node: inactive Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu04 allocated to no node: inactive Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (stage6) notice: Delaying fencing operations until there are resources to manage Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started unmanaged) Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started unmanaged) Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started unmanaged) Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started unmanaged) Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped unmanaged) Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped unmanaged) Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped unmanaged) Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped unmanaged) Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped unmanaged) Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped unmanaged) Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started unmanaged) Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started unmanaged) Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started unmanaged) Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started unmanaged) Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started unmanaged) Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started unmanaged) Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped unmanaged) Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped unmanaged) Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped unmanaged) Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped unmanaged) Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 298, saving inputs in /var/lib/pacemaker/pengine/pe-input-311.bz2 Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 298 (ref=pe_calc-dc-1627090755-1046) derived from /var/lib/pacemaker/pengine/pe-input-311.bz2 Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 298 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-311.bz2): Complete Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 18:39:15 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 18:39:20 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: f547ca90883bfb0b2a2b45acc6a1533e for 0.549.0 (0x56197614b090 0) Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.549.0 2 Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.550.0 (null) Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=550 Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.550.0) Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 298 aborted by fencing-level.fl-an-a02n01-2 'create': Configuration change | cib=0.550.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology complete=true Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.550.0 for /cib/configuration/fencing-topology Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (1 active entry) Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-81.raw Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.550.0 of the CIB to disk (digest: 99c05dd18cc0fd9175722ef91c6a3294) Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.pO7eQN (digest: /var/lib/pacemaker/cib/cib.bQhddM) Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node1 isn't managed Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node2 isn't managed Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv22-machineinterface isn't managed Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv24-inventory isn't managed Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv21-tickets isn't managed Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node1 isn't managed Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node2 isn't managed Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv19-uls isn't managed Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv20-fileprocessor isn't managed Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv23-orderprocessing isn't managed Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n02 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now cef2c24c7e11f1292c46d7137f454a30 (reload:3.10.2) 0:1;33:256:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu04 on an-a02n02 changed: hash was dcfe282eb9c3d7f7189c538ed5e16421 vs. now 6b32ecd9197e2963fbde8b23a095287a (reload:3.10.2) 0:1;36:258:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu04 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu04_clear_failcount_0 Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node2 allocated to an-a02n01: active Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node2 allocated to an-a02n01: active Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node1 allocated to an-a02n01: active Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node1 allocated to an-a02n01: active Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv13-psql allocated to no node: inactive Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv14-file-processor allocated to no node: inactive Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv15-tickets allocated to no node: inactive Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv16-machine-interface allocated to no node: inactive Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv17-order-processing allocated to no node: inactive Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv18-inventory allocated to no node: inactive Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv19-uls allocated to an-a02n01: active Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv20-fileprocessor allocated to an-a02n01: active Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv21-tickets allocated to an-a02n01: active Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv22-machineinterface allocated to an-a02n01: active Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv23-orderprocessing allocated to an-a02n01: active Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv24-inventory allocated to an-a02n01: active Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu03 allocated to no node: inactive Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu04 allocated to no node: inactive Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu03 allocated to no node: inactive Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu04 allocated to no node: inactive Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (stage6) notice: Delaying fencing operations until there are resources to manage Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started unmanaged) Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started unmanaged) Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started unmanaged) Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started unmanaged) Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped unmanaged) Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped unmanaged) Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped unmanaged) Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped unmanaged) Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped unmanaged) Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped unmanaged) Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started unmanaged) Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started unmanaged) Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started unmanaged) Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started unmanaged) Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started unmanaged) Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started unmanaged) Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped unmanaged) Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped unmanaged) Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped unmanaged) Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped unmanaged) Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 299, saving inputs in /var/lib/pacemaker/pengine/pe-input-312.bz2 Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 299 (ref=pe_calc-dc-1627090807-1047) derived from /var/lib/pacemaker/pengine/pe-input-312.bz2 Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 299 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-312.bz2): Complete Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 18:40:07 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 18:40:12 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: aa6ec5e0f5056e7627b3e2970e395c7b for 0.550.0 (0x5619760902f0 0) Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.550.0 2 Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.551.0 63bd7ee763cb3b3ac90c6da144d59ff9 Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=551 Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.551.0) Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 299 aborted by fencing-level.fl-an-a02n02-2 'create': Configuration change | cib=0.551.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology complete=true Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.551.0 for /cib/configuration/fencing-topology Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (1 active entry) Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-82.raw Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.551.0 of the CIB to disk (digest: d9e5f6e7e34e78483b322661ce3dd6ec) Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.pqP2Pn (digest: /var/lib/pacemaker/cib/cib.eTx83l) Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node1 isn't managed Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node2 isn't managed Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv22-machineinterface isn't managed Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv24-inventory isn't managed Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv21-tickets isn't managed Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node1 isn't managed Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node2 isn't managed Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv19-uls isn't managed Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv20-fileprocessor isn't managed Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv23-orderprocessing isn't managed Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n02 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now cef2c24c7e11f1292c46d7137f454a30 (reload:3.10.2) 0:1;33:256:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu04 on an-a02n02 changed: hash was dcfe282eb9c3d7f7189c538ed5e16421 vs. now 6b32ecd9197e2963fbde8b23a095287a (reload:3.10.2) 0:1;36:258:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu04 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu04_clear_failcount_0 Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node2 allocated to an-a02n01: active Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node2 allocated to an-a02n01: active Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node1 allocated to an-a02n01: active Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node1 allocated to an-a02n01: active Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv13-psql allocated to no node: inactive Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv14-file-processor allocated to no node: inactive Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv15-tickets allocated to no node: inactive Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv16-machine-interface allocated to no node: inactive Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv17-order-processing allocated to no node: inactive Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv18-inventory allocated to no node: inactive Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv19-uls allocated to an-a02n01: active Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv20-fileprocessor allocated to an-a02n01: active Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv21-tickets allocated to an-a02n01: active Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv22-machineinterface allocated to an-a02n01: active Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv23-orderprocessing allocated to an-a02n01: active Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv24-inventory allocated to an-a02n01: active Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu03 allocated to no node: inactive Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu04 allocated to no node: inactive Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu03 allocated to no node: inactive Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu04 allocated to no node: inactive Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (stage6) notice: Delaying fencing operations until there are resources to manage Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started unmanaged) Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started unmanaged) Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started unmanaged) Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started unmanaged) Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped unmanaged) Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped unmanaged) Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped unmanaged) Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped unmanaged) Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped unmanaged) Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped unmanaged) Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started unmanaged) Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started unmanaged) Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started unmanaged) Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started unmanaged) Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started unmanaged) Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started unmanaged) Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped unmanaged) Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped unmanaged) Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped unmanaged) Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped unmanaged) Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 300, saving inputs in /var/lib/pacemaker/pengine/pe-input-313.bz2 Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 300 (ref=pe_calc-dc-1627090817-1048) derived from /var/lib/pacemaker/pengine/pe-input-313.bz2 Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 300 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-313.bz2): Complete Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 18:40:17 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 18:40:22 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 63bd7ee763cb3b3ac90c6da144d59ff9 for 0.551.0 (0x56197614b070 0) Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.551.0 2 Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.552.0 (null) Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=552 Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.552.0) Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 300 aborted by fencing-level.fl-an-a02n01-3 'create': Configuration change | cib=0.552.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology complete=true Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.552.0 for /cib/configuration/fencing-topology Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (1 active entry) Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-83.raw Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.552.0 of the CIB to disk (digest: 8c043735124c85f316ccdbc7620fe962) Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.Js4de7 (digest: /var/lib/pacemaker/cib/cib.bLtaG4) Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node1 isn't managed Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node2 isn't managed Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv22-machineinterface isn't managed Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv24-inventory isn't managed Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv21-tickets isn't managed Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node1 isn't managed Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node2 isn't managed Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv19-uls isn't managed Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv20-fileprocessor isn't managed Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv23-orderprocessing isn't managed Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n02 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now cef2c24c7e11f1292c46d7137f454a30 (reload:3.10.2) 0:1;33:256:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu04 on an-a02n02 changed: hash was dcfe282eb9c3d7f7189c538ed5e16421 vs. now 6b32ecd9197e2963fbde8b23a095287a (reload:3.10.2) 0:1;36:258:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu04 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu04_clear_failcount_0 Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node2 allocated to an-a02n01: active Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node2 allocated to an-a02n01: active Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node1 allocated to an-a02n01: active Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node1 allocated to an-a02n01: active Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv13-psql allocated to no node: inactive Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv14-file-processor allocated to no node: inactive Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv15-tickets allocated to no node: inactive Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv16-machine-interface allocated to no node: inactive Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv17-order-processing allocated to no node: inactive Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv18-inventory allocated to no node: inactive Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv19-uls allocated to an-a02n01: active Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv20-fileprocessor allocated to an-a02n01: active Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv21-tickets allocated to an-a02n01: active Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv22-machineinterface allocated to an-a02n01: active Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv23-orderprocessing allocated to an-a02n01: active Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv24-inventory allocated to an-a02n01: active Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu03 allocated to no node: inactive Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu04 allocated to no node: inactive Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu03 allocated to no node: inactive Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu04 allocated to no node: inactive Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (stage6) notice: Delaying fencing operations until there are resources to manage Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started unmanaged) Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started unmanaged) Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started unmanaged) Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started unmanaged) Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped unmanaged) Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped unmanaged) Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped unmanaged) Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped unmanaged) Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped unmanaged) Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped unmanaged) Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started unmanaged) Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started unmanaged) Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started unmanaged) Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started unmanaged) Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started unmanaged) Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started unmanaged) Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped unmanaged) Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped unmanaged) Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped unmanaged) Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped unmanaged) Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 301, saving inputs in /var/lib/pacemaker/pengine/pe-input-314.bz2 Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 301 (ref=pe_calc-dc-1627090834-1049) derived from /var/lib/pacemaker/pengine/pe-input-314.bz2 Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 301 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-314.bz2): Complete Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 18:40:34 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2) Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.552.0 2 Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.553.0 (null) Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=553 Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/configuration/fencing-topology: Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.553.0) Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 301 aborted by fencing-level.fl-an-a02n02-3 'create': Configuration change | cib=0.553.0 source=te_update_diff_v2:474 path=/cib/configuration/fencing-topology complete=true Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (update_fencing_topology) info: Re-initializing fencing topology after top-level create operation 0.553.0 for /cib/configuration/fencing-topology Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n01 (0 active entries) Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 1 active fencing level Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: No fencing topology found for an-a02n02 (1 active entry) Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 1 active fencing level Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 2 active fencing levels Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 2 active fencing levels Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-84.raw Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.553.0 of the CIB to disk (digest: da8c5ceedc442ed7108e438014bc3293) Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.2XgH9H (digest: /var/lib/pacemaker/cib/cib.tmDLhL) Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node1 isn't managed Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource delay_node2 isn't managed Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv22-machineinterface isn't managed Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv24-inventory isn't managed Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv21-tickets isn't managed Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node1 isn't managed Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource ipmilan_node2 isn't managed Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv19-uls isn't managed Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv20-fileprocessor isn't managed Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (native_add_running) info: resource srv23-orderprocessing isn't managed Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n02 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now cef2c24c7e11f1292c46d7137f454a30 (reload:3.10.2) 0:1;33:256:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu04 on an-a02n02 changed: hash was dcfe282eb9c3d7f7189c538ed5e16421 vs. now 6b32ecd9197e2963fbde8b23a095287a (reload:3.10.2) 0:1;36:258:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu04 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu04_clear_failcount_0 Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node2 allocated to an-a02n01: active Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node2 allocated to an-a02n01: active Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource ipmilan_node1 allocated to an-a02n01: active Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource delay_node1 allocated to an-a02n01: active Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv13-psql allocated to no node: inactive Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv14-file-processor allocated to no node: inactive Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv15-tickets allocated to no node: inactive Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv16-machine-interface allocated to no node: inactive Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv17-order-processing allocated to no node: inactive Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv18-inventory allocated to no node: inactive Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv19-uls allocated to an-a02n01: active Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv20-fileprocessor allocated to an-a02n01: active Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv21-tickets allocated to an-a02n01: active Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv22-machineinterface allocated to an-a02n01: active Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv23-orderprocessing allocated to an-a02n01: active Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource srv24-inventory allocated to an-a02n01: active Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu03 allocated to no node: inactive Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node1_an-pdu04 allocated to no node: inactive Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu03 allocated to no node: inactive Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Unmanaged resource apc_snmp_node2_an-pdu04 allocated to no node: inactive Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (stage6) notice: Delaying fencing operations until there are resources to manage Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started unmanaged) Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started unmanaged) Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started unmanaged) Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started unmanaged) Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped unmanaged) Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped unmanaged) Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped unmanaged) Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped unmanaged) Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped unmanaged) Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped unmanaged) Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started unmanaged) Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started unmanaged) Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started unmanaged) Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started unmanaged) Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started unmanaged) Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started unmanaged) Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu03 (Stopped unmanaged) Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node1_an-pdu04 (Stopped unmanaged) Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu03 (Stopped unmanaged) Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave apc_snmp_node2_an-pdu04 (Stopped unmanaged) Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 302, saving inputs in /var/lib/pacemaker/pengine/pe-input-315.bz2 Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 302 (ref=pe_calc-dc-1627090838-1050) derived from /var/lib/pacemaker/pengine/pe-input-315.bz2 Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-controld [121633] (run_graph) notice: Transition 302 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-315.bz2): Complete Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd Jul 23 18:40:38 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd Jul 23 18:40:43 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_ping) info: Reporting our current digest to an-a02n01: 96ab29d97550d37061067b5c58057b44 for 0.553.0 (0x5619762e5ae0 0) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.553.0 2 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.554.0 90a8f9490deacbbaef660c64c39b29a6 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @epoch=554 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-maintenance-mode']: @value=false Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-1']: @id=fl-an-a02n01-1, @index=1 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-1']: @id=fl-an-a02n02-1, @index=1 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-2']: @id=fl-an-a02n01-2, @index=2 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-2']: @id=fl-an-a02n02-2, @index=2 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n01-3']: @id=fl-an-a02n01-3, @index=3 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/configuration/fencing-topology/fencing-level[@id='fl-an-a02n02-3']: @id=fl-an-a02n02-3, @index=3 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_replace_notify) info: Local-only Replace: 0.554.0 from an-a02n01 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section configuration: OK (rc=0, origin=an-a02n01/cibadmin/2, version=0.554.0) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 302 aborted by cib-bootstrap-options-maintenance-mode doing modify maintenance-mode=false: Configuration change | cib=0.554.0 source=te_update_diff_v2:474 path=/cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-maintenance-mode'] complete=true Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_replaced_cb) notice: Updating all attributes after cib_refresh_notify event Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Unset DC. Was an-a02n01 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 617 with 1 change for last-failure-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 618 with 1 change for fail-count-apc_snmp_node2_an-pdu03#start_0 (id n/a, set n/a) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Processed 1 private change for #attrd-protocol, id=n/a, set=n/a Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 619 with 1 change for fail-count-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 620 with 1 change for last-failure-apc_snmp_node1_an-pdu03#start_0 (id n/a, set n/a) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 621 with 1 change for fail-count-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 622 with 1 change for last-failure-apc_snmp_node2_an-pdu04#start_0 (id n/a, set n/a) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 623 with 1 change for fail-count-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (write_attribute) info: Sent CIB request 624 with 1 change for last-failure-apc_snmp_node1_an-pdu04#start_0 (id n/a, set n/a) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-controld [121633] (election_check) info: election-DC won by local node Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-controld [121633] (do_log) info: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) notice: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/2080) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/2081) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 1 from fencing topology for an-a02n01 (2 active levels remaining) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 1 from fencing topology for an-a02n02 (2 active levels remaining) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 2 from fencing topology for an-a02n01 (2 active levels remaining) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 2 from fencing topology for an-a02n02 (2 active levels remaining) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 3 from fencing topology for an-a02n01 (2 active levels remaining) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n01 has 3 active fencing levels Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_remove) info: Removed level 3 from fencing topology for an-a02n02 (2 active levels remaining) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-fenced [121629] (stonith_level_register) info: Target an-a02n02 has 3 active fencing levels Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-85.raw Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/617) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/618) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/619) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/620) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/621) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.554.0 of the CIB to disk (digest: 277f183a104b1a808867b2095e7c5151) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.KGh88g (digest: /var/lib/pacemaker/cib/cib.8fNg6h) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/2080, version=0.554.0) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.554.0 2 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.554.1 (null) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=1 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/2081, version=0.554.1) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.554.1 2 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.554.2 (null) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=2 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/617, version=0.554.2) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 617 result for last-failure-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.554.2 2 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.554.3 (null) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=3 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/618, version=0.554.3) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 618 result for fail-count-apc_snmp_node2_an-pdu03#start_0: OK | rc=0 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu03#start_0[an-a02n01]=(null) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.554.3 2 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.554.4 (null) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=4 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/619, version=0.554.4) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 619 result for fail-count-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.554.4 2 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.554.5 (null) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=5 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/620, version=0.554.5) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 620 result for last-failure-apc_snmp_node1_an-pdu03#start_0: OK | rc=0 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]=(null) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.554.5 2 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.554.6 (null) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=6 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/621, version=0.554.6) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 621 result for fail-count-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/622) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/623) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/attrd/624) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.554.6 2 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.554.7 (null) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=7 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/622, version=0.554.7) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 622 result for last-failure-apc_snmp_node2_an-pdu04#start_0: OK | rc=0 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node2_an-pdu04#start_0[an-a02n01]=(null) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.554.7 2 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.554.8 (null) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=8 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/623, version=0.554.8) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 623 result for fail-count-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * fail-count-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.554.8 2 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.554.9 (null) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=9 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_takeover) info: Taking over DC status for this partition Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/attrd/624, version=0.554.9) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: CIB update 624 result for last-failure-apc_snmp_node1_an-pdu04#start_0: OK | rc=0 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-attrd [121631] (attrd_cib_callback) info: * last-failure-apc_snmp_node1_an-pdu04#start_0[an-a02n01]=(null) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/2084, version=0.554.9) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/2085) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/2085, version=0.554.9) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/2087) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/2087, version=0.554.9) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/2089) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Not making join-42 offer to inactive node an-a02n02 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-controld [121633] (join_make_offer) info: Sending join-42 offer to an-a02n01 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-controld [121633] (do_dc_join_offer_all) info: Waiting on join-42 requests from 1 outstanding node Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/2089, version=0.554.9) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/2091) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section crm_config: OK (rc=0, origin=an-a02n01/crmd/2091, version=0.554.9) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-controld [121633] (update_dc) info: Set DC to an-a02n01 (3.10.2) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Digest matched on replace from an-a02n01: 0b2a751b40d35fe86f8c7fe2042ccaa1 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_replace) info: Replaced 0.554.9 with 0.554.9 from an-a02n01 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_replace operation for section 'all': OK (rc=0, origin=an-a02n01/crmd/2094, version=0.554.9) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/2095) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_backup) info: Archived previous version as /var/lib/pacemaker/cib/cib-86.raw Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/2095, version=0.554.9) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Wrote version 0.554.0 of the CIB to disk (digest: 277f183a104b1a808867b2095e7c5151) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-based [121628] (cib_file_write_with_digest) info: Reading cluster configuration file /var/lib/pacemaker/cib/cib.nVyppt (digest: /var/lib/pacemaker/cib/cib.MrSpAs) Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[359837] exited with status 0 Jul 23 18:41:37 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[359839] exited with status 0 Jul 23 18:41:38 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[359840] exited with status 0 Jul 23 18:41:38 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[359841] exited with status 0 Jul 23 18:41:38 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[359842] exited with status 0 Jul 23 18:41:38 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[359843] exited with status 0 Jul 23 18:41:38 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[359845] exited with status 0 Jul 23 18:41:39 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[359850] exited with status 0 Jul 23 18:41:39 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[359860] exited with status 0 Jul 23 18:41:39 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[359863] exited with status 0 Jul 23 18:41:39 an-a02n01.alteeve.com pacemaker-controld [121633] (controld_delete_node_state) info: Deleting resource history for node an-a02n01 (via CIB call 2096) | xpath=//node_state[@uname='an-a02n01']/lrm Jul 23 18:41:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_delete operation for section //node_state[@uname='an-a02n01']/lrm to all (origin=local/crmd/2096) Jul 23 18:41:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.554.9 2 Jul 23 18:41:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.554.10 (null) Jul 23 18:41:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: -- /cib/status/node_state[@id='1']/lrm[@id='1'] Jul 23 18:41:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=10 Jul 23 18:41:39 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_delete operation for section //node_state[@uname='an-a02n01']/lrm: OK (rc=0, origin=an-a02n01/crmd/2096, version=0.554.10) Jul 23 18:41:39 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[359866] exited with status 0 Jul 23 18:41:39 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_delay_metadata_1[359867] exited with status 0 Jul 23 18:41:40 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[359868] exited with status 0 Jul 23 18:41:40 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[359869] exited with status 0 Jul 23 18:41:40 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[359870] exited with status 0 Jul 23 18:41:40 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[359871] exited with status 0 Jul 23 18:41:40 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: fence_ipmilan_metadata_1[359873] exited with status 0 Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[359875] exited with status 0 Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[359885] exited with status 0 Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-controld [121633] (action_synced_wait) info: server_meta-data_0[359892] exited with status 0 Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/2097) Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.554.10 2 Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.554.11 (null) Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=11 Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']: Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/2097, version=0.554.11) Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-controld [121633] (abort_transition_graph) info: Transition 302 aborted: Peer Cancelled | source=do_te_invoke:129 complete=true Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/2100) Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/2101) Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section cib to all (origin=local/crmd/2102) Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section nodes: OK (rc=0, origin=an-a02n01/crmd/2100, version=0.554.11) Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.554.11 2 Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.554.12 (null) Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=12 Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section status: OK (rc=0, origin=an-a02n01/crmd/2101, version=0.554.12) Jul 23 18:41:41 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Completed cib_modify operation for section cib: OK (rc=0, origin=an-a02n01/crmd/2102, version=0.554.12) Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status_fencing) info: Node an-a02n01 is active Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (determine_online_status) info: Node an-a02n01 is online Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu03 on an-a02n02 changed: hash was 3fd89635ef735220dd13344790fbf853 vs. now cef2c24c7e11f1292c46d7137f454a30 (reload:3.10.2) 0:1;33:256:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (custom_action) warning: apc_snmp_node1_an-pdu03_clear_failcount_0 on an-a02n02 is unrunnable (node is offline) Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu03_clear_failcount_0 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 17:16:20 2021 | rc=1 id=apc_snmp_node1_an-pdu03_last_failure_0 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_digest_cmp) info: Parameters to 0ms-interval start action for apc_snmp_node1_an-pdu04 on an-a02n02 changed: hash was dcfe282eb9c3d7f7189c538ed5e16421 vs. now 6b32ecd9197e2963fbde8b23a095287a (reload:3.10.2) 0:1;36:258:0:789131c0-5164-4a52-9822-30c5026c3827 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (custom_action) warning: apc_snmp_node1_an-pdu04_clear_failcount_0 on an-a02n02 is unrunnable (node is offline) Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pe__clear_failcount) notice: Clearing failure of apc_snmp_node1_an-pdu04 on an-a02n02 because resource parameters have changed | apc_snmp_node1_an-pdu04_clear_failcount_0 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (unpack_rsc_op_failure) warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 17:16:33 2021 | rc=1 id=apc_snmp_node1_an-pdu04_last_failure_0 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv13-psql cannot run anywhere Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv14-file-processor cannot run anywhere Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv15-tickets cannot run anywhere Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv16-machine-interface cannot run anywhere Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv17-order-processing cannot run anywhere Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__native_allocate) info: Resource srv18-inventory cannot run anywhere Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for ipmilan_node2 on an-a02n01 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for delay_node2 on an-a02n01 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for ipmilan_node1 on an-a02n01 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for delay_node1 on an-a02n01 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for srv19-uls on an-a02n01 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for srv20-fileprocessor on an-a02n01 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for srv21-tickets on an-a02n01 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for srv22-machineinterface on an-a02n01 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for srv23-orderprocessing on an-a02n01 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for srv24-inventory on an-a02n01 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu03 on an-a02n01 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node1_an-pdu04 on an-a02n01 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu03 on an-a02n01 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (RecurringOp) info: Start recurring monitor (60s) for apc_snmp_node2_an-pdu04 on an-a02n01 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node2 (Started an-a02n01) Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node2 (Started an-a02n01) Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave ipmilan_node1 (Started an-a02n01) Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave delay_node1 (Started an-a02n01) Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv13-psql (Stopped) Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv14-file-processor (Stopped) Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv15-tickets (Stopped) Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv16-machine-interface (Stopped) Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv17-order-processing (Stopped) Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv18-inventory (Stopped) Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv19-uls (Started an-a02n01) Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv20-fileprocessor (Started an-a02n01) Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv21-tickets (Started an-a02n01) Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv22-machineinterface (Started an-a02n01) Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv23-orderprocessing (Started an-a02n01) Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (rsc_action_default) info: Leave srv24-inventory (Started an-a02n01) Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu03 ( an-a02n01 ) Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node1_an-pdu04 ( an-a02n01 ) Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node2_an-pdu03 ( an-a02n01 ) Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (log_list_item) notice: Actions: Start apc_snmp_node2_an-pdu04 ( an-a02n01 ) Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-schedulerd[121632] (pcmk__log_transition_summary) notice: Calculated transition 303, saving inputs in /var/lib/pacemaker/pengine/pe-input-316.bz2 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-controld [121633] (do_state_transition) info: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-controld [121633] (do_te_invoke) info: Processing graph 303 (ref=pe_calc-dc-1627090902-1056) derived from /var/lib/pacemaker/pengine/pe-input-316.bz2 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation ipmilan_node2_monitor_60000 locally on an-a02n01 | action 5 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of monitor operation for ipmilan_node2 on an-a02n01 | transition_key=5:303:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=ipmilan_node2_monitor_60000 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/2105) Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation delay_node2_monitor_60000 locally on an-a02n01 | action 8 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of monitor operation for delay_node2 on an-a02n01 | transition_key=8:303:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=delay_node2_monitor_60000 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/2106) Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation ipmilan_node1_monitor_60000 locally on an-a02n01 | action 11 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of monitor operation for ipmilan_node1 on an-a02n01 | transition_key=11:303:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=ipmilan_node1_monitor_60000 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-controld [121633] (te_rsc_command) notice: Initiating monitor operation delay_node1_monitor_60000 locally on an-a02n01 | action 14 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-controld [121633] (do_lrm_rsc_op) notice: Requesting local execution of monitor operation for delay_node1 on an-a02n01 | transition_key=14:303:0:789131c0-5164-4a52-9822-30c5026c3827 op_key=delay_node1_monitor_60000 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: --- 0.554.12 2 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: Diff: +++ 0.554.13 (null) Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib: @num_updates=13 Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource Jul 23 18:41:42 an-a02n01.alteeve.com pacemaker-based [121628] (cib_perform_op) info: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ipmilan_node2']: