Configure Multiple Fencing Devices Using pcs

This describes how to Configure Multiple Fencing Devices (using that page's example of IPMI followed by two switched PDUs) via the pcs tool.

Starting Point

For a frame of reference, the cluster starts with this configuration:

Cluster Name: an-cluster-03
Corosync Nodes:
 pcmk-1 pcmk-2 
Pacemaker Nodes:
 pcmk-1 pcmk-2 

Resources: 

Stonith Devices: 
Fencing Levels: 

Location Constraints:
Ordering Constraints:
Colocation Constraints:

Cluster Properties:
 cluster-infrastructure: corosync
 dc-version: 1.1.14-70404b0
 no-quorum-policy: ignore
 stonith-enabled: false

We will need to make a few assumptions about our example cluster;

• It is a two-node cluster with the node names "pcmk-1" and "pcmk-2".
• The two PDUs are accessible at the network address "pdu-1" and "pdu-2" and will be accessed using the "fence_apc_snmp" fence agent.
• The fencing details for "pcmk-1" are:
  • IPMI device address is "pcmk-1.ipmi", the login name is "admin" and the password is "secret".
  • Its power supplies are connected to "pdu-1" on port 1 and "pdu-2" on port 1.
• The fencing details for "pcmk-2" are:
  • IPMI device address is "pcmk-2.ipmi", the login name is "admin" and the password is "secret".
  • Its power supplies are connected to "pdu-1" on port 2 and "pdu-2" on port 2.

Please adapt the example below to the names, addresses, credentials and fence agents you are using in your cluster.

Configure IPMI fencing

Configure the IPMI fence device for "pcmk-1":

pcs stonith create fence_pcmk1_ipmi fence_ipmilan \
    pcmk_host_list="pcmk-1" ipaddr="pcmk-1.ipmi" \
    action="reboot" login="admin" passwd="secret" delay=15 \
    op monitor interval=60s

Configure the IPMI fence device for "pcmk-2":

pcs stonith create fence_pcmk2_ipmi fence_ipmilan \
    pcmk_host_list="pcmk-2" ipaddr="pcmk-2.ipmi" \
    action="reboot" login="admin" passwd="secret" delay=15 \
    op monitor interval=60s

Configure PDU fencing

If using Pacemaker 1.1.14 or newer

Configure the PDU fence devices for "pcmk-1":

pcs stonith create fence_pcmk1_psu1 fence_apc_snmp \
    pcmk_host_list="pcmk-1" ipaddr="pdu-1" \
    port="1" op monitor interval="60s"
pcs stonith create fence_pcmk1_psu2 fence_apc_snmp \
    pcmk_host_list="pcmk-1" ipaddr="pdu-2" \
    port="1" power_wait="5" op monitor interval="60s"

Configure the PDU fence devices for "pcmk-2":

pcs stonith create fence_pcmk2_psu1 fence_apc_snmp \
    pcmk_host_list="pcmk-2" ipaddr="pdu-1" \
    port="2" op monitor interval="60s"
pcs stonith create fence_pcmk2_psu2 fence_apc_snmp \
    pcmk_host_list="pcmk-2" ipaddr="pdu-2" \
    port="2" power_wait="5" op monitor interval="60s"

If using an older Pacemaker version

The PDU fencing is more complicated with version of Pacemaker older than 1.1.14.

In order for fencing to work when two separate PDUs are used, we must ensure that there is a period of time where both PDUs have their ports powered off at the same time. To do this, we need to setup four primitives; One for each device set to an "off" action and another for each device set to an "on" action. This will allow us to call "pdu1:x off -> pdu2:x off -> pdu1:x on -> pdu2:x on".

Note: Prior to version 1.1.10, 'action="..."' was ignored. If you have a version of pacemaker below this (including 1.1.10 rc5 and older), you will need to replace 'action="..."' with 'pcmk_reboot_action="..."'.

Now configure the four PDU fence methods for "pcmk-1". Note that we've added 'power_wait="5"' to the second PDU's "off" action. Later, we will stitch these actions together and this argument will tell pacemaker to wait 5 seconds after turning off the second PDU before restoring power. This gives plenty of time for the node's power supplies to completely drain, ensuring that the node loses power. You will also note that the "monitor" operation is only set on the "off" actions. There is no need to monitor the status of the "on" actions as it would be redundant.

# Node 1 - Off
pcs stonith create fence_pcmk1_psu1_off fence_apc_snmp \
    pcmk_host_list="pcmk-1" ipaddr="pdu-1" action="off" \
    port="1" op monitor interval="60s"
pcs stonith create fence_pcmk1_psu2_off fence_apc_snmp \
    pcmk_host_list="pcmk-1" ipaddr="pdu-2" action="off" \
    port="1" power_wait="5" \
    op monitor interval="60s"

# Node 1 - on
pcs stonith create fence_pcmk1_psu1_on fence_apc_snmp \
    pcmk_host_list="pcmk-1" ipaddr="pdu-1" action="on" \
    port="1"
pcs stonith create fence_pcmk1_psu2_on fence_apc_snmp \
    pcmk_host_list="pcmk-1" ipaddr="pdu-2" action="on" \
    port="1"

Finally, configure the four PDU fence methods for "pcmk-2";

# Node 2 - Off
pcs stonith create fence_pcmk2_psu1_off fence_apc_snmp \
    pcmk_host_list="pcmk-2" ipaddr="pdu-1" action="off" \
    port="2" op monitor interval="60s"
pcs stonith create fence_pcmk2_psu2_off fence_apc_snmp \
    pcmk_host_list="pcmk-2" ipaddr="pdu-2" action="off" \
    port="2" power_wait="5" \
    op monitor interval="60s"

# Node 2 - on
pcs stonith create fence_pcmk2_psu1_on fence_apc_snmp \
    pcmk_host_list="pcmk-2" ipaddr="pdu-1" action="on" \
    port="2"
pcs stonith create fence_pcmk2_psu2_on fence_apc_snmp \
    pcmk_host_list="pcmk-2" ipaddr="pdu-2" action="on" \
    port="2"

Enable fencing

Now that fencing is configured, we can enable the "stonith-enabled" property.

pcs property set stonith-enabled=true

Use IPMI as first fencing level

The next step is to tell pacemaker what order we want the fencing methods to run, using fencing levels.

Each fencing level may have one or more fence devices. When fencing is required, Pacemaker will try each level in sequence, stopping at the first level that succeeds. Therefore, separate levels function as a "fallback" mechanism (logical "or"). At any given level, all the devices in that level will be tried in succession, and all must succeed for the level to succeed (logical "and").

In this example, tell pacemaker that the IPMI-based fence devices are the first methods to use:

pcs stonith level add 1 pcmk-1 fence_pcmk1_ipmi
pcs stonith level add 1 pcmk-2 fence_pcmk2_ipmi

Use PDUs as second fencing level

If using Pacemaker 1.1.14 or newer

Next, we tell pacemaker to use the switched PDUs as the second method:

pcs stonith level add 2 pcmk-1 fence_pcmk1_psu1,fence_pcmk1_psu2
pcs stonith level add 2 pcmk-2 fence_pcmk2_psu1,fence_pcmk2_psu2

Pacemaker will automatically remap sequential reboots in a fencing level to all-off-then-all-on.

If using an older Pacemaker version

Next, we tell pacemaker to use the switched PDUs as the second method:

pcs stonith level add 2 pcmk-1 fence_pcmk1_psu1_off,fence_pcmk1_psu2_off,fence_pcmk1_psu1_on,fence_pcmk1_psu2_on
pcs stonith level add 2 pcmk-2 fence_pcmk2_psu1_off,fence_pcmk2_psu2_off,fence_pcmk2_psu1_on,fence_pcmk2_psu2_on

Test

You can (and should!) test this my unplugging the IPMI interface for "pcmk-1" and then crashing it, triggering "pcmk-2" to call a fence against it. After the IPMI interface times out, you should see PDU 1, port 1 turn off, then PDU 2, port 1 turn off, the crashed node will power down, then PDU 1 port 1 should turn back on and finally PDU 2 port 1 should turn back on. If you configured your server's BIOS to power on after power loss or to return to last state after power loss, your server should start to power back on.