Using crm simulate
crm_simulate is a Pacemaker command-line tool for simulating the same logic Pacemaker uses to respond to a particular cluster configuration and status.
As always, the man page is the primary documentation, and should be consulted for further details. This page aims for a better conceptual explanation and practical examples.
Replaying cluster decision-making logic
At any given time, one node in a Pacemker cluster will be elected DC, and that node will run Pacemaker's policy engine to make decisions.
Each time decisions need to be made (a "transition"), the DC will have logs like "pengine: ... saving inputs in <filename>". You can grab this file and replay the cluster logic to see why particular decisions were made. The file contains the live cluster configuration at that moment, so you can also look at it directly to see the value of node attributes, etc., at that time.
The simplest usage is (replacing $FILENAME with the actual file name):
crm_simulate -Sx $FILENAME
That will show the cluster state when the process started, the actions that need to be taken ("Transition Summary"), and the resulting cluster state if the actions succeed. With recent versions of Pacemaker, most actions will have a brief description of why they were required.
Why decisions were made
To get further insight into the "why", it gets user-unfriendly very quickly. If you add
-s, you will also see all the scores that went into the decision-making. The node with the highest cumulative score for a resource will run it. You can look for
-INFINITY scores in particular to see where complete bans came into effect.
You can also add
-VVVV to get more detailed messages about what's happening under the hood. You can add up to two more V's even, but that's usually useful only if you're a masochist or tracing through the source code.
You can make changes to the file and simulate it again, to see how Pacemaker would react differently. You can edit the XML by hand, or use the command-line tools such as cibadmin with the CIB_file environment variable set to the filename first, or use higher-level tool support (such as the "-f" option to pcs).
You can also inject node failures and/or action failures into the simulation; see the crm_simulate man page for more details.
Visualizing the action sequence
Another sometimes handy feature is the ability to generate a visual graph of the actions needed:
crm_simulate -Ssx $FILENAME -D $FILENAME.dot dot $FILENAME.dot -Tsvg > $FILENAME.svg
Then view the svg in your browser or app of choice.