Page MenuHomeClusterLabs Projects

Pacemaker Feature Set
Updated 78 Days AgoPublic

Besides a release version, Pacemaker has a separate CRM feature set. This version number applies to the communication between cluster nodes, and is used to avoid problems in mixed-version clusters.

Most of the time, CRM feature sets are for Pacemaker's internal use. However, they can also be used by resource agents and other external code to detect support for various features. The table below lists those features.

Pacemaker CRM Feature Sets
CRM feature setAvailable in Pacemaker releaseFeatures
3.19.02.1.7Pacemaker supports CIB transactions and node-pending-timeout cluster property; crm_node supports --output-as (text, none, or xml) and --output-to arguments
3.17.42.1.6utilization attributes may be set as transient; alerts and alert recipients support enabled meta-attribute in CIB; attrd_updater supports --pattern, --utilization, and --wait arguments; crm_attribute supports --wait argument, and supports --pattern argument with --update and --delete and with permanent node attributes; crm_mon supports --interval argument with text and xml output; crm_resource supports --element option; crm_shadow, pacemaker-based, pacemaker-controld, pacemaker-execd, and pacemaker-fenced support --output-as (text, none, or xml) and --output-to arguments
3.16.12.1.5crm_error supports --output-as (text, none, or xml) and --output-to arguments; crm_attribute supports --pattern with --update, and with --delete and --query when used with --lifetime=reboot; crm_attribute with --query and --node but not --name shows all node attributes on the given node
3.15.12.1.5Supported feature set stored as #feature-set node attribute
3.15.02.1.3Pacemaker supports setting multiple-active to stop_unexpected
3.14.02.1.3Pacemaker supports allow-unhealthy-nodes resource meta-attribute
3.13.42.1.3attrd_updater supports --output-as (text, none, or xml) and --output-to options
3.13.32.1.3crm_attribute supports --output-as (text, none, or xml) and --output-to options
3.13.22.1.3crm_rule supports --output-as (text, none, or xml) and --output-to options, and specifying -r option more than once
3.13.12.1.3cibadmin supports --show-access option
3.11.02.1.2fence_watchdog is available (allowing watchdog fencing to be restricted to specific nodes)
3.10.22.1.1crm_resource --validate and --force-check arguments accept an optional argument for OCF check level
3.10.12.1.1pacemakerd accepts --output-as (text, none, or xml) and --output-to options, mainly for use with the --features argument
3.10.02.1.0Pacemaker supports OCF resource agents' reload-agent action
3.9.02.1.0"Promoted" and "Unpromoted" are accepted as role names; crm_attribute accepts the --promotion option; crm_resource accepts the --promoted option; promotable clone notify actions will have notify_promoted_resource, notify_unpromoted_resource, notify_promoted_uname, and notify_unpromoted_uname environment variables; crm_mon XML output for bans includes a "promoted-only" attribute
3.8.02.1.0crm_simulate accepts --show-attrs and --show-failcounts options
3.7.42.1.0crm_simulate accepts --output-as (text, none, or xml) and --output-to options, and its output is formatted more similarly to crm_mon's
3.7.32.1.0CIB_file="-" for standard input is supported by tools that support CIB_file
3.7.22.1.0crm_verify accepts --output-as (text, none, or xml) and --output-to options; crmadmin drops -E/--election and -K/--kill options
3.7.12.1.0support for node type argument to crmadmin --nodes
3.7.02.1.0support for "critical" resource meta-attribute and "influence" colocation option
3.6.42.1.0crm_resource offers --digests option
3.6.32.1.0crm_resource accepts --output-as (text or xml) and --output-to
3.6.22.1.0crmadmin accepts --output-as (text or xml) and --output-to
3.6.12.0.5cibsecret returns standard Pacemaker exit status codes
3.6.02.0.5
3.5.02.0.5"integer" supported as value for "type" in rules, and "number" compares as double-precision floating-point value
3.4.12.0.5crmadmin supports -P/--pacemakerd and -i/--ipc-name options
3.4.02.0.5op_expression and rsc_expression supported in rules; "demote" supported as value for "no-quorum-policy" and "on-fail" options; priority-fencing-delay cluster option supported; stonith_admin supports --delay option; crm_mon supports --node and --resource options; crm_rule --check can now check "date_spec" entries that contain "years" but not "moon"
3.3.02.0.4fence-reaction, shutdown-lock, and shutdown-lock-limit cluster options supported; crm_mon supports --include and --exclude options
3.2.02.0.3new tool crm_rule; crm_mon and stonith_admin support --output-as and --output-to options
3.1.02.0.0

(Note: in some cases, the specified features were available in the prior Pacemaker release, but the listed feature set version is the first that includes them.)

How CRM feature sets are numbered

The major version increases when nodes with different versions would not work together. Rolling upgrades would not be possible in this case.

The minor version increases when mixed-version clusters are allowed only during rolling upgrades. If one of the older nodes drops out of the cluster for any reason, it will not be able to rejoin until it is upgraded. In this case, you should run a mixed-version cluster only during a small upgrade window.

The minor-minor version number is ignored by Pacemaker. Clusters with different versions should work without problems. This number allows resource agents and other external code to detect cluster support for various features.

Pacemaker ensures that the longest-running node is the cluster’s DC. This ensures new features are not enabled until all nodes are upgraded to support them.

In code

The feature set is defined as CRM_FEATURE_SET in include/crm/crm.h.

Last Author
kgaillot
Last Edited
Jan 10 2024, 5:47 PM