Pacemaker 2.1 Changes

The first release candidate for Pacemaker 2.1.0 has been released, with the final release expected by June.

Pacemaker 2.1 is a minor Pacemaker release that preserves backward compatibility for both rolling upgrades and the C API, which means there should be no major concerns for individuals or distributions who upgrade. The minor version bump is intended to draw attention to larger changes than usual, including changes that may affect any custom scripts users have written.

As with Pacemaker 2.0, Corosync 2 or later is supported as the underlying cluster layer, and rolling upgrades from Pacemaker 1.1.11 or later are possible as long as the cluster layer stays at the same Corosync major version (2 or 3).

Since backward compatibility will be maintained, there will be no need to backport future changes to the 2.0 series, and there will be no more 2.0 series releases.

New Features

OCF 1.1 Support

Pacemaker now supports the recently adopted OCF Resource Agent API 1.1 standard. Related changes are described in the relevant sections of this page.

Noncritical resources

Colocation constraints may take a new "influence" option that determines whether the dependent resource influences the location of the main resource, if the main resource is already active. The default of true preserves the previous behavior. Setting it to false has the effect of making the dependent resource stop if it reaches its migration-threshold in failures, rather than cause the main resource to move to another node.

Resources may take a new "critical" meta-attribute that serves as a default for all colocation constraints involving the resource as the dependent, as well as all groups involving the resource.

As an example, a large, important database that takes a long time to start and stop may have a less important, occasionally used reporting tool colocated or grouped with it. In this situation, it might be preferable to stop the reporting tool if it fails too many times, rather than cause a database outage to move both resources. In this case, either the colocation constraint could be marked with influence=false, or the reporting tool resource could be give the meta-attribute critical=false, to achieve the desired effect.

Sync before panic

Pacemaker's existing PCMK_panic_action environment variable (typically set in /etc/sysconfig/pacemaker, /etc/default/pacemaker, or similar) determines how Pacemaker responds to certain critical errors. Previously, it could be set to "crash" to trigger a kernel crash (generating a kernel dump for investigation) or "reboot" (actually any other value) to trigger a reboot (the default behavior).

Now, it may also take the values "sync-crash" or "sync-reboot", which will attempt to sync the host's disks before crashing or rebooting. This can be helpful to record any log messages cached just before the failure, however it runs the risk of the sync hanging (for example if a disk controller has issues), which would leave the host running after a critical error.

Build process changes

Dependency versions

The minimum dependency versions required to build Pacemaker have been raised to these (all released 6 or more years ago and available in recent and even many not-so-recent versions of all major Linux distributions):

  • Python 3.2 or later (support for Python 2.7 has been removed)
  • glib 2.32.0 or later
  • libqb 0.17.0 or later
  • GnuTLS 2.12.0 or later (to enable Pacemaker Remote support)
  • rpm 4.11.0 or later (if building RPMs)
  • C library must provide setenv() and unsetenv()

configure script: changes in option defaults

  • The default for --enable-legacy-links is now "no" instead of "yes". This means that symbolic links at the pacemaker 1.1 daemon names, such as crmd linked to pacemaker-controld, will no longer be installed by default. However, a link to the old pacemaker_remoted name will now always be created, regardless of the value of this option, to ensure that container images with Pacemaker 2 can be used in bundles on cluster nodes running Pacemaker 1. This option is now deprecated, and will be removed in a future release.

configure script: new options

  • If set to "yes", --enable-compat-2.0 (which defaults to "no") will revert certain output changes, to maintain compatibility with tools, scripts, and resource agents that rely on the previous output. The exact changes that are reverted will be noted below with the changes themselves.
  • --with-concurrent-fencing-default can be used to set the default for the concurrent-fencing option. The default of "false" preserves the previous behavior. Setting it to "true" replaces the previous method of setting -DDEFAULT_CONCURRENT_FENCING_TRUE in CFLAGS or CPPFLAGS before configuring (which will no longer have any effect).
  • --with-resource-stickiness-default can be used to set the default for the resource-stickiness resource meta-attribute. The default of "0" preserves the previous behavior. If positive, this value will be added as an explicit resource default in new CIBs (which means it will not affect any existing clusters, even in a rolling upgrade). Distributions and users who build from source can set this, if their users expect sticky behavior by default and are regularly surprised by resources moving around by default.
  • --with-gnutls can now be used to determine support for Pacemaker Remote and remote-tls-port. The default of "try" preserves the former behavior (enable support if a recent enough GnuTLS is available). "yes" will fail if support cannot be enabled, and "no" will disable support even if it's possible.

configure script: removed options

  • Remove support for the --with-acl option. Support for ACLs will now always be built. Setting --with-acl to "no" has been broken since the 1.1.16 release, and since no one's even noticed it until now, dropping the option seems like a better choice than fixing it.
  • Remove support for the deprecated --with-pkgname and --with-pkg-name options, which have no effect
  • Remove support for the --with-brand option, which was only needed for Publican (see "Documentation changes" below)
  • Remove support for the --enable-ansi option, which hasn't worked in a very long time
  • Remove support for the --enable-no-stack option, which builds only the scheduler

configure script: other changes

  • The configure script already supports a --with-runstatedir option that is ignored if the installed autoconf version supports the --runstatedir option. Until recently, only Debian patched its autoconf package to support --runstatedir, but autoconf 2.70 was released in December 2020 with support, so as time goes on, more builds will need to use the standard --runstatedir instead of --with-runstatedir.

Removed files

  • The clustermon.sh script, which was the pre-1.1.15 way of doing alerts
  • The nonfunctional ocf:pacemaker:pingd resource agent (the separate and widely used ocf:pacemaker:ping agent is unaffected)
  • The cibmon utility, which was a no-longer-useful holdover from the heartbeat days typically installed in /usr/libexec/pacemaker
  • The showscores.sh script, which was in the extra directory (not installed by default) and hasn't worked in several releases
  • The openstack.md documentation file, which was in the doc directory (not installed by default), not of general interest, and outdated

Other

  • The Pacemaker environment variables file (typically installed as /etc/sysconfig/pacemaker, /etc/default/pacemaker, etc., depending on OS and distribution) will now be installed when make install is run. If the file already exists, the new file will be installed with a .new file extension. Previously, the user had to install the file themselves, or use the make rpm command which would install the file via the generated RPM packages (which still works).
  • Remove support for the deprecated and undocumented SUSE_ACL_COMPAT compile-time option. (Official SUSE builds have never used this option.)
  • Make "subtree" the default for RPMDEST in GNUmakefile, which means that the "make rpm" target will put all artifacts in a dedicated "rpm/" subdirectory rather than a mix of the top-level directory and system defaults. You can run "make RPMDEST=toplevel rpm" to get the previous behavior, though that will be considered deprecated and eventually removed.

Logging changes

  • If Pacemaker is built with libqb 2.0 or later, the detail log will use millisecond-resolution timestamps. (If for some reason this causes a problem for some users, we could potentially make this behavior optional.)
  • Pacemaker now uses the OCF 1.1 role names, "Promoted" and "Unpromoted", in log messages. (If Pacemaker is built with the --enable-compat-2.0 configure option, the old names will still be used in some messages.)

Cluster Information Base (CIB) changes

  • Pacemaker now accepts the OCF 1.1 standard role names ("Promoted" and "Unpromoted") in addition to the legacy terms "Master" and "Slave", which are now deprecated, anywhere roles are specified in Pacemaker configuration (such as the "target-role" resource meta-attribute). Using the new names in "role" in <op>, <rsc_location>, or <resource_set>, or in "rsc-role" or "with-rsc-role" in <rsc_ticket> or <rsc_colocation>, requires CIB schema 3.7 (in other words, "cibadmin --upgrade" or an equivalent must be run on an existing cluster to use the new names).
  • The undocumented can_fail operation meta-attribute, which serves no useful purpose, is now deprecated and will be removed in a future version. A deprecation warning will be logged if it is changed from its default (unless Pacemaker is built with the --enable-compat-2.0 configure option).
  • The obscure and useless remove-after-stop cluster option is now deprecated and will be removed in a future version. A deprecation warning will be logged if it is changed from its default (unless Pacemaker is built with the --enable-compat-2.0 configure option).
  • Support for Upstart (both starting Pacemaker via Upstart, and managing Upstart-based resources in the cluster) is officially deprecated, and will be removed in a future release.
  • Pacemaker will stop adding the last-run attribute to <lrm_rsc_op> entries in the <status> section. This attribute hasn't been added to recurring operations since version 1.1.5, and is always identical to last-rc-change for non-recurring operations. As a result, last-run will no longer appear in crm_mon XML output either.

Tool changes

All

  • More commands support the --output-as and --output-to options, including XML output. These include crmadmin, crm_resource, crm_simulate, and crm_verify (in addition to crm_mon and stonith_admin, which already did). The intent is that the XML output will remain stable across releases as much as possible, for reliable parsing by scripts, while the text output may change from release to release for human-friendliness.
  • Some tools currently have (or had) undocumented support for long option abbreviation; for example, if the long option is "--upgrade", then any shorter abbreviation ("--upgr", "--up", etc.) would also work as long as it is unique for the tool. This support has been removed from many tools and is now considered deprecated for remaining tools, and it will be dropped from those in future releases.
  • Most short options that take arguments currently make the space after the short option optional, for example "-X saved.xml" or "-Xsaved.xml". Specifying a short option without a space is now considered deprecated; a future release will print a warning when this is used, and an even later release will drop support.
  • OCF resource agents will be displayed as ocf:PROVIDER:AGENT rather than ocf::PROVIDER:AGENT in crm_mon and crm_simulate output and pacemaker-schedulerd logs unless Pacemaker is built with the --enable-compat-2.0 configure option.
  • The OCF 1.1 role names ("Promoted" and "Unpromoted") will be used in all tool output instead of the legacy names unless Pacemaker is built with the --enable-compat-2.0 configure option.

crm_resource

  • The -a/--constraints and -A/--stack options (to display location and colocation constraints related to a given resource) now have considerably different (and hopefully more readable) text output.
  • The constraints created by crm_resource --ban will now use the OCF 1.1 standard role name "Promoted" unless Pacemaker is built with the --enable-compat-2.0 configure option.
  • The crm_resource --master option is now deprecated (but still accepted) and replaced with a new --promoted option.

crm_simulate

  • Output is now formatted similarly to crm_mon's.
  • New --show-attrs option will display node attribute values in simulation output.
  • New --show-failcounts option will display resource fail counts in simulation output.
  • stderr will no longer automatically be redirected to stdout when --verbose is given unless Pacemaker is built with the --enable-compat-2.0 configure option.

crmadmin

  • The -q/--quiet option previously printed its usual output to stdout and a briefer version to stderr. For consistency with other tools, it now prints only the brief version, to stdout.
  • The -t/--timeout option, which previously took an integer number of milliseconds, now takes a Pacemaker time specification (bare number of seconds, integer plus units, or ISO 8601 duration) for consistency with the other tools and configuration.
  • The text output for the -S/--status option has been changed to be more readable. (Scripts are recommended to use the new XML output support if parsing the output.)
  • The long-broken -E/--election and -K/--kill options have been completely removed. (They were only ever useful for debugging during early Pacemaker development.)

Other

  • When showing ban constraints, crm_mon --output-as=xml (and --as-xml) will now show promoted-only=true/false in addition to master_only=true/false, which is now deprecated.
  • The crm_master command has been deprecated (but still works) and replaced with a new crm_attribute --promotion option that defaults to --lifetime=reboot. This is typically called from within a resource agent. For example, "crm_master -l reboot -v 10" becomes "crm_attribute --promotion -v 10".
  • The output of the pacemakerd --features command better reflects the build options currently available. (Feature listings are not guaranteed to stay the same from release to release, and are intended only for troubleshooting.)

Resource agents

  • Pacemaker now supports OCF 1.1 compliant resource agents (in particular, reloadable parameters and the reload-agent action).
  • Pacemaker now provides resource agents with new environment variables (in addition to the existing ones) for promotable clone notifications, with "master" replaced with "promoted" and "slave" replaced with "unpromoted", for example OCF_RESKEY_CRM_meta_notify_unpromoted_resource will be identical to OCF_RESKEY_CRM_meta_notify_slave_resource. The old names are now deprecated.
  • The ocf:pacemaker:Dummy, ocf:pacemaker:Stateful, and ocf:pacemaker:remote resource agents now support OCF 1.1 (reloadable parameters, UI parameter hints, and advertising OCF 1.1 role names in monitor action meta-data).

Documentation changes

  • The source for the Pacemaker "books" (Clusters from Scratch, Pacemaker Explained, etc.) has been switched from using asciidoc source and Publican to using reStructuredText source and Sphinx. This is mainly because Publican is a dead project and distributions are dropping it, but it will also give us a cleaner, more modern look.
  • The --with-brand configure option has been removed. The books will now be built based on whether the sphinx-build and inkscape commands are available at build time.
  • The --with doc RPM build option remains, to decide whether to enable python3-sphinx and inkscape as build requirements for the pacemaker-doc subpackage (effectively controlling whether the books are built or not). The previous build requirement for publican has been removed. (The build requirement of asciidoc or asciidoctor remains, for generating the deprecated text file documentation, but that may change.)
  • The directory structure for the generated books has changed:
Generated content 2.0 location 2.1 location
Books built locally (in source directory) doc/TITLE/publish/FORMAT/en-US/ doc/sphinx/TITLE/_build/FORMAT/
Installed books (in package documentation directory a.k.a. docdir) TITLE/FORMAT/en-US/ TITLE/FORMAT/
Uploaded books (in RSYNC_DEST directory) pacemaker/doc/en-US/Pacemaker/2.0/FORMAT/TITLE/ pacemaker/doc/2.1/TITLE/FORMAT
  • Pacemaker uses the OCF 1.1 role names, "Promoted" and "Unpromoted", in the books, as well as tool help and man pages.

Development changes

Public C API changes

If you develop code using the Pacemaker C libraries, these could affect you.

  • A new constant, PCMK_ALLOW_DEPRECATED, enables control of API availability. It should normally be left undefined, but if you want to check whether your code uses any deprecated Pacemaker APIs, define it to 0 before including any Pacemaker headers.
  • The following APIs have been deprecated and will be removed in a future release (replacements are often not one-to-one, so read their relevant API documentation):
libcrmcluster
Deprecated Recommended replacement
crm_terminate_member() Use libstonithd APIs instead
crm_terminate_member_no_mainloop() Use libstonithd APIs instead
libcrmcommon
Deprecated Recommended replacement
__builtin_expect() Use only with GNU C compiler
__likely() __builtin_expect()
__unlikely() __builtin_expect()
EOS '\0'
MAX_IPC_DELAY none
GListPtr GList*
PCMK_OCF_DEGRADED_MASTER PCMK_OCF_DEGRADED_PROMOTED
PCMK_OCF_FAILED_MASTER PCMK_OCF_FAILED_PROMOTED
PCMK_OCF_RUNNING_MASTER PCMK_OCF_RUNNING_PROMOTED
XML_CIB_TAG_MASTER PCMK_XE_PROMOTABLE_LEGACY
XML_RSC_ATTR_MASTER_MAX PCMK_XE_PROMOTED_MAX_LEGACY
XML_RSC_ATTR_MASTER_NODEMAX PCMK_XE_PROMOTED_NODE_MAX_LEGACY
crm_atoi() strtoll()
crm_build_path() none
crm_config_error none
crm_config_warning none
crm_ftoa() a printf()-style function
crm_hash_table_size() g_hash_table_size()
crm_itoa() a printf()-style function
crm_itoa_stack() snprintf()
crm_log_cli_init() none
crm_parse_int() strtoll()
crm_parse_ll() strtoll()
crm_str_hash() g_str_hash()
crm_str_table_dup() none
crm_str_table_new() g_hash_table_new_full()
crm_strcase_equal() none
crm_strcase_hash() none
crm_strcase_table_new() g_hash_table_new_full()
crm_strip_trailing_newline() none
crm_ttoa() a printf()-style function
g_str_hash_traditional() g_str_hash()
pcmk_format_named_time() printf()-style function
pcmk_format_nvpair() printf()-style function
pcmk_numeric_strcasecmp() none
pcmk_scan_nvpair() sscanf()
libpe_status
Deprecated Recommended replacement
pe_order_implies_first_master pe_order_promoted_implies_first
pe_print_details none
pe_print_dev none
pe_print_html none
pe_print_log none
pe_print_max_details none
pe_print_ncurses none
pe_print_xml none
pe_resource_t parameters member pe_rsc_params()
RSC_ROLE_MASTER RSC_ROLE_PROMOTED
RSC_ROLE_MASTER_S RSC_ROLE_PROMOTED_LEGACY_S
RSC_ROLE_SLAVE RSC_ROLE_UNPROMOTED
RSC_ROLE_SLAVE_S RSC_ROLE_UNPROMOTED_LEGACY_S
  • The following public APIs have changed behavior:
    • libpe_status: resource_object_functions_t parameter() member now ignores its create argument (parameters will always be created if not already available)
  • The following public APIs have been added:
    • libcrmcommon
      • CRMD_ACTION_RELOAD_AGENT string constant
      • enum ocf_exitcode (moved from libcrmservice)
      • PCMK_OCF_MAJOR_VERSION string constant
      • PCMK_OCF_MINOR_VERSION string constant
      • PCMK_OCF_RUNNING_PROMOTED enum value
      • PCMK_OCF_VERSION string constant
      • PCMK_XE_PROMOTABLE_LEGACY string constant
      • PCMK_XE_PROMOTED_MAX_LEGACY string constant
      • PCMK_XE_PROMOTED_NODE_MAX_LEGACY string constant
    • libpacemaker
      • pcmk_controller_status()
      • pcmk_designated_controller()
      • pcmk_list_nodes()
      • pcmk_pacemakerd_status()
      • pcmk_resource_digests()
    • libpe_status
      • RSC_ROLE_PROMOTED enum value
      • RSC_ROLE_PROMOTED_LEGACY_S string constant
      • RSC_ROLE_PROMOTED_S string constant
      • RSC_ROLE_UNPROMOTED enum value
      • RSC_ROLE_UNPROMOTED_LEGACY_S string constant
      • RSC_ROLE_UNPROMOTED_S string constant
      • pcmk_working_set_t priv member (for Pacemaker use only)
      • pe_order_promoted_implies_first enum value
      • pe_resource_t parameter_cache member (managed via pe_rsc_params())
      • pe_rsc_params()

Source files moved

If you make changes to the Pacemaker source code, or build from the master branch, these could affect you.

  • Certain build artifacts are now created in the libltdl subdirectory instead of the top-level directory. This is uninteresting unless you update an existing source checkout and want to rebuild, in which case you first need to run:
rm -rf aclocal.m4 autom4te.cache config.guess config.log config.status config.sub \
       configure depcomp install-sh libltdl libtool ltmain.sh missing py-compile test-driver
./autogen.sh
  • Certain files changed locations in the source tree. Their installed locations did not change.
2.0 location 2.1 location
version.m4 m4/version.m4
extra/logrotate/pacemaker.in etc/logrotate.d/pacemaker.in
daemons/pacemakerd/pacemaker.in etc/init.d/pacemaker.in
daemons/pacemakerd/pacemaker.sysconfig etc/sysconfig/pacemaker
tools/crm_mon.sysconfig etc/sysconfig/crm_mon
cts/cts-support.in and related files cts/support/*
cts/CTSlab.py.in, cts/CTS.py.in, and related files cts/lab/*