Pacemaker Release Checklist

From ClusterLabs

Preparation for rc1

Submit pull requests against the main branch to clean things up if necessary:

  • Things that don't need to be done every release but are helpful every now and then:
    • cts/cts-scheduler --update (only if the tests already all pass!), to update whitespace usage in expected output (which is ignored by the regression tests, but can help manual comparisons)
    • (Do not do this until we can raise the minimum automake dependency to 1.1.14) make -C maint gnulib-update to update the code we bundle from gnulib
  • Compare scheduler timings against the previous release to make sure there are no large unexplained regressions. From a checkout, run the first command below for the old and new builds, then run the second command to compare the two. Minimize variation however you can (use same host, don't do anything else while running it, maybe try to seed the disk read cache). There still is a good bit of variation from run to run so it is unclear at this point what is significant.
    • PCMK_schema_directory=xml tools/crm_simulate --profile cts/scheduler/xml --repeat 1000 > ~/crm_simulate.$VERSION.out
    • tools/pcmk_simtimes ~/crm_simulate.*.out -s 0.1 -p 5
  • Ideally, we've bumped CRM_FEATURE_SET in include/crm/crm.h for every significant change during the development cycle. If any new features require DC support (including any schema changes), the minor version should increase (the y in x.y.z); otherwise, if any new features at all have been added (including things like tool command-line arguments), the minor-minor version should increase (the z in x.y.z; this is ignored by pacemaker, but allows external code such as resource agents and GUIs to check for support). Double-check that an appropriate bump has been made at least once.
  • Similarly, we should have bumped PCMK__API_VERSION in include/crm/common/output_internal.h if we made any changes to the schema files in xml/api. Verify that it's equal to the most highest version in the source file names there.
  • Verify formatted output args. See
  • Stress-test the current code with CTS, valgrind, and static analysis, and address any issues found.
    • Run the tests against a several node cluster. See cts/ Either use cts/cluster_test or your own site-specific wrapper. Ask someone else for theirs if you need help.
    • CI only runs the default regression tests, so run the rest manually. First, install the pacemaker RPMs on the cluster nodes. Then, shut down the cluster and run the following on the node:
      • /usr/share/pacemaker/tests/cts-regression -V exec pacemaker_remote fencing
    • There are three make targets for convenient static analysis:
      • make cppcheck
      • make clang
      • make coverity (or coverity-corp)

Repeat until final

All work is in the release branch (2.1).

rc1 only

  • Pull in the main branch (you can do this for later release candidates if really sure it is desired, but normally changes should go in the other direction once rc1 is released)
  • Update translation files: make -C po update-po (can repeat for later rc's and final if messages changed)
  • Update xml/ for pacemaker-to-schema version mapping, if needed
    • To see schema changes since last release: git log --no-merges Pacemaker-$LAST_VERSION.. xml
  • Run maint/bumplibs to update shared library (libtool) versions for changes since last point release, which will show all library diffs since the last release and ask whether the changes involve compatible public API additions, incompatible public API additions/removals, or fixes. Verify the script's changes (which it displays at the end).
    • Versions can be updated again for later release candidates if they break the public API (which should be avoided unless absolutely necessary) by running the script with Pacemaker-$PREVIOUS_RC_TAG as the argument.
    • For details about library versioning, see Updating library version information.
  • Update m4/version.m4 to new release version, without -rc suffix.
  • Update Changelog:
    • make -C maint LAST_RELEASE=Pacemaker-$LAST_VERSION changelog and then edit manually (and heavily!) for user audience
    • Ideally, we've updated the documentation along with the code during the development cycle. However it's worth double-checking that significant new features are in there (with "(since <version>)" for new options etc.).

Later release candidates only

  • Update Changelog:
    • make; cat <(make -C maint LAST_RC=Pacemaker-$LAST_RC_VERSION rc-changes) <(git show HEAD:ChangeLog) > ChangeLog and edit appropriately

Final release only

  • It is a good idea to locally generate and proof all documentation that will be uploaded to the website.
  • Update Changelog: merge the rc entries into a single entry
    • to update summary: make -C maint LAST_RELEASE=Pacemaker-$LAST_VERSION NEXT_RELEASE=Pacemaker-$NEXT_VERSION summary
  • Update all relevant tasks from "Merged" status to "Released" in the project manager

All release candidates and final release

  • You can repeat part or all of the stress-testing if there have been major changes.
  • Update translation files: make -C po update-po
  • Make a github release:
    • Go to Pacemaker releases on Github and select Draft a new release
    • Choose a tag: Pacemaker-X.Y.Z-rcN or Pacemaker-X.Y.Z
    • Target: release branch
    • Release title: "Pacemaker X.Y.Z - Release Candidate N" or "Pacemaker X.Y.Z - Final"
    • Describe this release: copy relevant part of ChangeLog
    • This is a pre-release: check if this is an rc
  • Pull release branch back into main branch and merge.
  • If this is an rc, wait for feedback, and address any reported issues. About 1-4 weeks between release candidates is good.

After final release

  • Set and export the RSYNC_RSH environment variable appropriately for the ClusterLabs website host (e.g. "ssh -p <#>")
  • Run make -C doc www to regenerate most documentation on website (doxygen, indexed source, Clusters From Scratch, Pacemaker Explained, etc.).
  • Run make -C doc LAST_RELEASE=Pacemaker-$PREVIOUS_RELEASE TAG=Pacemaker-$NEW_RELEASE abi-www to generate the ABI compatibility report.
  • Announce request policy on (all requests into main).
  • Edit any pull requests open against the release branch to submit against main.

Possible improvements

  • rcX releases should not pretend to be full-fledged release (version.m4, etc.)
  • before any tagged/point release (at latest), check the current buildability in COPR (default sort starts with the newest builds, optionally use pacemaker in the filter box, check at least Git branch field on the details page, optionally also git commit encoded in the R part of package identifier)