Pacemaker Release Checklist

Preparation for rc1

Submit pull requests against the master branch to clean things up if necessary:

  • Things that don't need to be done every release but are helpful every now and then:
    • make gnulib-update to update the code we bundle from gnulib
    • cts/cts-scheduler --update (only if the tests already all pass!), to update whitespace usage in expected output (which is ignored by the regression tests, but can help manual comparisons)
  • Compare scheduler timings against the previous release to make sure there are no large unexplained regressions. From a checkout, run the first command below for the old and new builds, then run the second command to compare the two. Minimize variation however you can (use same host, don't do anything else while running it, maybe try to seed the disk read cache). There still is a good bit of variation from run to run so it is unclear at this point what is significant.
    • PCMK_schema_directory=xml tools/crm_simulate --profile cts/scheduler/xml --repeat 1000 > ~/crm_simulate.$VERSION.out
    • tools/pcmk_simtimes ~/crm_simulate.*.out -s 0.1 -p 5
  • Ideally, we've bumped CRM_FEATURE_SET in include/crm/crm.h for every significant change during the development cycle. If any new features require DC support (including any schema changes), the minor version should increase (the y in x.y.z); otherwise, if any new features at all have been added (including things like tool command-line arguments), the minor-minor version should increase (the z in x.y.z; this is ignored by pacemaker, but allows external code such as resource agents and GUIs to check for support). Double-check that an appropriate bump has been made at least once.
  • Verify formatted output args. See https://github.com/clumens/fosa/blob/master/README.md.
  • Run the unit tests: make check
  • Stress-test the current code with CTS, valgrind, and static analysis, and address any issues found.
    • Run the CTSlab.py tests against a several node cluster. See cts/README.md. Either use cts/cluster_test or your own site-specific wrapper. Ask someone else for theirs if you need help.
    • Travis only runs the default regression tests, so run the rest manually. First, install the pacemaker RPMs on the cluster nodes. Then, shut down the cluster and run the following on the node:
      • /usr/share/pacemaker/tests/cts-regression -V exec pacemaker_remote fencing
    • There are three make targets for convenient static analysis:
      • make cppcheck
      • make clang
      • make coverity (or coverity-corp)
    • Verify that any public API changes are compatible with C++:
      • make -C maint testcc

Repeat until final

All work is in the release branch (2.1).

rc1 only

  • Pull in the master branch (you can do this for later release candidates if really sure it is desired, but normally changes should go in the other direction once rc1 is released)
  • Update xml/README.md for pacemaker-to-schema version mapping, if needed
    • To see schema changes since last release: git log Pacemaker-$LAST_VERSION.. xml
  • Run maint/bumplibs.sh to update shared library (libtool) versions for changes since last point release, which will show all library diffs since the last release and ask whether the changes involve compatible public API additions, incompatible public API additions/removals, or fixes. Verify the script's changes (which it displays at the end).
    • Versions can be updated again for later release candidates if they break the public API (which should be avoided unless absolutely necessary) by running the script with Pacemaker-$PREVIOUS_RC_TAG as the argument.
    • For details about library versioning, see Updating library version information.
  • Update m4/version.m4 to new release version, without -rc suffix.
  • Update Changelog:
    • make LAST_RELEASE=Pacemaker-$LAST_VERSION changelog and then edit manually (and heavily!) for user audience
    • Ideally, we've updated the documentation along with the code during the development cycle. However it's worth double-checking that significant new features are in there (with "(since <version>)" for new options etc.).

Later release candidates only

  • Update Changelog:
    • make; cat <(make LAST_RC=Pacemaker-$LAST_RC_VERSION rc-changes) <(git show HEAD:ChangeLog) > ChangeLog and edit appropriately

Final release only

  • It is a good idea to locally generate and proof all documentation that will be uploaded to the website.
  • Update Changelog: merge the rc entries into a single entry
    • to update summary: make LAST_RELEASE=Pacemaker-$LAST_VERSION NEXT_RELEASE=Pacemaker-$NEXT_VERSION summary

All release candidates and final release

  • You can repeat part or all of the stress-testing if there have been major changes.
  • Make a github release:
    • Go to Pacemaker releases on Github and select Draft a new release
    • Tag version: Pacemaker-X.Y.Z-rcN or Pacemaker-X.Y.Z
    • Target: release branch
    • Release title: "Pacemaker X.Y.Z - Release Candidate N" or "Pacemaker X.Y.Z - Final"
    • Describe this release: copy relevant part of ChangeLog
    • This is a pre-release: check if this is an rc
  • Pull release branch back into master branch and merge.
  • If this is an rc, wait for feedback, and address any reported issues. About 1-4 weeks between release candidates is good.

After final release

  • Set and export the RSYNC_RSH environment variable appropriately for the ClusterLabs website host (e.g. "ssh -p <#>")
  • Run make -C doc www to regenerate most documentation on ClusterLabs.org website (doxygen, indexed source, Clusters From Scratch, Pacemaker Explained, etc.).
  • Run make -C doc LAST_RELEASE=Pacemaker-$PREVIOUS_RELEASE TAG=Pacemaker-$NEW_RELEASE abi-www to generate the ABI compatibility report.
  • Announce request policy on developers@clusterlabs.org (all requests into master).
  • Edit any pull requests open against the release branch to submit against master.

Possible improvements

  • rcX releases should not pretend to be full-fledged release (version.m4, etc.)
  • before any tagged/point release (at latest), check the current buildability in COPR (default sort starts with the newest builds, optionally use pacemaker in the filter box, check at least Git branch field on the details page, optionally also git commit encoded in the R part of package identifier)