Running CTS lab on OpenStack hosts

Note: This was originally written in 2012-2013 with minimal changes since, and is unlikely to be up to date.

If we ever get the OpenStack networking sorted out to the point where Corosync can use it, this documents the process of getting a Pacemaker cluster set up for testing with CTS.

Prep

Install python-novaclient

   yum install -y python-novaclient

Export your OpenStack credentials

   export OS_REGION_NAME=...
   export OS_TENANT_NAME=...
   export OS_AUTH_URL=...
   export OS_USERNAME=...
   export OS_PASSWORD=...
   export IMAGE_USER=fedora

Allocate 5 floating IPs. For the purposes of the setup instructions (and probably your sanity), they need to be consecutive and to remain sane, should ideally start with a multiple of 10. Below we will assume 10.16.16.60-64

   for n in `seq 1 5`; do nova floating-ip-create; done

Create some variables based on the IP addresses nova created for you:

   export IP_BASE=10.16.16.60

and a function for calculating offsets

   function nth_ipaddr() {
       echo $IP_BASE | awk -F. -v offset=$1 '{ printf "%s.%s.%s.%s\n", $1, $2, $3, $4 + offset }'
   }
   function ip_net() {
       echo $IP_BASE | awk -F. '{ printf "%s.%s.%s.*\n", $1, $2, $3 }'
   }

Upload a public key that we can use to log into the images we create. I created one especially for cluster testing and left it without a password.

   nova keypair-add --pub-key ~/.ssh/cluster Cluster

Make sure it gets used when connecting to the CTS lab exerciser

   cat << EOF >> ~/.ssh/config
   Host cts-exerciser \`echo $IP_BASE | awk -F. '{ printf "%s.%s.%s.*", \$1, \$2, \$3 }'\`
   	 User	    	    root
        IdentityFile       ~/.ssh/cluster
        UserKnownHostsFile ~/.ssh/known.openstack
   EOF

Punch a hole in the firewall for SSH access and ping

   nova secgroup-add-rule default tcp 23 23 10.0.0.0/8
   nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0

Add the CTS lab exerciser to /etc/hosts

   cat << EOF >> /etc/hosts
   `nth_ipaddr 0` cts-exerciser
   EOF

Create helper scripts on a local host

   cat << END > ./exerciser.sh
   echo export OS_REGION_NAME=$OS_REGION_NAME >> ~/.bashrc
   echo export OS_TENANT_NAME=$OS_TENANT_NAME >> ~/.bashrc
   echo export OS_AUTH_URL=$OS_AUTH_URL >> ~/.bashrc
   echo export OS_USERNAME=$OS_USERNAME >> ~/.bashrc
   echo export OS_PASSWORD=$OS_PASSWORD >> ~/.bashrc
   function nth_ipaddr() {
       echo $IP_BASE | awk -F. -v offset=\$1 '{ printf "%s.%s.%s.%s\n", \$1, \$2, \$3, \$4 + offset }'
   }
   yum install -y python-novaclient git screen pdsh pdsh-mod-dshgroup
   git clone --depth 0 git://github.com/beekhof/fence_openstack.git
   ln -s /root/fence_openstack/fence_openstack /sbin
   mkdir -p  /root/.dsh/group/
   echo export cluster_name=openstack >> ~/.bashrc
   rm -f /root/.dsh/group/openstack
   for n in `seq 1 4`; do
   	echo "cluster-\$n" >> /root/.dsh/group/openstack
       echo \`nth_ipaddr \$n\` cluster-\$n >> /etc/hosts
   done
   cat << EOF >> /root/.ssh/config
       Host cts-exerciser \`echo $IP_BASE | awk -F. '{ printf "%s.%s.%s.*", \$1, \$2, \$3 }'\`
       User       root
       IdentityFile ~/.ssh/cluster
   EOF
   END

Some images do not allow root to log in by default and insist on a 'fedora' user. Create a script to disable this "feature":

   cat << EOF > fix-guest.sh
   #!/bin/bash
   # Re-allow root to log in
   sudo sed -i s/.*ssh-/ssh-/ /root/.ssh/authorized_keys
   EOF

CTS lab exerciser (Fedora-18)

Create and update the lab exerciser

   nova boot --poll --image "Fedora 18" --key_name Cluster --flavor m1.tiny cts-exerciser
   nova add-floating-ip cts-exerciser `nth_ipaddr 0`

If your image does not allow root to log in by default, disable this "feature" with the script we created earlier:

   scp fix-guest.sh $IMAGE_USER@cts-exerciser:
   ssh -l $IMAGE_USER -t cts-exerciser -- bash ./fix-guest.sh

Now we can set up the CTS lab exerciser with the script we created earlier:

   scp ~/.ssh/cluster cts-exerciser:.ssh/id_rsa
   scp exerciser.sh cts-exerciser:
   ssh cts-exerciser -- bash ./exerciser.sh

Create the Guests

First create the guests

   for n in `seq 1 4`; do
      nova boot --poll --image "Fedora 18" --key_name Cluster --flavor m1.tiny cluster-$n;
      nova add-floating-ip cluster-$n `nth_ipaddr $n`
   done

Then wait for everything to settle

   sleep 10

Fix the Guests

If your image does not allow root to log in by default, disable this "feature" with the script we created earlier:

   for n in `seq 1 4`; do
      scp fix-guest.sh $IMAGE_USER@`nth_ipaddr $n`:
      ssh -l $IMAGE_USER -t `nth_ipaddr $n` -- bash ./fix-guest.sh;
   done

Run CTS

Prep

Switch to the CTS lab exerciser

   ssh cts-exerciser

Clone Pacemaker for the latest version of CTS:

   git clone --depth 0 git://github.com/ClusterLabs/pacemaker.git
   echo 'export PATH=$PATH:/root/pacemaker/extra:/root/pacemaker/cts' >> ~/.bashrc
   echo alias c=\'cluster-helper\' >> ~/.bashrc
   . ~/.bashrc

Now set up CTS to run from the local source tree

   cts local-init

Configure a cluster (this will install all needed packages and configure corosync on the guests in the $cluster_name group)

   cluster-init -g openstack --yes --unicast --hosts fedora-18

Run

   cd pacemaker
   cts clean run --stonith openstack