Puppetize

Background

I'm trying to puppetize my server so that it will be possible to upgrade to new hardware.

I have been doing Ubuntu upgrades for several releases and there is a lot of cruft gathering. I should do a clean install, but that seems like too much work and is too dangerous when so many services are running on one machine.

Enter puppet.

Now my strategy is to define each service that is running on my server on a separate puppet master. I copy the configuration over, describe it to puppet, and have puppet enforce that configuration on a test VM. I can work on one service at a time until they are all defined properly. I can destroy and recreate the VM occasionally to make sure that the puppet config can get all the dependencies right. And eventually, I'll be able to buy a new machine, point puppet at it and press the button: wham, a new functioning server.

So how is that beautiful dream working out for me?

Ubuntu version of chkconfig

I'm running into troubles with init scripts. I've only started defining a few services, apache, mysql, mediawiki. And already I get this:

root@ferret:/var/lib/mediawiki/maintenance# puppetd -t --noop
info: Caching catalog for ferret.finninday.net
info: Applying configuration version '1305993346'
notice: /Stage[main]/Mysql::Server/Service[mysql]/enable: is false, should be true (noop)
notice: /Stage[main]/Apache/Service[apache2]/enable: is false, should be true (noop)
notice: Finished catalog run in 12.84 seconds

Puppet thinks that mysql and apache are not enabled to start at boot, so it tries to fix that. But ubuntu's method of starting services at boot and puppet don't get along. The symptom is that the test repeatedly fails and puppet repeatedly enables them. If I watch closely, after puppet claims to have enabled the services at boot, they aren't really enabled.

root@ferret:/etc/rc2.d# puppetd -t
info: Caching catalog for ferret.finninday.net
info: Applying configuration version '1306022659'
notice: /Stage[main]/Apache/Service[apache2]/enable: enable changed 'false' to 'true'
notice: /Stage[main]/Mysql::Server/Service[mysql]/enable: enable changed 'false' to 'true'
notice: /Stage[main]/Phprecipebook/File[/var/www/phprecipebook-3.0.tgz]/ensure: defined content as '{md5}33181139a8d7ff8770059d9da6a5ab42'
notice: /Stage[main]/Phprecipebook/Exec[tar xzvf phprecipebook-3.0.tgz]/returns: executed successfully
notice: Finished catalog run in 19.29 seconds
root@ferret:/etc/rc2.d# ls
README         S20speech-dispatcher  S25bluetooth   S50saned      S75sudo            S99grub-common
S20fancontrol  S20virtualbox-ose     S50pulseaudio  S70dns-clean  S90binfmt-support  S99ondemand
S20kerneloops  S21puppet             S50rsync       S70pppd-dns   S99acpi-support    S99rc.local

I'm testing for the presence of symlinks in /etc/rc2.d that point into /etc/init.d. But according to /etc/init.d/README, Debian policy says that there are two methods of enabling services and they both must be maintained. The policy document is here:

http://www.debian.org/doc/debian-policy/#contents

update-rc.d

The Debian equivalent of chkconfig is update-rc.d and invoke-rc.d. When I ran

update-rc.d mysql defaults
update-rc.d apache2 defaults

puppet no longer thought that the services were disabled at boot.

How do you test if a service is enabled at boot? And why did puppet always determine that apache2 was disabled at boot?

Default ubuntu runlevel is 2, so if this statement finds a link, then apache2 is enabled at boot:

ls -l /etc/rc2.d/*apache2

Also, it is helpful to know that

update-rc.d -n apache2 defaults

does not actually create any links, it just tells you what would be done. So that is another way of knowing if something is enabled at boot.

But there is another process at work undoing the links that update-rc.d creates. After a few minutes, myslqd and apache2 are disabled at boot, that is, the links from /etc/rc2.d to /etc/init.d are deleted for apache2 and mysql.

puppetca --clean

When you are ready to test your puppet config against a fresh VM, you'll need to know about puppet certs.

You've been testing you puppet config against a VM and have slowly been building up modules. You wipe your VM so you can test that dependencies are being handled properly. You spin up a fresh OS and install puppet on it and try to connect to the puppet master for the first time and you see this:

err: Could not request certificate: Retrieved certificate does not match private key; please remove certificate from server and regenerate it with the current key

So you go to the puppetmaster and run:

puppetca --clean <hostname>

But that doesn't solve the problem.

You’ll also need to remove

/var/lib/puppet/ssl/certs/<hostname>.pem

on the client before running puppetd again.