Puppetize

Background

I'm trying to puppetize my server so that it will be possible to upgrade to new hardware.

I have been doing Ubuntu upgrades for several releases and there is a lot of cruft gathering. I should do a clean install, but that seems like too much work and is too dangerous when so many services are running on one machine.

Enter puppet.

Now my strategy is to define each service that is running on my server on a separate puppet master. I copy the configuration over, describe it to puppet, and have puppet enforce that configuration on a test VM. I can work on one service at a time until they are all defined properly. I can destroy and recreate the VM occasionally to make sure that the puppet config can get all the dependencies right. And eventually, I'll be able to buy a new machine, point puppet at it and press the button: wham, a new functioning server.

So how is that beautiful dream working out for me?

Ubuntu version of chkconfig

I'm running into troubles with init scripts. I've only started defining a few services, apache, mysql, mediawiki. And already I get this:

root@ferret:/var/lib/mediawiki/maintenance# puppetd -t --noop
info: Caching catalog for ferret.finninday.net
info: Applying configuration version '1305993346'
notice: /Stage[main]/Mysql::Server/Service[mysql]/enable: is false, should be true (noop)
notice: /Stage[main]/Apache/Service[apache2]/enable: is false, should be true (noop)
notice: Finished catalog run in 12.84 seconds

Puppet thinks that mysql and apache are not enabled to start at boot, so it tries to fix that. But ubuntu's method of starting services at boot and puppet don't get along. The symptom is that the test repeatedly fails and puppet repeatedly enables them. If I watch closely, after puppet claims to have enabled the services at boot, they aren't really enabled.

root@ferret:/etc/rc2.d# puppetd -t
info: Caching catalog for ferret.finninday.net
info: Applying configuration version '1306022659'
notice: /Stage[main]/Apache/Service[apache2]/enable: enable changed 'false' to 'true'
notice: /Stage[main]/Mysql::Server/Service[mysql]/enable: enable changed 'false' to 'true'
notice: /Stage[main]/Phprecipebook/File[/var/www/phprecipebook-3.0.tgz]/ensure: defined content as '{md5}33181139a8d7ff8770059d9da6a5ab42'
notice: /Stage[main]/Phprecipebook/Exec[tar xzvf phprecipebook-3.0.tgz]/returns: executed successfully
notice: Finished catalog run in 19.29 seconds
root@ferret:/etc/rc2.d# ls
README         S20speech-dispatcher  S25bluetooth   S50saned      S75sudo            S99grub-common
S20fancontrol  S20virtualbox-ose     S50pulseaudio  S70dns-clean  S90binfmt-support  S99ondemand
S20kerneloops  S21puppet             S50rsync       S70pppd-dns   S99acpi-support    S99rc.local

update-rc.d

The Debian equivalent of chkconfig is update-rc.d and invoke-rc.d. When I ran

update-rc.d mysql defaults
update-rc.d apache2 defaults

puppet no longer thought that the services were disabled at boot.

How do you test if a service is enabled at boot? And why did puppet always determine that apache2 was disabled at boot?

Default ubuntu runlevel is 2, so if this statement finds a link, then apache2 is enabled at boot:

ls -l /etc/rc2.d/*apache2

Also, it is helpful to know that

update-rc.d -n apache2 defaults

does not actually create any links, it just tells you what would be done. So that is another way of knowing if something is enabled at boot.

But there is another process at work undoing the links that update-rc.d creates. After a few minutes, myslqd and apache2 are disabled at boot, that is, the links from /etc/rc2.d to /etc/init.d are deleted for apache2 and mysql.