Puppet - How to Regenerate All the Certs

Hopefully you will never need to do this, :). But we are humans and we make mistakes. It is very unfortunate that you will have to regenerate all the certs, but sometimes you just have to. I will share my experience here for how to do it safely.

On puppet master

Shutdown puppetmaster and puppetdb services.

# service puppetmaster stop
# service puppetdb stop
# ps -ef | grep puppet - to make sure no puppet process is running

Backup the ssl directory

# puppet master --configprint ssldir
/var/lib/puppet/ssl
# mv /var/lib/puppet/ssl ~/puppet-ssl-bak
# tree -CDups /var/lib/puppet/ssl
/var/lib/puppet/ssl [error opening dir]

0 directories, 0 files

Regenerate the CA

Puppet defines it's own certificate authority (CA) that is usually running on the master. Normally the "ca.pem" is used to:
    - generate new certificate for a given client out-of-bound
    - sign a new node that just sent his Certificate Signing Request
    - revoke any signed certificate
    - display certificate fingerprints

Every puppet client knows the CA certs, this allows a puppet client to verify puppet master. But the master does not need the client's CA, since it is sent by the client when connecting. It just need to make sure the client knows the private key and this certificate has been signed by the master CA. So first step we generate a ca.

# puppet cert list -a
# tree -CDups /var/lib/puppet/ssl

/var/lib/puppet/ssl
├── [drwxr-xr-x puppet          4096 Apr 28 10:22]  ca
│   ├── [-rw-r--r-- puppet           983 Apr 28 10:22]  ca_crl.pem
│   ├── [-rw-r--r-- puppet          2013 Apr 28 10:22]  ca_crt.pem
│   ├── [-rw-r--r-- puppet          3243 Apr 28 10:22]  ca_key.pem
│   ├── [-rw-r--r-- puppet            97 Apr 28 10:22]  inventory.txt
│   ├── [drwxr-x--- puppet            20 Apr 28 10:22]  private
│   │   └── [-rw-r--r-- puppet            20 Apr 28 10:22]  ca.pass
│   ├── [drwxr-xr-x puppet             6 Apr 28 10:22]  requests
│   ├── [-rw-r--r-- puppet             4 Apr 28 10:22]  serial
│   └── [drwxr-xr-x puppet             6 Apr 28 10:22]  signed
├── [drwxr-xr-x puppet             6 Apr 28 10:22]  certificate_requests
├── [drwxr-xr-x puppet             6 Apr 28 10:22]  certs
├── [drwxr-x--- puppet             6 Apr 28 10:22]  private
├── [drwxr-x--- puppet             6 Apr 28 10:22]  private_keys
└── [drwxr-xr-x puppet             6 Apr 28 10:22]  public_keys

9 directories, 6 files

Generate the puppet master’s new certs

# puppet master --no-daemonize –verbose
# Just after seeing “Notice: Starting Puppet master <your Puppet version>” hit CTRL+C
# tree -CDups /var/lib/puppet/ssl
/var/lib/puppet/ssl
├── [drwxr-xr-x puppet          4096 Apr 28 10:22]  ca
│   ├── [-rw-r--r-- puppet           983 Apr 28 10:22]  ca_crl.pem
│   ├── [-rw-r--r-- puppet          2013 Apr 28 10:22]  ca_crt.pem
│   ├── [-rw-r----- puppet          3243 Apr 28 10:22]  ca_key.pem
│   ├── [-rw-r--r-- puppet           183 Apr 28 10:23]  inventory.txt
│   ├── [drwxr-x--- puppet            20 Apr 28 10:22]  private
│   │   └── [-rw-r----- puppet            20 Apr 28 10:22]  ca.pass
│   ├── [drwxr-xr-x puppet             6 Apr 28 10:23]  requests
│   ├── [-rw-r--r-- puppet             4 Apr 28 10:23]  serial
│   └── [drwxr-xr-x puppet            45 Apr 28 10:23]  signed
│       └── [-rw-r--r-- puppet          2053 Apr 28 10:23]  puppet.xxx.com.pem
├── [drwxr-xr-x puppet             6 Apr 28 10:23]  certificate_requests
├── [drwxr-xr-x puppet            58 Apr 28 10:23]  certs
│   ├── [-rw-r--r-- puppet          2013 Apr 28 10:23]  ca.pem
│   └── [-rw-r--r-- puppet          2053 Apr 28 10:23]  puppet.xxx.com.pem
├── [drwxr-x--- puppet             6 Apr 28 10:22]  private
├── [drwxr-x--- puppet            45 Apr 28 10:23]  private_keys
│   └── [-rw-r--r-- puppet          3243 Apr 28 10:23]  puppet.xxx.com.pem
└── [drwxr-xr-x puppet            45 Apr 28 10:23]  public_keys
    └── [-rw-r--r-- puppet           800 Apr 28 10:23]  puppet.xxx.com.pem

9 directories, 11 files

If you use any extensions like puppetdb, replace the old certs:

# cp /var/lib/puppet/ssl/certs/ca.pem /etc/puppetdb/ssl/ca.pem
# cp /var/lib/puppet/ssl/private_keys/puppet.xxx.com.pem /etc/puppetdb/ssl/private.pem
# cp /var/lib/puppet/ssl/certs/puppet.xxx.com.pem /etc/puppetdb/ssl/public.pem
Make sure whoever runs the puppetdb have permissions to read.

Start puppetmaster and puppetdb

# service puppetdb start
# service puppetmaster start
Also watch the puppetdb log file for any weirdness.

Test master

# puppet cert list --all
# puppet agent --test --noop

At this point, you should have your puppet master certs regenerated and works fine.

On puppet client:

Make a copy of the old ssl directory:

# service puppet stop
# mv /var/lib/puppet/ssl ~/puppet-ssl-bak
# puppet agent -t

Now sign the new cert in puppet master
# puppet agent --test --noop
If no complains:
# service puppet start

Repeat this on all puppet client. Make a script to do it!