Certificate Authority and Puppet 6

Configuration Management Camp is the event for technologists interested in Open Source Infrastructure automation and related topics. Among other configuration management tools, Puppet was also present. Currently I’m working on a puppet3->5 migration and on that account I attended several presentations about puppet. Because puppet 6 was released in September 2018 version 6 was one of the topics.

In this blog I would like to describe the differences in Puppet 6 regarding the Certificate Authority (CA), which is an important part of the infrastructure in Puppet Server.

In 2014, with the release of Puppet Server, Puppetlabs switched from Ruby to Clojure. Clojure is a functional programming language that runs in JVM. Because Puppet Master was originally writen in Ruby, the CA was based on the openssl libraries of Ruby. With the transition to Clojure the CA is also rewritten in Clojure. Since Puppet 6 the CA is fully integrated in Puppet Server. When you start working with Puppet 6 you’ll notice that the previous CLI tools for interacting with the CA and SSL artifacts like puppet certpuppet certificatepuppet capuppet certificate_request, and puppet certificate_revocation_list, are all replaced with one single command puppetserver ca.

Puppetserver ca 
Usage: puppetserver ca <action> [options]
Manage the Private Key Infrastructure for
Puppet Server's built-in Certificate Authority
Available Actions:
  Certificate Actions (requires a running Puppet Server):
    clean       Revoke cert(s) and remove related files from CA
    generate    Generate a new certificate signed by the CA
    list        List certificates and CSRs
    revoke      Revoke certificate(s)
    sign        Sign certificate request(s)

Redesign of CA

The biggest change in Puppet 6 is the design of the CA. Originally the puppet server comes with a CA that contains the root certificate which is also the signing certificate.

Security wise that’s undesirable because the root key has to be on the system for certificate signing requests.

One of the primary complaints about this approach is that this conflation of “self signed root” and “signing certificate” violates a security policy. In Puppet 6 the root and signing CA are split into two separate CA’s. The intermediate CA handles all the certificate requests and the root certificate is only used to sign the intermediate certificate, which makes it a trust of chains. In Puppet 6 the CA and PKI tools are now built-in.

A user can also call setup to create a default intermediate CA for their installion. Both setupand import should be run before starting Puppet Server for the first time, as the previous (Puppet 5) behavior of the server creating a self-signed CA cert on start-up is still in place. setup and import will not overwrite an existing CA; you’ll get a warning.

Puppet uses certificates in a client-authentication mode. The agent checks the validity of the master’s certificate, and the master checks the validity and identity of the agents as well. This is called a two-way authentication. When an agent connects to the master for the first time it will generate a Certificate Signing Request. The CA has to sign this certificate with its intermediate signing certificate. This can be done by auto signing or manually.

puppetserver ca sign --certname <certname>

The next time a client connects, it will find the new signed certificate, download it and use it to request the catalog. The ca and crl files are immediately downloaded by the client after submitting the CRL, so these files are already in place. Additionally, since the agent is initiating the connection, it verifies that the hostname of the server it’s connecting to is either in the Subject of the server’s certificate or it’s listed in the SubjectAltNames field, which acts as a list of valid aliases for the server.

To verify it’s certificate the client will check if the certificate is being issued by a trusted CA. Therefore the ca.pem is needed. The client also checks if the certificate of the master hasn’t been revoked (crl.pem). The crl file is managed by the master. In Puppet 6 this is also improved. More on this later.

The agent-side tasks like requesting and downloading are all embedded in the puppet ssl sub-command. This is also new in Puppet 6. Only those operations which are applicable to the Puppet agents are there. The various puppet ssl commands are:

* submit_request:
  Generate a certificate signing request (CSR) and submit it to the CA. If
  a private and public key pair already exist, they will be used to generate
  the CSR. Otherwise a new key pair will be generated. If a CSR has already
  been submitted with the given `certname`, then the operation will fail.

* download_cert:
  Download a certificate for this host. If the current private key matches
  the downloaded certificate, then the certificate will be saved and used
  for subsequent requests. If there is already an existing certificate, it
  will be overwritten.

* verify:
  Verify the private key and certificate are present and match, verify the
  certificate is issued by a trusted CA, and check revocation status.

* clean:
  Remove the private key and certificate related files for this host. If
  `--localca` is specified, then also remove this host's local copy of the
  CA certificate(s) and CRL bundle. if `--target CERTNAME` is specified, then
  remove the files for the specified device on this host instead of this host.

A verification of the certificate on the client is done by:

puppet ssl verify

To better understand what this command is actually doing, you can do the same with openssl. For example:

cd /etc/puppetlabs/puppet/ssl
cat crl.pem certs/ca.pem > chain.pem # Combine crl and ca
openssl verify -crl_check -CAfile chain.pem certs/<certname>.pem

Authorization rules

Authentication to the API endpoints in Puppet Server is managed in /etc/puppetlabs/puppetserver/conf.d/auth.conf. This file contains all rules for authorizing access. Prior to Puppet 6 authorization for accessing the CA endpoint was granted based on IP-address or hostname of the Puppet master. In Puppet 6 the certificate of the master is extended with an OID : pp_cli_auth. Within the allow section we see the default settings, which now requires the Puppet Server CA certificate to have the pp_cli_auth extension set to true.

            # Allow the CA CLI to access the certificate_status endpoint
            match-request: {
                path: "/puppet-ca/v1/certificate_status"
                type: path
                method: [get, put, delete]
            allow: {
               extensions: {
                   pp_cli_auth: "true"
            sort-order: 500
            name: "puppetlabs cert status"

You can find the OID in the certificate with openssl. Here you’ll see the value is set to true.

openssl x509 -in /etc/puppetlabs/puppet/ssl/certs/`hostname`.pem -text -noout 

It can be interesting to enable debugging for auth.conf. This can be done by adding the following line to /etc/puppetlabs/puppet/logback.xml

<logger name="puppetlabs.trapperkeeper.authorization.rules" level="TRACE"/>

After restarting puppetserver you will see which rules are checked and allowed.

tail -f /var/log/puppetlabs/puppetserver/puppetserver.log

The Certificate Revocation List

Originally the CRL design in Puppet was based on the standard CRL that is built into the Ruby openssl libraries. The CRL file is maintained by the Puppet Server and downloaded to the clients at the moment they first connect with the server. One unintended consequence of this simple strategy is that sites with large numbers or nodes, or who churn through a lot of nodes due to frequent re-provisioning, end up with extremely large CRL files, which could affect performance both at agent start-up (as it reads in the file) and when creating network connections (as it validated the remote end’s certificate against an extremely large list, containing mostly other agents).

The Puppet Server CA may now create a smaller CRL that contains only revocations of those nodes that agents are expected to talk to during normal operations (like compile masters or hosts that agents connect to as part of agent side functions) and distribute that CRL to agents, rather than the CRL that contains all agent revocations.

Once enabled, you’ll need to manage a file at $cadir/infra_inventory.txt. It should be a newline separated list of the certnames, that when revoked should be added to the Infra CRL. The certnames must match existing certificates issued and maintained by the Puppet Server CA. Setting the value “certificate-authority.enable-infra-crl” to true (see Puppet Server’s config syntax for details) will cause Puppet Server to update both its Full CRL and its Infra CRL with the certs that match those certnames when revoked. Now when agents first check in they will receive a CRL that includes only the revocations of certnames listed in the infra_inventory.txt.

This feature is disabled by default because it is site specific and depends on the setup (multiple compile masters etc). A single master doesn’t need this feature enabled.



Actieve filters: Wis alle filters