haproxy and certbot on ubuntu

This is largely inspired from https://phansch.net/posts/haproxy-letsencrypt-certbot/

I recently changed my setup such that haproxy listens on 80 and 443 and proxies into various services. I like it MUCH better now. IMO, it is far easier to manage.

One outstanding issue is certificate renewal. The certbot package on ubuntu (and probably debian, but I'm on ubuntu) includes /lib/systemd/system/certbot.service and /lib/systemd/system/certbot.timer files and I would like to work with them. Systemd is awesome and allows me to do so using overrides. I don't even have to think about how to manage the overrides. I use sudo systemctl edit certbot.

I add this to the editor which starts:

[Service]
ExecStart=
ExecStart=-/usr/bin/certbot -q renew --preferred-challenges http --http-01-port 9785
ExecStartPost=/etc/haproxy/certs/certbot-renew

There is an excellent post here which describes why the blank = and the -=. I shall not repeat it here. Go read the reply. It is great: https://askubuntu.com/a/659268/1668

To my haproxy.cfg I added to both frontends:

        acl letsencrypt-req path_beg /.well-known/acme-challenge/
        use_backend letsencrypt if letsencrypt-req

and that letsencrypt backend:

backend letsencrypt
   server letsencrypt 127.0.0.1:9785

Cool, now when letsencrypt tries to verify on my system, that request for /.well-known/acme-challenge/ will be redirected to the certbot process listening on port 9785.

There is also that ExecStartPost= which I must explain.

My haproxy.cfg is setup with crt-base /etc/haproxy/certs in the global section and an HTTPS frontend which starts like this:

frontend https-in
  bind :::443 v4v6 ssl crt one.example.com crt two.example.com crt e.example.net alpn h2,ht
tp/1.1

Yes, I have a few different certificates from letsencrypt. One with a dozen or so SNI names and the other two as wildcard certificates. I use the haproxy features of reading all pem files in a directory to load them. I'd like any certbot renew to automatically configure them for haproxy.

This is done with that ExecStartPost=/etc/haproxy/certs/certbot-renew script:

#!/bin/bash
set -e
cd /etc/haproxy/certs
LELIVE=/etc/letsencrypt/live
RELOAD=false
for d in $(find . -type d -not -name .) ; do
        if [[ $LELIVE/$d/fullchain.pem -nt $d/fullchain.pem ]] ; then
                RELOAD=true
                cp $LELIVE/$d/fullchain.pem $d/fullchain.pem
                cp $LELIVE/$d/privkey.pem $d/fullchain.pem.key
                chown haproxy:haproxy $d/fullchain.pem $d/fullchain.pem.key
        fi
done

if [[ $RELOAD == true ]] ; then
        systemctl reload haproxy
fi

The script looks at each directory in /etc/letsencrypt/live and if the fullchain.pem in that directory is newer than the one in /etc/haproxy/certs then it copies the pem file and keys into the haproxy directory with a filename which haproxy will read. If it did anything then it calls systemctl to reload haproxy.

It is possible that I won't have to deal with certificate renewal every again.