Florian Storz for devlix Blog

Posted on Apr 6, 2021 • Originally published at devlix.de

Automate Let's Encrypt Wildcard Certificate creation with Ionos DNS Rest API

#ionos #letsencrypt #dns #certificate

“At the end of the day, the goals are simple: safety and security.” – Jodi Rell

The majority of page views on the Internet are using encrypted connections.

**Figure 1:** Source: Google Transparency Report

A website has to provide a certificate, issued by a trusted certification authority to establish those connections.

Until a few years ago, you only got such certificates with an annual fee for issuing from commercial companies like GeoTrust, DigiCert, etc.

With Let's Encrypt, an open-source certification authority has been available since the end of 2015, which provides free certificates via a more automated protocol.

To avoid the effort of managing many subdomains and their associated certificates, there is the option of using so-called wildcard certificates. However, applying for such certificates is a bit more complicated and was previously only possible through manual interaction with the DNS provider Ionos in combination with Let's Encrypt.

What are wildcard certificates?

In order for your own certificate to be displayed as "secure" in the browser, it must first be applied for from a trustworthy certification authority (so-called CAs / Certificate Authorities). When the certificate for a website is presented to the browser, it is checked against a list of CAs stored in the operating system.

Each certificate has a unique name (the so-called "CN - Common Name"), which represents the domain that is to be protected by the certificate.

For example, if a website is accessed under example.com, the common name of the associated certificate must be CN=example.com. Now, if a website is provided under sub.example.com, it needs its own certificate with the common name CN=sub.example.com. The website under the subdomain cannot be protected by the top-level domain (TLD) certificate.

If a company uses many subdomains, it can become much of administrative work to maintain all the certificates of the individual subdomains. For this reason there are the so-called wildcard certificates.

The name already implies the functionality of this special type. If the CN contains e.g. the value CN=*.example.com, the TLD and all direct subdomains of the TLD are automatically protected by this certificate.

What is Let's Encrypt?

There have been established certification authorities for a very long time, such as DigiCert, GeoTrust or similar. Since these are commercial companies, they usually charge an annual fee for the "authentication" of the certificates.

In contrast, since the end of 2015, there is an open-source certification authority called Let's Encrypt. Let's Encrypt is a free, automated and open certification authority provided by the Internet Security Research Group (ISRG).

Let's Encrypt uses the Automated Certificate Management Environment (ACME) protocol, which was standardized in RFC 8555, to validate the ownership of a domain. Since March 2018, Let's Encrypt also offers wildcard certificates using the ACMEv2 protocol (see also Let's Encrypt blog post).

How to request a (wildcard) certificate at Let's Encrypt?

Let's Encrypt certificates for a specific domain can be requested through the HTTP Challenge or the DNS Challenge. For wildcard certificates from Let's Encrypt, only the DNS Challenge can be used (see the article in the Let's Encrypt FAQ).

With the DNS Challenge, validation is accomplished by setting dedicated DNS records. This means that during the process a TXT entry must be created, e.g. for the domain _acme-challenge.example.com with a unique value defined by Let's Encrypt. It then checks whether the specified value can be retrieved via the DNS system under the named domain. To ensured that the person making the request has access to the domain and is therefore authorized to carry out the process.

Workflow until now with Let's Encrypt and Ionos DNS

In the past, Ionos as domain and DNS provider at the same time only offered the possibility to create, change or delete DNS entries via web UI. Since TXT entries must be available in the DNS for the domain for the validation process, it was previously necessary to add these manually and then remove them again.

The Let's Encrypt certificates are only valid for 3 months. Therefore, this process must be performed regularly and manually. The following steps are showing how the steps looked like so far.

In order to create or delete DNS entries, authentication at the web UI and navigation to the DNS management.
The validation process is started e.g. with Certbot by executing the following command (replace with the desired TLD)
```
certbot certonly 
  -d *.<your.domain> 
  --preferred-challenges dns 
  --manual
```

During the DNS challenge protocol, the process is paused for a short time so that a TXT entry with a value defined by Let's Encrypt (<acme-challenge-value>) can be entered in the DNS under a defined sub-domain (_acme-challenge.<your.domain>).

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator manual, Installer None
Obtaining a new certificate
Performing the following challenges:
dns-01 challenge for <your.domain>

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please deploy a DNS TXT record under the name
_acme-challenge.<your.domain> with the following value:

<acme-challenge-value>

Before continuing, verify the record is deployed.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue

The values are entered and saved in the Ionos web UI.

When the DNS entries have been saved, the process can be continued. The certificate is stored in the filesystem after the DNS entries have been successfully validated.

Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at:
  /etc/letsencrypt/live/<your.domain>/fullchain.pem
  Your key file has been saved at:
  /etc/letsencrypt/live/<your.domain>/privkey.pem
  Your cert will expire on 2019-08-01. To obtain a new or tweaked
  version of this certificate in the future, simply run certbot
  again. To non-interactively renew *all* of your certificates, 
  run "certbot renew"
- If you like Certbot, please consider supporting our work by:

  Donating to ISRG / Let's Encrypt:   <https://letsencrypt.org/donate>
  Donating to EFF:                    <https://eff.org/donate-le>

After the DNS challenge has finished, the created DNS entries should be deleted again. The deletion of the entries needs to be done manually in the Ionos web UI.

The new Ionos DNS API

Since December 2020 Ionos provides a beta API for their DNS system.

In the Developer Portal of Ionos you can find a section for the DNS API with a Swagger UI. For an easier start, a "Getting Started" is provided, which provides some details on how to use and register for the API.

In order to use the DNS API, a registration must be completed. On the page for retrieving the associated API keys, you will be directed to the contact page of Ionos for this purpose.

Hint: In our own experience, the registration can only be done via the support hotline, since the activation of the API is a free "order"!

After successful ordering, the "access key" can be requested in the Developer Portal via the "Create a new key" button. The key consists of a publicprefix and a secret. The secret is only displayed once and should be stored securely.

Automation Scripts

There is a plugin for the Certbot tool for some DNS providers to automate the validation process and to create or remove entries via their APIs.

There is no such plugin for Ionos. However, there is the possibility to automate the process with shell scripts for Certbot itself.

We want to show you such shell scripts, which can be used to create wildcard certificates with the DNS provider Ionos in an automated way and how the scripts can be used in combination with Certbot.

Auth Hook

The Auth Hook Script creates the DNS entries required for validating the domain at Ionos via the DNS API. In the process, the Certbot tool sets certain environment variables at runtime, which can be used during processing. These include, for example, the domain to be validated (CERTBOT_DOMAIN) or the token for validating the domain (CERTBOT_VALIDATION).

Precondition: On the system, which executes the script, curl & python must be installed!

Execution note: To display debug logs, remove the # character before the command set -x.

Auth Hook Shell Script

In the first step, the script searches for the name of the DNS zone based on the domain for which a certificate is requested.

In the next step, the identifier for the zone is requested from Ionos via the DNS API. This identifier is needed to create DNS entries in this zone via the API.

The TXT DNS entry is then created with the validation token under the subdomain expected by Let's Encrypt (_acme-challenge.<your.domain>).

Finally, the identifier of the zone is written to a temporary file, which is later used for the cleanup.

#!/bin/sh
#set -x

API_URL="https://api.hosting.ionos.com/dns/v1"
API_KEY_HEADER="X-API-Key: $API_KEY"

# Strip only the top domain to get the zone id
ZONE_NAME=$(expr match "$CERTBOT_DOMAIN" '.*\.\(.*\..*\)')
# When already the TLD then use it
if [ -z "$ZONE_NAME" ]; then
  ZONE_NAME="$CERTBOT_DOMAIN"
fi

# Get the Ionos zone id
ZONE_RESPONSE=$(curl -s -X GET "$API_URL/zones" \
                     -H "$API_KEY_HEADER" \
                     -H "Accept: application/json")
ZONE_ID=$(echo $ZONE_RESPONSE \
     | python -c "import sys,json;response=json.load(sys.stdin);print(next((x for x in response if x['name']=='$ZONE_NAME'))['id'])")

# Create TXT record
CREATE_DOMAIN="_acme-challenge.$CERTBOT_DOMAIN"
RECORD_CREATE_RESPONSE=$(curl -s -X POST "$API_URL/zones/$ZONE_ID/records" \
                              -H "$API_KEY_HEADER" \
                              -H "Content-Type: application/json" \
                              --data '[{"name": "'"$CREATE_DOMAIN"'", "type": "TXT", "content": "'"$CERTBOT_VALIDATION"'", "ttl": 3600, "prio": 100, "disabled": false}]')

# Save info for cleanup
echo $ZONE_ID > /tmp/CERTBOT_$CERTBOT_DOMAIN

# Sleep to make sure the change has time to propagate over to DNS
sleep 25

Cleanup Hook

The cleanup hook script deletes the DNS records created for domain validation at Ionos via the DNS API.

Precondition: On the system, which executes the script, curl & python must be installed!

Execution note: To display debug logs, remove the # character before the command set -x.

Cleanup Hook Shell Script

In the first step, the script checks whether a file exists for the validated domain, which contains an identifier of a DNS zone.

If this exists, all identifiers of the DNS entries that were created for the validation of the domain are retrieved for this zone via the DNS API.

Finally, all DNS entries found are deleted using their identifiers via the DNS API.

#!/bin/sh
#set -x

API_URL="https://api.hosting.ionos.com/dns/v1"
API_KEY_HEADER="X-API-Key: $API_KEY"

if [ -f /tmp/CERTBOT_$CERTBOT_DOMAIN ]; then
    ZONE_ID=$(cat /tmp/CERTBOT_$CERTBOT_DOMAIN)
    rm -f /tmp/CERTBOT_$CERTBOT_DOMAIN

    CREATE_DOMAIN="_acme-challenge.$CERTBOT_DOMAIN"
    # request the created records
    RECORD_GET_RESPONSE=$(curl -s -X GET "$API_URL/zones/$ZONE_ID?recordName=$CREATE_DOMAIN&recordType=TXT" \
                             -H "$API_KEY_HEADER" \
                             -H "Accept: application/json")
    RECORD_IDS=$(echo $RECORD_GET_RESPONSE \
            | python -c "import sys,json;records=json.load(sys.stdin)['records'];print('\n'.join([x['id'] for x in records]))")
fi

# Remove the challenge TXT record from the zone
if [ -n "$ZONE_ID" -a -n "$RECORD_IDS" ]; then
    echo "$RECORD_IDS" \
    | xargs -n1 -I {} curl -s -X DELETE "$API_URL/zones/$ZONE_ID/records/{}" \
            -H "$API_KEY_HEADER"
fi

Usage

For simplicity, we use the official Certbot docker image for this demo. The certbot tool and python are already installed. In order to create the certificate, several things have to be prepared.

A folder for the Let's Encrypt certificate structure must be created. In our example this is located under /opt/letsencrypt/cert.
A folder where the scripts are located. In our example these are located under /opt/letsencrypt/scripts and are executable (chmod +x /opt/letsencrypt/scripts/*.sh).
- /opt/letsencrypt/scripts/authenticate.sh
- /opt/letsencrypt/scripts/cleanup.sh
API access must have been requested (see above) and the <publicprefix> and <secret> must be set in the following code block.
- NOTE: A dot (.) must be placed between publicprefix and secret!
An email for expiration notifications and as identifier for a Let's Encrypt account must be set in the following code block (<email-address>).
The domain for which the certificate is to be created must be set in the following code block (<your.domain>).

Execution note: Since the Certbot docker image does not have the tool curl installed, it has to be added additionally. The easiest way to do this is to add the apk add curl command in the auth hook script, e.g. in line 3. Alternatively, you could create your own Docker image, which installs the curl tool in the associated Dockerfile.

docker run -i --rm \
  -v /opt/letsencrypt/cert:/etc/letsencrypt \
  -v /opt/letsencrypt/scripts:/tmp/scripts \
  -e "API_KEY=<publicprefix>.<secret>" \
  certbot/certbot \
  certonly \
  --keep-until-expiring \
  --preferred-challenges dns \
  --non-interactive \
  --agree-tos \
  -m <email-address> \
  --manual \
  --manual-auth-hook /tmp/scripts/authenticate.sh \
  --manual-cleanup-hook /tmp/scripts/cleanup.sh \
  -d *.<your.domain>

After successful validation, the certificate, the complete certificate chain and the corresponding private key are located in the directory /opt/letsencrypt/cert/live/<your.domain>.

Conclusion

The two shell scripts can be used to completely automate the process of applying for and renewing wildcard certificates.

For example, the command shown above can be executed regularly using a cron job on a server. This prevents the certificate from expiring and the website visitors from receiving an error message/warning.

The implementation of a native plugin for the Certbot tool would be best, but until then, the scripts shown can be used :)

Florian Storz is writing for the devlix Blog at https://www.devlix.de/blog
This article was published first here (german): https://www.devlix.de/lets-encrypt-wildcard-zertifikate-mit-ionos-dns-api-erzeugen/