Debugging issues with libcurl and publickey authentication

Curl has support for sftp and scp, among many other protocols. In one of our php-applications weuse libcurl to transfer files via sftp to a remote datacenter.

We recently migrated this application into a docker-container.With it we transferred the private key into the container, so the application can connect to the datacenter.

However the job that periodically transfers these files wasn’t working anymore when run inside the container.

So we connected into the container and tried to reproduce the problem with curl:

root@fac69c13bef4:/usr/share/nginx/html/typo3-web# curl sftp://10.30.2.121/ -u root: --key /root/.ssh/id_rsa
curl: (67) Authentication failure

We definetly knew that the key was correct, so we put some verbose-flags in the command:

root@fac69c13bef4:/usr/share/nginx/html/typo3-web# curl sftp://10.30.2.121/ -u root: --key /root/.ssh/id_rsa -vvv
* Trying 10.30.2.121...
* TCP_NODELAY set
* Connected to 10.30.2.121 (10.30.2.121) port 22 (#0)
* SSH MD5 fingerprint: e4dd112e8234ab62591cc8621d4b4899
* SSH host check: 0, key: AAAAB3NzaC1yc2EAAAADAQABAAAAgQDCh9qdcv1i9Y6nDwpspLaW1OosdrrtOl0t7uiof2/QYs0RTmT1DVRz0D0SNweNjtB/5069pFaNMthEh591gNrnipxy2FA2Zz7x5fv0v/AbTjmTujK14GYDBvMQTA58jGf1NWRn0+CkJvhCqY4eylkYgXdn4Y5QgGQYoEvN9P6zdQ==
* SSH authentication methods available: password,publickey
* Using SSH private key file '/root/.ssh/id_rsa'
* SSH public key authentication failed: Unable to extract public key from private key file: Method unimplemented in libgcrypt backend
* Failure connecting to agent
* Authentication failure
* Closing connection 0
curl: (67) Authentication failure

There was a more explicit error:

SSH public key authentication failed: Unable to extract public key from private key file: Method unimplemented in libgcrypt backend

That’s interesting. Normally it’s enough to just provide the private-key when connecting to other servers. That’s because the public keycan be computed from the private key. However that’s not implemented in our version of the libgcrypt backend used by libcurl.

Here’s a bug we found that mentions this: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=881720

The manpage of curl also states this:

(As of 7.39.0, curl attempts to automatically extract the public key from the private key file, so passing this option is generally not required. Note that this public key extraction requires libcurl to be linked against a copy of libssh2 1.2.8 or higher that is itself linked against OpenSSL.)

The solution was rather simple: copy the public key into the container and provide it to the curl command:

root@fac69c13bef4:/usr/share/nginx/html/typo3-web# curl sftp://10.30.2.121/ -u root: --key /root/.ssh/id_rsa --pubkey /root/.ssh/id_rsa.pub

That worked. Now we just had change the application to use the public key too and we were good.

We then asked us why this hadn’t happened outside of the container where the application was running before.

The answer to this is easy: On the host we used CentOS, the container used Debian. Implementation details.