How to fix huggingface ssh authentication error with git

So I recently started to use huggingface.co to store and retrieve models for our AI project at work, and as it essentially works like any other git repositories, I wanted to clone a model I needed to use, and used the SSH protocol to do so, as I really don't want to enter credentials every time, and it's also a bit more secure overall.
So I added my public SSH Key in the account settings, and went my to clone the repo, expecting to work straight away.

But it didn't.

What I was getting was the following error:

git clone git@hf.co:your-org/your-model
Cloning into 'your-model'...
remote: 
remote: ========================================================================
remote: 
remote: ERROR: This action requires authentication.
Make sure your SSH public key is linked with your profile on huggingface.co.
See https://huggingface.co/settings/keys.

remote: 
remote: ========================================================================
remote: 
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

I took me a while to figure it out, I added and remove the same key from the account, checked the permissions (but the key was working fine for Github, so clearly that wasn't the issue, but always better to check anyway).

I tried to follow the steps in this page, and again, everything seemed in order, until I run this command they listed at the end:

ssh -T git@hf.co

# Hi anonymous, welcome to Hugging Face.

And instead of getting the username from the welcome message, as I do from github.com, it was returning an anonimous user.
So I quickly checked if the SSH agent was actually using the key I use for other sites, by running the following command:

ssh-add -l

The above will list all the keys you have in the SSH Agent, and in my case I had other keys, but not the one I was using!

To fix that all you need to do is to run the following command:

ssh-add ~/.ssh/your-key

# Or if you want to use the default id_* key, just do
ssh-add

Ran again the commands above, and finally I could see that my key was used, and I could finally clone the repo!