DEV Community

Hackers And Slackers profile image Todd Birchard
Todd Birchard for Hackers And Slackers

Posted on • Originally published at hackersandslackers.com on

SSH & SCP in Python with Paramiko

#python #automation #devops

SSH & SCP in Python with Paramiko

Cloud providers have made a killing from neatly-packaged managed services for years. Whether it be databases or message brokers, developers like ourselves don't seem to have a problem paying a bit extra to have things taken care of. But wait, aren't we typically the last people to opt for less optimization and less control? Why is this the time we decide otherwise? If I had to make a guess, I'd wager it's partly because server-side DevOps kind of sucks.

As a developer, configuring or debugging a VPS is usually work which is unaccounted for, and it isn't particularly rewarding. At best, your application will probably end up running the same as your local environment. How could we make this inevitable part of our jobs better? Well, we could automate it.

Paramiko and SCP are two Python libraries we can use together to automate tasks we'd want to run on a remote host such as restarting services, making updates, or grabbing log files. We're going to take a look at what scripting with these libraries looks like. Fair warning: there's a sizable amount of code in this tutorial, which tends to make me excited enough to coerce others into my manic code-tutorial episodes. If you start to feel lost, the full repo can be found here:

GitHub logo hackersandslackers / paramiko-tutorial

📡🐍SSH & SCP in Python with Paramiko

Paramiko SSH & SCP Tutorial

Python Paramiko SCP GitHub Last Commit GitHub Issues GitHub Stars GitHub Forks

Paramiko Tutorial

Source code for the accompanying tutorial found here: https://hackersandslackers.com/ssh-scp-in-python-with-paramiko/

Getting Started

Installation

$ git clone https://github.com/hackersandslackers/paramiko-tutorial.git
$ cd paramiko-tutorial
$ make install
$ make run
Enter fullscreen mode Exit fullscreen mode

Configuration

Replace the values in .env.example with your values and rename this file to .env:

  • ENVIRONMENT: Contextual environment the script is being on.
  • SSH_REMOTE_HOST: IP address (or URL) of remote host to SSH into.
  • SSH_USERNAME: Username for connecting to remote host.
  • SSH_PASSWORD (optional): Password of user SSHing into remote host via basic auth.
  • SSH_KEY_FILEPATH: /path/to/local/sshkey
  • SCP_DESTINATION_FOLDER (optional): Remote directory to serve as destination for file uploads.

Remember to never commit secrets saved in .env files to Github.

Hackers and Slackers tutorials are free of charge. If you found this tutorial helpful, a small donation would be greatly appreciated to keep us in business. All proceeds go towards coffee, and all coffee goes towards…

View on GitHub

Paramiko leans heavily on the "in-the-weeds" side of Python libraries. If you're looking for something easy that can just get the job done, pyinfra is supposedly a great (and easy) alternative.

Setting up SSH Keys

To authenticate an SSH connection, we need to set up a private RSA SSH key. We can generate a key using the following command:

$ ssh-keygen -t rsa
Enter fullscreen mode Exit fullscreen mode
Generate an RSA key

This will prompt us to provide a name for our key. Name it whatever you like:

>> Generating a public/private rsa key pair.
>> Enter the file in which you wish to save they key (i.e., /home/username/.ssh/id_rsa):
Enter fullscreen mode Exit fullscreen mode
RSA prompt

Next, you'll be prompted to provide a password (recommended, but up to you). This will result in the creation of two keys: a private key (which I'll refer to as sshkey) and a public key which takes the naming scheme of sshkey.pub:

>> Enter passphrase (empty for no passphrase):
>> Enter same passphrase again: 
>> Your identification has been saved in /Users/username/.ssh/sshkey
>> Your public key has been saved in /Users/username/.ssh/sshkey.pub
>> The key fingerprint is:
SHA256:UFYfdydftf6764576787648574 user@My-MacBook-Pro.local
>> The key's randomart image is:
+---[RSA 3072]----+
|o*-HHFTEf.       |
|++o=++.+E.       |
|OoO + ..         |
|.= + o           |
|+ . o . S        |
|.. o o .         |
|o .+ O .         |
|.+               |
|..o.             |
+----[SHA256]-----+
Enter fullscreen mode Exit fullscreen mode

Consider the private key to be sacred; this key and everything about it should remain on your machine (the example I posted above is fake). The corresponding public key is what we put on remote hosts in our possession to authenticate a connection.

The easiest way to do this is by using ssh-copy-id, which is a command that exists for this exact purpose:

$ ssh-copy-id -i ~/.ssh/mykey user@example.com
Enter fullscreen mode Exit fullscreen mode
Copy key to remote host

In the above example, ~/.ssh/mykey is the file path of the public key we created. user@example.com should of course be replaced with the address of your remote host, where user is the username of the preferred user to connect with.

Verifying our SSH Key

It's always best to check our work. Go ahead and SSH into your remote host; if you didn't set a password when creating your key, you may be surprised to find that you're no longer prompted for a password. That's a good sign.

Check your host's .ssh directory for the public key we created:

$ ssh user@example.com
$ cd ~/.ssh
$ ls
Enter fullscreen mode Exit fullscreen mode
Check /.ssh directory

We're looking for keys that begin with the following header:

-----BEGIN RSA PRIVATE KEY-----

...

-----END RSA PRIVATE KEY-----
Enter fullscreen mode Exit fullscreen mode
id_rsa

Feel free to do the same on your VPS.

Starting our Script

Let's install our libraries. Fire up whichever virtual environment you prefer and let em rip:

$ pip3 install paramiko scp
Enter fullscreen mode Exit fullscreen mode
Install paramiko & scp

Just one more thing before we write some meaningful Python code! Create a config file to hold the variables we'll need to connect to our host. Here are the barebones of what we need to get into our server:

  • Host: The IP address or URL of the remote host we're trying to access.
  • Username: This is the username you use to SSH into your server.
  • Passphrase (optional): If you specified a passphrase when you created your ssh key, specify that here. Remember that your SSH key passphrase is not the same as your user's password.
  • SSH Key: The file path of the key we created earlier. On OSX, these live in your system's ~/.ssh folder. SSH key we're targeting must have an accompanying key with a .pub file extension. This is our public key; if you were following along earlier, this should have already been generated for you.

If you're trying to upload or download files from your remote host, you'll need to include two more variables:

  • Remote Path: The path to the remote directory we're looking to target for file transfers. We can either upload things to this folder or download the contents of it.
  • Local Path: Same idea as above, but the reverse. For our convenience, the local path we'll be using is simply /data, and contains pictures of cute fox gifs.

Now we have everything we need to make a respectable config.py file:

"""Remote host configuration."""
from os import getenv, path
from dotenv import load_dotenv

# Load environment variables from .env
BASE_DIR = path.abspath(path.dirname( __file__ ))
load_dotenv(path.join(BASE_DIR, ".env"))

# SSH Connection Variables
ENVIRONMENT = getenv("ENVIRONMENT")
SSH_REMOTE_HOST = getenv("SSH_REMOTE_HOST")
SSH_USERNAME = getenv("SSH_USERNAME")
SSH_PASSWORD = getenv("SSH_PASSWORD")
SSH_KEY_FILEPATH = getenv("SSH_KEY_FILEPATH")
SCP_DESTINATION_FOLDER = getenv("SCP_DESTINATION_FOLDER")

# Local file directory
LOCAL_FILE_DIRECTORY = f"{BASE_DIR}/files"
Enter fullscreen mode Exit fullscreen mode
config.py

Creating an SSH Client

We're going to create a class called RemoteClient to handle the interactions we'll be having with our remote host. Before we get too fancy, let's just start things off by instantiating the RemoteClient class with the variables we created in config.py :

"""Client to handle connections and actions executed against a remote host."""

class RemoteClient:
    """Client to interact with a remote host via SSH & SCP."""

    def __init__ (
        self,
        host: str,
        user: str,
        password: str,
        ssh_key_filepath: str,
        remote_path: str,
    ):
        self.host = host
        self.user = user
        self.password = password
        self.ssh_key_filepath = ssh_key_filepath
        self.remote_path = remote_path
        self.client = None
        self._upload_ssh_key()
Enter fullscreen mode Exit fullscreen mode
server.py

You'll notice I added a few things to our constructor, besides the config values we pass in:

  • self.client: self.client will ultimately serve as the connection objection in our class, similar to how you have dealt with terminology like conn in database libraries. Our connection will be None until we explicitly connect to our remote host.
  • self._upload_ssh_key() isn't a variable, but rather a function to be run automatically whenever our client is instantiated. Calling _upload_ssh_key() is telling our RemoteClient object to check for local ssh keys immediately upon creation so that we can try to pass them to our remote host. Otherwise, we wouldn't be able to establish a connection at all.

Uploading SSH Keys to a Remote Host

We've reached the section of this exercise where we need to knock out some devastatingly inglorious boilerplate code. This is typically where emotionally inferior individuals succumb to the sheer dull obscurity of understanding SSH keys and maintaining connections. Make no mistake: authenticating and managing connections to anything programmatically is overwhelmingly dull... unless your tour guide happens to be an enchanting wordsmith, serving as your loving protector through perilous obscurity. Some people call this post a tutorial. I intend to call it art.

RemoteClient will start with two private methods: _get_ssh_key() and _upload_ssh_key(). The former will fetch a locally stored public key, and if successful, the latter will deliver this public key to our remote host as an olive branch of access. Once a locally created public key exists on a remote machine, that machine will then forever trust us with our requests to connect to it: no passwords required. We'll be including proper logging along the way, just in case we run into any trouble:

"""Client to handle connections and actions executed against a remote host."""
from os import system
from paramiko import SSHClient, AutoAddPolicy, RSAKey
from paramiko.auth_handler import AuthenticationException, SSHException
from scp import SCPClient, SCPException
from .log import logger

class RemoteClient:
    """Client to interact with a remote host via SSH & SCP."""

    ...

    def _get_ssh_key(self):
        """Fetch locally stored SSH key."""
        try:
            self.ssh_key = RSAKey.from_private_key_file(
                self.ssh_key_filepath
            )
            LOGGER.info(
                f"Found SSH key at self {self.ssh_key_filepath}"
            )
            return self.ssh_key
        except SSHException as e:
            LOGGER.error(f"SSHException while getting SSH key: {e}")
        except Exception as e:
            LOGGER.error(f"Unexpected error while getting SSH key: {e}")

   def _upload_ssh_key(self):
        try:
            system(
                f"ssh-copy-id -i {self.ssh_key_filepath}.pub {self.user}@{self.host}>/dev/null 2>&1"
            )
            LOGGER.info(
                f"{self.ssh_key_filepath} uploaded to {self.host}"
            )
        except FileNotFoundError as e:
            LOGGER.error(
                f"FileNotFoundError while uploading SSH key: {e}"
            )
        except Exception as e:
            LOGGER.error(
                f"Unexpected error while uploading SSH key: {e}"
            )
Enter fullscreen mode Exit fullscreen mode
server.py

_get_ssh_key() is quite simple: it verifies that an SSH key exists at the path we specified in our config to be used for connecting to our host. If the file does in fact exist, we happily set our self.ssh_key variable, so this key can be uploaded and used by our client from here forward. Paramiko provides us with a submodule called RSAKey to easily handle all things RSA key related, like parsing a private key file into a usable connection authentication. That's what we get here:

RSAKey.from_private_key_file(self.ssh_key_filepath)
Enter fullscreen mode Exit fullscreen mode
Read RSA key from local file

If our RSA key were incomprehensible nonsense instead of a real key, Paramiko's SSHException would have caught this and raised an exception early on explaining just that. Properly utilizing a library's error handling takes a lot of the guesswork out of "what went wrong," especially in cases like where there's potential for numerous unknowns in a niche space neither of us mess with often.

_upload_ssh_key() is where we get to jam our SSH key down the throat of our remote server while shouting, "LOOK! YOU CAN TRUST ME FOREVER NOW!" To accomplish this, I go a bit "old school" by passing bash commands via Python's os.system. Unless somebody makes me aware of a cleaner approach in the comments, I'll assume this is the most badass way to handle passing keys to a remote server.

The standard non-Python way of passing keys to a host looks like this:

ssh-copy-id -i ~/.ssh/mykey user@host
Enter fullscreen mode Exit fullscreen mode
Pass SSH key to remote host

This is precisely what we accomplish in our function in Python, which looks like this:

system(
    f"ssh-copy-id -i {self.ssh_key_filepath} \
       {self.user}@{self.host}>/dev/null 2>&1"
 )
Enter fullscreen mode Exit fullscreen mode

I suppose you won't let me slip that /dev/null 2>&1 bit by you? Fine. If you must know, here's some guy on StackOverflow explaining it better than I can:

> is for redirect /dev/null is a black hole where any data sent, will be discarded. 2 is the file descriptor for Standard Error. > is for redirect. & is the symbol for file descriptor (without it, the following 1 would be considered a filename). 1 is the file descriptor for Standard O.

So we're basically telling our remote server we're giving it something, and it's all like "where do I put this thing," to which we reply "nowhere in physical in space, as this is not an object, but rather an eternal symbol of our friendship. Our remote host is then flooded with gratitude and emotion, because yes, computers do have emotions, but we can't be bothered by that right now.

Connecting to our Client

We'll add a method to our client called connect() to handle connecting to our host:

...

class RemoteClient:
    """Client to interact with a remote host via SSH & SCP."""

    ...

    @property
    def connection(self):
        """Open connection to remote host. """
        try:
            client = SSHClient()
            client.load_system_host_keys()
            client.set_missing_host_key_policy(AutoAddPolicy())
            client.connect(
                self.host,
                username=self.user,
                password=self.password,
                key_filename=self.ssh_key_filepath,
                timeout=5000,
            )
            return client
        except AuthenticationException as e:
            LOGGER.error(
                f"AuthenticationException occurred; did you remember to generate an SSH key? {e}"
            )
        except Exception as e:
            LOGGER.error(
                f"Unexpected error occurred while connecting to host: {e}"
            )
Enter fullscreen mode Exit fullscreen mode
server.py

Let's break this down:

  • client = SSHClient() sets the stage for creating an object representing our SSH client. The following lines will configure this object to make it more useful.
  • load_system_host_keys() instructs our client to look for all the hosts we've connected to in the past by looking at our system's known_hosts file and finding the SSH keys our host is expecting. We've never connected to our host in the past, so we need to specify our SSH key explicitly.
  • set_missing_host_key_policy() tells Paramiko what to do in the event of an unknown key pair. This is expecting a "policy" built-in to Paramiko, to which we're going to specific AutoAddPolicy(). Setting our policy to "auto-add" means that if we attempt to connect to an unrecognized host, Paramiko will automatically add the missing key locally.
  • connect() is SSHClient's most important method (as you might imagine). We're finally able to pass our host , user , and SSH key to achieve what we've all been waiting for: a glorious SSH connection into our server! The connect() method allows for a ton of flexibility via a vast array of optional keyword arguments too. I happen to pass a few here: setting look_for_keys to True gives Paramiko permission to look around in our ~/.ssh folder to discover SSH keys on its own, and setting timeout will automatically close connections we'll probably forget to close. We could even pass variables for things like port and password , if we had elected to connect to our host this way.

Disconnecting

We should close connections to our remote host whenever we're done using them. Failing to do so might not necessarily be disastrous, but I've had a few instances where enough hanging connections would eventually max out inbound traffic on port 22. Regardless of whether your use case might consider a reboot to be a disaster or mild inconvenience, let's just close our damn connections like adults as though we were wiping our butts after pooping. No matter your connection hygiene, I advocate for setting a timeout variable (as we saw earlier). Anyway. voila:

class RemoteClient:
    ...

    def disconnect(self):
        """Close ssh connection."""
        if self.client:
            self.client.close()
        if self.scp:
            self.scp.close()
Enter fullscreen mode Exit fullscreen mode
server.py

Fun fact: setting self.client.close() actually sets self.client to equal None, which is useful in cases where you might want to check if a connection is already open.

Executing Unix Commands

We now have a wonderful Python class that can find RSA keys, connect, and disconnect. It does lack the ability to do, well, anything useful.

We can fix this and finally begin doing "stuff" with a brand new method to execute commands, which I'll aptly dub execute_commands() (that's correct, "commands" as in potentially-more-than-one, we'll touch on that in a moment). The legwork of all this is done by the Paramiko client's built-in exec_command() method, which accepts a single string as a command and executes it:

...

class RemoteClient:
    ...

    def execute_commands(self, commands: List[str]):
        """
        Execute multiple commands in succession.

        :param List[str] commands: List of unix commands as strings.
        """
        for cmd in commands:
            stdin, stdout, stderr = self.connection.exec_command(cmd)
            stdout.channel.recv_exit_status()
            response = stdout.readlines()
            for line in response:
                LOGGER.info(
                    f"INPUT: {cmd}\n \
                    OUTPUT: {line}"
                )
Enter fullscreen mode Exit fullscreen mode
server.py

The function we just created execute_commands() expects a list of strings to execute as commands. That's partially for convenience, but it's also because Paramiko won't run any "state" changes (like changing directories) between commands, so each command we pass to Paramiko should assume we're working out of our server's root. I took the liberty of passing three such commands like so:

...

def execute_command_on_remote(remote):
    """Execute UNIX command on the remote host."""
    remote.execute_commands(
        [
            "cd /var/www/ && ls",
            "tail /var/log/nginx/access.log",
            "ps aux | grep node",
            "cd /uploads/ && ls",
        ]
    )
Enter fullscreen mode Exit fullscreen mode
__init__.py

I can view the contents of a directory by chaining cd path/to/dir && ls, but running cd path/to/dir followed by ls would result in nothingness because ls the second time returns the list of files in our server's root.

You'll notice client.exec_command(cmd) returns three values as opposed to one: this can be useful to see which input produced which output. For example, here are the full logs for the example I provided where I passed three commands to remote.execute_commands():

2020-01-02 23:20:16.103 | paramiko_tutorial.client:execute_cmd:95 - INPUT: cd /var/www/ && ls | OUTPUT: django
2020-01-02 23:20:16.103 | paramiko_tutorial.client:execute_cmd:95 - INPUT: cd /var/www/ && ls | OUTPUT: ghost
2020-01-02 23:20:16.103 | paramiko_tutorial.client:execute_cmd:95 - INPUT: cd /var/www/ && ls | OUTPUT: hackers-hbs
2020-01-02 23:20:16.103 | paramiko_tutorial.client:execute_cmd:95 - INPUT: cd /var/www/ && ls | OUTPUT: html
2020-01-02 23:20:16.104 | paramiko_tutorial.client:execute_cmd:95 - INPUT: cd /var/www/ && ls | OUTPUT: hustlers
2020-01-02 23:20:16.104 | paramiko_tutorial.client:execute_cmd:95 - INPUT: cd /var/www/ && ls | OUTPUT: pizza
2020-01-02 23:20:16.104 | paramiko_tutorial.client:execute_cmd:95 - INPUT: cd /var/www/ && ls | OUTPUT: toddbirchard

2020-01-02 23:20:16.196 | paramiko_tutorial.client:execute_cmd:95 - INPUT: tail /var/log/nginx/access.log | OUTPUT: 134.209.37.49 - - [03/Jan/2020:03:42:34 +0000] "GET / HTTP/2.0" 404 139 "-" "Mozilla/5.0 (compatible; NetcraftSurv
eyAgent/1.0; +info@netcraft.com)"
2020-01-02 23:20:16.196 | paramiko_tutorial.client:execute_cmd:95 - INPUT: tail /var/log/nginx/access.log | OUTPUT: 54.36.148.187 - - [03/Jan/2020:03:43:49 +0000] "GET /robots.txt HTTP/1.1" 404 149 "-" "Mozilla/5.0 (compatible; Ah
refsBot/6.1; +https://ahrefs.com/robot/)"
2020-01-02 23:20:16.196 | paramiko_tutorial.client:execute_cmd:95 - INPUT: tail /var/log/nginx/access.log | OUTPUT: 54.36.150.104 - - [03/Jan/2020:03:43:50 +0000] "GET / HTTP/1.1" 404 139 "-" "Mozilla/5.0 (compatible; AhrefsBot/6.
1; +https://ahrefs.com/robot/)"
2020-01-02 23:20:16.196 | paramiko_tutorial.client:execute_cmd:95 - INPUT: tail /var/log/nginx/access.log | OUTPUT: 46.229.168.146 - - [03/Jan/2020:03:48:49 +0000] "GET /robots.txt HTTP/1.1" 200 92 "-" "Mozilla/5.0 (compatible; Se
mrushBot/6~bl; +https://www.semrush.com/bot.html)"
2020-01-02 23:20:16.197 | paramiko_tutorial.client:execute_cmd:95 - INPUT: tail /var/log/nginx/access.log | OUTPUT: 46.229.168.153 - - [03/Jan/2020:03:48:50 +0000] "GET /the-art-of-technical-documentation/ HTTP/1.1" 200 9472 "-" "
Mozilla/5.0 (compatible; SemrushBot/6~bl; +https://www.semrush.com/bot.html)"
2020-01-02 23:20:16.197 | paramiko_tutorial.client:execute_cmd:95 - INPUT: tail /var/log/nginx/access.log | OUTPUT: 157.55.39.171 - - [03/Jan/2020:03:50:20 +0000] "GET /robots.txt HTTP/1.1" 200 94 "-" "Mozilla/5.0 (compatible; bin
gbot/2.0; +https://www.bing.com/bingbot.htm)"
2020-01-02 23:20:16.197 | paramiko_tutorial.client:execute_cmd:95 - INPUT: tail /var/log/nginx/access.log | OUTPUT: 40.77.167.220 - - [03/Jan/2020:03:50:24 +0000] "GET / HTTP/1.1" 200 3791 "-" "Mozilla/5.0 (compatible; bingbot/2.0
; +https://www.bing.com/bingbot.htm)"
2020-01-02 23:20:16.197 | paramiko_tutorial.client:execute_cmd:95 - INPUT: tail /var/log/nginx/access.log | OUTPUT: 54.36.150.64 - - [03/Jan/2020:03:56:47 +0000] "GET /the-ruin-of-feeling-powerless/ HTTP/2.0" 200 9605 "-" "Mozilla
/5.0 (compatible; AhrefsBot/6.1; +https://ahrefs.com/robot/)"
2020-01-02 23:20:16.197 | paramiko_tutorial.client:execute_cmd:95 - INPUT: tail /var/log/nginx/access.log | OUTPUT: 54.36.150.187 - - [03/Jan/2020:04:05:01 +0000] "GET /bigquery-and-sql-databases/ HTTP/2.0" 404 146 "-" "Mozilla/5.
0 (compatible; AhrefsBot/6.1; +https://ahrefs.com/robot/)"
2020-01-02 23:20:16.197 | paramiko_tutorial.client:execute_cmd:95 - INPUT: tail /var/log/nginx/access.log | OUTPUT: 46.229.168.149 - - [03/Jan/2020:04:13:41 +0000] "GET /robots.txt HTTP/1.1" 502 182 "-" "Mozilla/5.0 (compatible; S
emrushBot/6~bl; +https://www.semrush.com/bot.html)"

2020-01-02 23:20:16.322 | paramiko_tutorial.client:execute_cmd:95 - INPUT: ps aux | grep node | OUTPUT: ghost 1354 39.2 3.4 1223844 140172 ? Sl 04:20 0:05 /usr/bin/node current/index.js
2020-01-02 23:20:16.323 | paramiko_tutorial.client:execute_cmd:95 - INPUT: ps aux | grep node | OUTPUT: ghost 1375 36.8 3.3 1217696 135548 ? Sl 04:20 0:04 /usr/bin/node current/index.js
2020-01-02 23:20:16.323 | paramiko_tutorial.client:execute_cmd:95 - INPUT: ps aux | grep node | OUTPUT: ghost 1395 46.8 3.6 1229824 147384 ? Sl 04:20 0:06 /usr/bin/node current/index.js
2020-01-02 23:20:16.323 | paramiko_tutorial.client:execute_cmd:95 - INPUT: ps aux | grep node | OUTPUT: ghost 1410 37.7 3.2 1216320 132912 ? Sl 04:20 0:04 /usr/bin/node current/index.js
2020-01-02 23:20:16.323 | paramiko_tutorial.client:execute_cmd:95 - INPUT: ps aux | grep node | OUTPUT: root 1848 0.0 0.0 13312 3164 ? Ss 04:20 0:00 bash -c ps aux | grep node
2020-01-02 23:20:16.323 | paramiko_tutorial.client:execute_cmd:95 - INPUT: ps aux | grep node | OUTPUT: root 1850 0.0 0.0 14856 1104 ? S 04:20 0:00 grep node
Enter fullscreen mode Exit fullscreen mode
Output

Some beautiful stuff here. Now you can see which sites are on my server, which bots are spamming me, and how many node processes I'm running.

I don't want to waste much more time on the art of executing commands, but it's worth mentioning the presence of why we call stdout.channel.recv_exit_status() after each command. Waiting for recv_exit_status() to come back after running client.exec_command() forces our commands to be run synchronously, otherwise there's a likely chance our remote machine won't be about to decipher commands as fast as we pass them along.

Uploading (and Downloading) Files via SCP

SCP refers to both the protocol for copying files to remote machines (secure copy protocol) as well as the Python library, which utilizes this. We've already installed the SCP library, so import that shit.

The SCP and Paramiko libraries complement one another to make uploading via SCP super easy. SCPClient() creates an object which expects "transport" from Paramiko, which we provide with self.conn.get_transport(). Creating an SCP connection piggybacks off of our SSH client in terms of syntax, but these connections are separate. It's possible to close an SSH connection and leave an SCP connection open, so don't do that. Open an SCP connection like this:

self.scp = SCPClient(self.client.get_transport())
Enter fullscreen mode Exit fullscreen mode
Open an SCP connection

Uploading a single file is boring, so let's upload an entire directory of files instead. bulk_upload() accepts a list of file paths:

class RemoteClient:
    ...

    def bulk_upload(self, filepaths: List[str]):
        """
        Upload multiple files to a remote directory.

        :param List[str] filepaths: List of local files to be uploaded.
        """
        try:
            self.scp.put(
                filepaths,
                remote_path=self.remote_path,
                recursive=True
            )
            LOGGER.info(
                f"Finished uploading {len(filepaths)} files to {self.remote_path} on {self.host}"
            )
        except SCPException as e:
            LOGGER.error(
                f"SCPException during bulk upload: {e}"
            )
        except Exception as e:
            LOGGER.error(
                f"Unexpected exception during bulk upload: {e}"
            )
Enter fullscreen mode Exit fullscreen mode
server.py

Our method is expecting to receive a list of strings, each representing the local path to a file, we'd like to upload.

SCP's put() method will upload any number of files to a remote host. This will replace existing files with the same name if they happen to exist at the destination we specify. That's all it takes!

Downloading Files

The counterpart to SCP's put() is the get() method:

class RemoteClient:

    ...

    def download_file(self, file: str):
        """Download file from remote host."""
        self.scp.get(file)
Enter fullscreen mode Exit fullscreen mode
server.py

Our Big Beautiful Script

We now have a sick Python class to handle SSH and SCP with a remote host... let's put it to work! The following snippet is a quick way to test what we've built so far. In short, this script looks for a local folder filled with files (in my case, I filled the folder with fox gifs 🦊).

Check out how easy it is to create a main.py that handles complex tasks on remote machines thanks to our RemoteClient class:

"""Perform tasks against a remote host."""
from config import (
    host,
    local_file_directory,
    password,
    remote_path,
    ssh_key_filepath,
    username,
)

from .client import RemoteClient
from .files import fetch_local_files

def run():
    """Initialize remote host client and execute actions."""
    client = RemoteClient(
        SSH_REMOTE_HOST,
        SSH_USERNAME,
        SSH_PASSWORD,
        SSH_KEY_FILEPATH,
        SCP_DESTINATION_FOLDER,
    )
    upload_files_to_remote(client)
    execute_command_on_remote(
        client,
        [
            "mkdir /uploads",
            "cd /uploads/ && ls",
        ],
    )

def upload_files_to_remote(client: RemoteClient):
    """
    Upload files to remote via SCP.

    :param RemoteClient client: Remote server client.
    """
    local_files = fetch_local_files(LOCAL_FILE_DIRECTORY)
    client.bulk_upload(local_files)

def execute_command_on_remote(client: RemoteClient, commands: List[str]):
    """
    Execute a UNIX command remotely on a given host.

    :param RemoteClient client: Remote server client.
    :param List[str] commands: List of commands to run on remote host.
    """
    client.execute_commands(commands)
Enter fullscreen mode Exit fullscreen mode
__init__.py

Here's the output of the our upload function:

2020-01-03 00:00:27.215 | paramiko_tutorial.client:__upload_single_file:85 - Uploaded data/fox1.gif to /uploads/
2020-01-03 00:00:27.985 | paramiko_tutorial.client:__upload_single_file:85 - Uploaded data/fox2.gif to /uploads/
2020-01-03 00:00:30.015 | paramiko_tutorial.client:__upload_single_file:85 - Uploaded data/fox3.gif to /uploads/
2020-01-03 00:00:30.015 | paramiko_tutorial.client:bulk_upload:73 - Finished uploading 3 files to /uploads/ on 149.433.117.1425
Enter fullscreen mode Exit fullscreen mode
Foxes uploaded successfully

It worked! Don't believe me? Why don't we check for ourselves by running remote.execute_commands(['cd /var/www/ && ls'])?

2020-01-03 00:08:55.955 | paramiko_tutorial.client:execute_commands:96 - INPUT: cd /uploads/ && ls | OUTPUT: fox1.gif
2020-01-03 00:08:55.955 | paramiko_tutorial.client:execute_commands:96 - INPUT: cd /uploads/ && ls | OUTPUT: fox2.gif
2020-01-03 00:08:55.956 | paramiko_tutorial.client:execute_commands:96 - INPUT: cd /uploads/ && ls | OUTPUT: fox3.gif
Enter fullscreen mode Exit fullscreen mode
Output of cd /var/www/ && ls

There you have it. Straight from the fox's mouth.

Take It And Run With It

This is where I'd like to take a moment to thank all of you, and apologize that you're still here. I swore an oath to myself to stop posting tutorials over two thousand words long, and this one is looking to push five thousand words of nonsense. I'll work on that. New year, new me.

For your convenience, I've uploaded the source for this tutorial to Github. Feel free to take this and run with it! To close things out, I'll leave you with the meat and potatoes of the RemoteClient class we put together:

"""Client to handle connections and actions executed against a remote host."""
from os import system
from typing import List

from paramiko import AutoAddPolicy, RSAKey, SSHClient
from paramiko.auth_handler import AuthenticationException, SSHException
from scp import SCPClient, SCPException

from log import LOGGER

class RemoteClient:
    """Client to interact with a remote host via SSH & SCP."""

    def __init__ (
        self,
        host: str,
        user: str,
        password: str,
        ssh_key_filepath: str,
        remote_path: str,
    ):
        self.host = host
        self.user = user
        self.password = password
        self.ssh_key_filepath = ssh_key_filepath
        self.remote_path = remote_path
        self.client = None
        self._upload_ssh_key()

    @property
    def connection(self):
        """Open SSH connection to remote host."""
        try:
            client = SSHClient()
            client.load_system_host_keys()
            client.set_missing_host_key_policy(AutoAddPolicy())
            client.connect(
                self.host,
                username=self.user,
                password=self.password,
                key_filename=self.ssh_key_filepath,
                timeout=5000,
            )
            return client
        except AuthenticationException as e:
            LOGGER.error(
                f"AuthenticationException occurred; did you remember to generate an SSH key? {e}"
            )
        except Exception as e:
            LOGGER.error(f"Unexpected error occurred while connecting to host: {e}")

    @property
    def scp(self) -> SCPClient:
        conn = self.connection
        return SCPClient(conn.get_transport())

    def _get_ssh_key(self):
        """Fetch locally stored SSH key."""
        try:
            self.ssh_key = RSAKey.from_private_key_file(self.ssh_key_filepath)
            LOGGER.info(f"Found SSH key at self {self.ssh_key_filepath}")
            return self.ssh_key
        except SSHException as e:
            LOGGER.error(f"SSHException while getting SSH key: {e}")
        except Exception as e:
            LOGGER.error(f"Unexpected error while getting SSH key: {e}")

    def _upload_ssh_key(self):
        try:
            system(
                f"ssh-copy-id -i {self.ssh_key_filepath}.pub {self.user}@{self.host}>/dev/null 2>&1"
            )
            LOGGER.info(f"{self.ssh_key_filepath} uploaded to {self.host}")
        except FileNotFoundError as e:
            LOGGER.error(f"FileNotFoundError while uploading SSH key: {e}")
        except Exception as e:
            LOGGER.error(f"Unexpected error while uploading SSH key: {e}")

    def disconnect(self):
        """Close SSH & SCP connection."""
        if self.connection:
            self.client.close()
        if self.scp:
            self.scp.close()

    def bulk_upload(self, filepaths: List[str]):
        """
        Upload multiple files to a remote directory.

        :param List[str] filepaths: List of local files to be uploaded.
        """
        try:
            self.scp.put(filepaths, remote_path=self.remote_path, recursive=True)
            LOGGER.info(
                f"Finished uploading {len(filepaths)} files to {self.remote_path} on {self.host}"
            )
        except SCPException as e:
            LOGGER.error(f"SCPException during bulk upload: {e}")
        except Exception as e:
            LOGGER.error(f"Unexpected exception during bulk upload: {e}")

    def download_file(self, filepath: str):
        """
        Download file from remote host.

        :param str filepath: Path to file hosted on remote server to fetch.
        """
        self.scp.get(filepath)

    def execute_commands(self, commands: List[str]):
        """
        Execute multiple commands in succession.

        :param List[str] commands: List of unix commands as strings.
        """
        for cmd in commands:
            stdin, stdout, stderr = self.connection.exec_command(cmd)
            stdout.channel.recv_exit_status()
            response = stdout.readlines()
            for line in response:
                LOGGER.info(
                    f"INPUT: {cmd}\n \
                    OUTPUT: {line}"
                )
Enter fullscreen mode Exit fullscreen mode
server.py

The full source code for this tutorial can be found here:

GitHub logo hackersandslackers / paramiko-tutorial

📡🐍SSH & SCP in Python with Paramiko

Paramiko SSH & SCP Tutorial

Python Paramiko SCP GitHub Last Commit GitHub Issues GitHub Stars GitHub Forks

Paramiko Tutorial

Source code for the accompanying tutorial found here: https://hackersandslackers.com/ssh-scp-in-python-with-paramiko/

Getting Started

Installation

$ git clone https://github.com/hackersandslackers/paramiko-tutorial.git
$ cd paramiko-tutorial
$ make install
$ make run
Enter fullscreen mode Exit fullscreen mode

Configuration

Replace the values in .env.example with your values and rename this file to .env:

  • ENVIRONMENT: Contextual environment the script is being on.
  • SSH_REMOTE_HOST: IP address (or URL) of remote host to SSH into.
  • SSH_USERNAME: Username for connecting to remote host.
  • SSH_PASSWORD (optional): Password of user SSHing into remote host via basic auth.
  • SSH_KEY_FILEPATH: /path/to/local/sshkey
  • SCP_DESTINATION_FOLDER (optional): Remote directory to serve as destination for file uploads.

Remember to never commit secrets saved in .env files to Github.

Hackers and Slackers tutorials are free of charge. If you found this tutorial helpful, a small donation would be greatly appreciated to keep us in business. All proceeds go towards coffee, and all coffee goes towards…

View on GitHub

Top comments (0)

Read next

blamsa0mine profile image

Building a Streamlit Inventory Management App with Fragment Decorators 🚀

A0mineTV -

holawagtail_3398ffb6acec6 profile image

¡Hola Wagtail!

HolaWagtail -

blamsa0mine profile image

Interactive DataFrame Management with Streamlit Fragments 🚀

A0mineTV -

danial profile image

Integrating AWS Services with Stripe for Seamless Payment Solutions

Danial Ranjha -