API && SSH: Create and setup a server with Python and Digital Ocean

Introduction

As developers, we often have to do with server management, instance creation, and so on. Obviously, many tools allow handling this process, but wouldn't be fun to write our own server management tool? In this article, we will write a simple tool for creating a server on DigitalOcean, ready to be set up and filled with your web apps!
So let's get to work!

What you will learn;

• Interacting programmatically with ssh through Paramiko library
• Creating an interactive Command Line Tool using Inquirer
• Creating a Server ( or Droplet ) on Digital Ocean through API
• Read a YAML in Python
• Launching terminal commands from python

A bit of design and setup first

What are we going to need to start? To connect to DigitalOcean API, we first need a token. You can generate one here or:

• From Digital Ocean Dashboard, click on the API link
• Click on Generate New Token
• Choose a name for the token
• Select both Read and Write capabilities
• Click on Generate Token
• Copy the generated token immediately
• Paste it in your project folder into a new env.yaml file with a custom name, here we use doAuthToken

The next thing you need is an sshKey. If you have any already uploaded on DigitalOcean, you can use that one; copy the fingerprint in your env.yaml file under a list sshKeys.
Otherwise, create a new sshKey and add it to Digital Ocean. If you don't know how to generate an sshKey or how to upload it to your Digital Ocean team, you can find all that information here
We will also need to install some packages, so run the following command:

pip install requests paramiko inquirer yaml

Now that we have what we need, let's get the code started!

First requests

To start working with Digital Ocean's APIs, we can call some easy endpoints, which we will need later at the step of Droplet creation: the distribution and the size endpoints.
We must add the previously created token to our request headers for DO's endpoints to work.
Since we will use this header a lot, we'd better write a function to handle it :

#utils.py
def buildBasicHeaders():
    configsFile = yaml.safe_load(open('./env.yaml'))
    token = configsFile['configs']['doAuthToken']
    headers = {'Content-Type':'application/json','Authorization':'Bearer '+token}
    return headers

With the headers builder done, we can now request the distributions list:

#utils.py
def getDistributions(distribution=""):
  url = "https://api.digitalocean.com/v2/images?type=distribution"
  headers = buildBasicHeaders()
  response = requests.get(url,headers=headers)
  images = response.json()['images']
  images = list(filter(lambda i: i['status'] == 'available', images))

  return images

With this function, we require all the possible distributions for our server from the DO endpoint. Though this, they're not always all available, so we filter the result to have a list of only the available distributions. We can now require the available sizes for our droplet.

#utils.py
def getSizes():
  url = "https://api.digitalocean.com/v2/sizes"
  headers = buildBasicHeaders()
  response = requests.get(url,headers=headers)
  return response.json()['sizes']

It is similar to the previous one but more straightforward, so we can move on. We created these functions first because they are the required parameters for a droplet creation, but you can set more configurations in the build step. You can check the possible parameters here.

The first lines of the creation script

Let's now create a createServer.py file that will provide the primary process to our program.
Since we're going to ask a bunch of questions to the user, we will use the inquirer library.
Let's start easy:

#createServer.py
import utils
import inquirer

questions = [
     inquirer.Text('machineName', message="Pick a name for your machine")
]

answers = inquirer.prompt(questions)
machineName = answers['machineName']

We first ask the user to name to the newly created droplet.
The Questions variable will be the array of all our questions to the user. With the line

answers = inquirer.prompt(questions)

We're telling Inquirer to ask all the questions in the list to the user and save the results inside answers, which will be a list, having as key the value provided as the first argument of every prompt ( in this case, machineName).

We can get our sizes and distributions now that we have grasped that. It's a little more complicated, but I will explain it step by step.

#createServer.py
#....
sizes = utils.getSizes()
sizeChoices = []
for i,size in enumerate(sizes, start=1):
    choice = f"[{i}] RAM: {size['memory']}MB, CPUs: {size['vcpus']}, disk: {size['disk']}GB"
     sizeChoices.append(choice)

images = utils.getDistributions()
imageChoices = []
for i,image in enumerate(images, start=1):
    choice = f"[{i}] {image['description']}"
    imageChoices.append(choice)

questions = [
    inquirer.Text('machineName', message="Pick a name for your machine"),  
    inquirer.List('dropletSize', message="What size do you need?", choices=sizeChoices ),
    inquirer.List('dropletImage', message="What OS do you prefer?", choices=imageChoices)
]


answers = inquirer.prompt(questions)
machineName = answers['machineName']

index = sizeChoices.index(answers['dropletSize'])
dropletSize = sizes[index]['slug']

index = imageChoices.index(answers['dropletImage'])
dropletImage = images[index]['id']

Let's take a step back to understand what's going on, shall we?

First thing: the options list creation:

sizes = utils.getSizes()
sizeChoices = []
for i,size in enumerate(sizes, start=1):
    choice = f"[{i}] RAM: {size['memory']}MB, CPUs: {size['vcpus']}, disk: {size['disk']}GB"
    sizeChoices.append(choice)

images = utils.getDistributions()
imageChoices = []
for i,image in enumerate(images, start=1):
    choice = f"[{i}] {image['description']}"
    imageChoices.append(choice)

For both sizes and images, we need to enumerate them first, so we can loop through the images and have a reference index to refer to later.
After we have our array of choices, we can add these others to Questions for the user.

questions = [
    inquirer.Text('machineName', message="Pick a name for your machine"),  
    inquirer.List('dropletSize', message="What size do you need?", choices=sizeChoices ),
    inquirer.List('dropletImage', message="What OS do you prefer?", choices=imageChoices)
]

answers = inquirer.prompt(questions)

As the previous question about the machine name, the inquirer.List question type needs a key ( like dropletSize or dropletImage) and a question shown to the user. In addition, we must provide a list of choices, the lists we prepared formerly.
At this point, if we exec the command, we should have something like this:

It's a good start; what do you think?
Let's quickly explain the last part of the above code:

#....

index = sizeChoices.index(answers['dropletSize'])
dropletSize = sizes[index]['slug']


index = imageChoices.index(answers['dropletImage'])
dropletImage = images[index]['id']

Here we're hacking around a bit. Since Inquirer returns only the text of the chosen answer, we're finding its index in the choices list to get it in the original lists. After that, we retrieve the part that we need for creating the droplet, so the size's slug and the image's id.
Now comes the fun part!

Creating the droplet

It's time to create our droplet finally!

#utils.py
def createDroplet(name, size, image):
    headers = buildBasicHeaders()
    get_droplets_url = "https://api.digitalocean.com/v2/droplets"
    configsFile = yaml.safe_load(open('./env.yaml'))
    keys = configsFile['configs']['sshKeys']
    keys = getConfig('sshKeys')
    data = {
      'name':name,
      'size':size,
      'image':int(image),
      'ssh_keys': keys
    }
    response = requests.post(get_droplets_url, headers=headers,json=data)
    return response.json()['droplet']

This is all pretty straightforward, so I'll just go through a couple of points about the sshKeys:

• the sshKeys parameter in data can take a list of values: these are the keys we have on DigitalOcean that we want to be put on our new droplet to connect through ssh using them instead of user:password authentication.
• in our YAML, the sshKeys parameter will be a list of the fingerprints that can be taken from DigitalOcean sshKeys panel

That being said, before going back to createServer.py, we know we probably want to get our droplet to check out its status, so let's write a function for this too.

#utils.py
def getDroplet(dropletId):
    headers = buildBasicHeaders()
    get_droplets_url = f"https://api.digitalocean.com/v2/droplets/{dropletId}"
    response = requests.get(get_droplets_url, headers=headers)
    return response.json()['droplet']

Okay, let's use our new functions to create our droplet!

#createServer.py
newDroplet  = utils.createDroplet(machineName,dropletSize,dropletImage)

newDroplet = utils.getDroplet(newDroplet['id'])
print('[*] Creating the droplet... ', end='', flush=True)
while newDroplet['status'] != 'active' :
    newDroplet = utils.getDroplet(newDroplet['id'])
    time.sleep(1)
print('OK')
print('[*] Powering the new droplet on...', end='', flush=True)
time.sleep(60)  
print('Droplet ready')

So, what's going on here?

1. we create a Droplet with our createDroplet function
2. as soon as we create a new droplet and have got its id, we make a request to check out the status ( that for sure will not be active yet)
3. We tell the user that we are creating the droplet, then we iterate requests to the server until it says that the droplet is active; we can give feedback to our user that the droplet has been created
4. Now, we have to wait until the droplet is powered on before we can operate it. Congratulations! Now you have a brand new droplet to work on!

Connecting via SSH and installing packages

Finally, we can connect via ssh, to automatically install some packages: it's time to introduce paramiko!

#createServer.py
ssh = paramiko.SSHClient()
ssh.load_system_host_keys()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ip = newDroplet['networks']['v4'][0]['ip_address']
configsFile = yaml.safe_load(open('./env.yaml'))
path = configsFile['configs']['localKeyFile']
ssh.connect(ip, username='root',key_filename=path)
print('CONNECTED')
commands = [
        "apt-get update",
        "apt-get install -y apache2",
        # add all the commands you'd like to exec
]

for command in commands:
  ssh_stdin, ssh_stdout, ssh_stderr = ssh.exec_command(command)
  print(ssh_stdout.read().decode())
ssh_stdin.close()

Okay, once again, what is going on here? Let's give it a deeper look!

ssh = paramiko.SSHClient()
ssh.load_system_host_keys()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

First of all, we are creating an sshClient instance thanks to paramiko. That is the center of all the operations that will follow.
After that, we're automating the ssh key search on our system. Within those two lines, paramiko loads the keys from the default locations in our machine and sets the default fallback key if we don't provide any precise key (a thing that still, we'll do when connecting)

ip = newDroplet['networks']['v4'][0]['ip_address']
configsFile = yaml.safe_load(open('./env.yaml'))
path = configsFile['configs']['localKeyFile']
ssh.connect(ip, username='root',key_filename=path)
print('CONNECTED')

Here we are getting the ip of our Droplet and the path to the desired private Key we use for connection; after that, we can connect via ssh using the ssh.connect(). Now we can exec the operations we want:

commands = [
        "apt-get update",
        "apt-get install -y apache2",
        # add all the commands you'd like to exec
]

for command in commands:
  ssh_stdin, ssh_stdout, ssh_stderr = ssh.exec_command(command)
  print(ssh_stdout.read().decode())
ssh_stdin.close()

We make a list of the commands we'd like to exec: after that, we loop trough them; the ssh.exec_command(command) method allows us to exec the command and receive the output inside the ssh_stdout variable, which we will print on the screen to follow the process.
When all the commands are executed, we can close the connection.

Getting control in the end

Now that the droplet is ready and the packages are installed, we want to login into the shell and check that apache has been correctly installed. So let's end it by adding these last lines:

#createServer.py
print(f"New machine is at IP: {ip}")
webbrowser.open(f'http://{ip}')
os.system(f"ssh -o StrictHostKeyChecking=no root@{ip}")

We write out the IP of our droplet so we know it and we can take note, in case we need to ( you can always find this information on your Digital Ocean Dashboard); we then open a new browser window to that IP and then login directly on ssh, on the terminal used for the creation of the droplet!
And there you go, you've got a running server, and you're already inside the ssh terminal to continue with the following manual tasks.

Conclusion

And that's it! It has been quite intense, but in the end, I hope also clear and exciting enough. What do you think?
You can find a more complete version of the project on my Github ( still working and having fun with it, though ).
If you have any questions, feedback or just wanna reach out, feel free to write me in the comments, on twitter @gosty93 or on Linkedin
Happy Coding 1_ 0