Python Scripting Toolbox: Part 3 - Project Skeleton Generator

#python #scripting #tutorial

Cover image by Mikael Kristenson on Unsplash

This is Part 3 in the Python Scripting Toolbox series. It's a three-part survey of the tools available to us for Python scripting. I'm showing off the functionality by creating three scripts that show off different parts of the standard library. In this one, we'll use sys, os, shutil, pathlib, and argparse. Keep in mind that I'm using Python 3.6 here, so you'll see some shwanky features like "f-strings" and the pathlib module that aren't necessarily available in earlier versions. Try to use a 3.6 (or at least 3.4) if you can while following along.

In Part 1, we built shout.py: a script that shouts everything you pass into it.
In Part 2, we created make_script.py: a script that generates a starter script from a template, for use in things like Project Euler or Rosalind.
In Part 3, we are going to create project_setup.py: a script that creates a basic project skeleton, and add a couple new tools to our toolbox.

Now, let's get started.

`project_setup.py`

Once again, before we get started, it's important to say that there are a number of great libraries out there that already do this for you. Cookiecutter is probably the most popular. It's fully featured, has great docs, and a huge ecosystem of plug-ins and pre-made templates. Now that we've got that out of the way, it's time for us to boldly go and reinvent the wheel using only the Standard Library! Onward!

Here are the requirements for project_setup.py. It needs to take an input that will point it towards a source template to copy, possibly with some sane defaults and error checking. It will also need to know where to put the copy. Most importantly, it must copy the source template to the target location.

First Draft: Basic Functionality

We're going to need one old friend and two new ones to start: sys, pathlib, and shutil. We'll use sys to handle our input arguments (at least for now), pathlib to handle file paths in a humane way, and shutil to do the heavy lifting in terms of copying. Here's our first cut.

# project_skeleton.py

import pathlib
import shutil
import sys


def main(source_template, destination):
    """
    Takes in a directory as a template and copies it to the requested
    destination.
    """

    # We'll use 'expanduser' and 'resolve' to handle any '~' directories
    # or relative paths that may cause any issues for `shutil`
    src = pathlib.Path(source_template).expanduser().resolve()
    dest = pathlib.Path(destination).expanduser().resolve()

    if not src.is_dir():
        exit("Project Skeleton: Source template does not exist.")

    if dest.is_dir():
        exit("Project Skeleton: Destination already exists.")

    shutil.copytree(src, dest)
    print(f"Project Skeleton: Complete! {src} template copied to {dest}")


if __name__ == "__main__":
    if len(sys.argv) != 3:
        print(f"Usage: {sys.argv[0]} SOURCE_TEMPLATE_PATH DESTINATION_PATH")
    source, destination = sys.argv[1:]
    main(source, destination)

pathlib is a super powerful library, and it's probably my favorite of the ones we'll cover in this tutorial. It has a lot of high-level easy-to-read methods, it's cross-platform without much fuss, and it makes it so you never have to type another darn os.path.join ever again (mostly). My favorite, favorite thing about it is that it overloads the division slash (/) so that "dividing" two paths or a path and a string creates a new path, regardless of what platform you're on:

p = pathlib.Path("/usr")
b = p / "bin"
print(b)
# => /usr/bin/

As you can see, we really only use shutil for one thing: copying the directory tree from one place to another. shutil is going to be your go-to module for copying, moving, renaming, overwriting, and getting data about files and directories.

Lastly, we use good ole' sys to do some rudimentary input argument validation.

Pass 2: Argument Parsing

Just like last time, I want to add some argument parsing, via argparse. Not nearly as many options are required as last time.

import argparse
# ... The other imports and main function don't change.

if __name__ == "__main__":
    parser = argparse.ArgumentParser(
        description="Generate a project skeleton from a template.")
    parser.add_argument(
        "source", help="The directory path of the source template.")
    parser.add_argument("destination", help="The location of the new project.")

    args = parser.parse_args()
    main(args.source, args.destination)

Now our command has nice argument counting, usage messages, and a help message!

Pass 3: Having a Templates directory

This script will work great, but we have to type the full path to our template skeleton every time we use it, like some kind of peasant! Let's use a handy trick to implement some sane, cascading defaults.

"""Generates a project skeleton from a template."""

import argparse
import os        # <= Adding an 'os' import.  You'll see why below.
import pathlib
import shutil


def main(source_template, destination):
    """
    Takes in a directory (string) as a template and copies it to the requested
    destination (string).
    """
    src = pathlib.Path(source_template).expanduser().resolve()
    dest = pathlib.Path(destination).expanduser().resolve()

    if not src.is_dir():
        exit(f"Project Skeleton: Source template at {src} does not exist.")

    if dest.is_dir():
        exit(f"Project Skeleton: Destination at {dest} already exists.")

    shutil.copytree(src, dest)
    print(f"Project Skeleton: Complete! {src} template copied to {dest}")


if __name__ == "__main__":
    parser = argparse.ArgumentParser(
        description="Generate a project skeleton from a template.")

    # I want to tweak the 'source' argument, since now all we want
    # is for the user to provide the name of the template.
    parser.add_argument(
        "template", help="The name of the template to use.")
    parser.add_argument("destination", help="The location of the new project.")

    # This is the magic.  We're going to add an argument that specifies where
    # our templates live.  If the user doesn't specify, we'll see if they have
    # an environment variable set that tells us where to look.  If they don't
    # have that, we'll use a sane default.
    parser.add_argument(
        "-d",
        "--template-dir",
        default=os.environ.get("SKELETON_TEMPLATE_DIR") or "~/.skeletons",
        help="The directory that contains the project templates.  "
              "You can also set the environment var SKELETON_TEMPLATE_DIR "
              "or use the default of ~/.skeletons."
    )

    args = parser.parse_args()

    # One last tweak: We want to append the name of the template skeleton
    # to the root templates directory.
    source_dir = pathlib.Path(args.template_dir) / args.template
    main(source_dir, args.destination)

And there you have it! You can call your script in a number of ways:

$ python project_skeleton.py big_project code/new_project
# This assumes you have a template called big_project in your ~/.skeletons dir

$ SKELETON_TEMPLATE_DIR="~/code/templates/" python project_skeleton.py big_project code/new_project
# Using an environment variable.  For long-term use, put this variable
# declaration in your .bashrc :)

$ python project_skeleton.py -d ~/my/favorite/secret/place/ big_project code/new_project

Wrap Up

I was going to wild with interpolated variables into directories, filenames, and files (similar to how Cookiecutter does it), cloning from remote URLs, and more, but that seems like a bit much for one article. I think I'll leave it here, and maybe put those into a future article. If anybody wants to give it a shot, send me a link to your solution for bonus internet points!