One of the apprenticeship patterns in Apprenticeship Patterns is Reflect as You Work. This pattern is about introspecting on how you work regularly. Doing this often allows developers to notice how their practices have changed and even how they haven't. This isn't just about observing yourself. As the book says,
"Unobtrusively watch the journeymen and master craftsmen on your team. Reflect on the practices, processes, and techniques they use to see if they can be connected to other parts of your experiences. Even as an apprentice, you can discover novel ideas simply by closely observing more experienced craftsmen as they go about their work." p. 36
I have been thinking about my own practices and those of others around me. The workflow I follow when I create new Python projects particularly stands out because I learned it from sitting with another engineer. I noted what they did and asked questions. Then, I went back to my desk and tried it myself while taking more notes. I followed the resulting workflow so many times that the steps now flow from my fingertips with ease.
I think there could be ways to optimize even this workflow but first I am going to note it down here for the potential future reader and for future me to look back on!
P.S. Many of the extra details I included here I learned from my colleagues. A big thank you to them for sharing what they know with me 💓
Prerequisites
-
pyenvis installed.
New Python Project Checklist
- Install a specific Python version.
- Create a project directory. Go to the directory.
- Set the Python version for the project.
- Create a virtual environment.
- Activate the virtual environment.
- Install dependencies.
- Save packages.
- Run the code.
Note: this workflow should work on MacOS.
PREREQUISITE: Install pyenv.
Mac OS X comes with Python 2.7 out of the box. If you haven't fiddled with anything, you should be able to open up a Terminal window and type in python --version and get some 2.7 variant. You probably don't want to use the version of Python that comes shipped with your OS (Operating System) though. There are many reasons for this like that the version may be out of date. I have even come across an important library that was missing.
Not only do you want to avoid using the version of Python that is shipped with your machine, in your work you will need to have several different versions of Python installed at once. For example, perhaps one codebase is using an older version of Python due to some library dependency. Upgrading the version of Python you are using for that project could require some refactoring of that project that you haven't prioritized. At the same time, you may be using a newer Python version on other projects because you want to take advantage of shiny new features.
Having several Python versions installed on your machine is a realistic scenario for a Python developer. Managing these versions effectively is important.
There are instructions on how to install pyenv here.
When you run a command like
pythonorpip, your operating system searches through a list of directories to find an executable file with that name. This list of directories lives in an environment variable calledPATH, with each directory in the list separated by a colon...
pyenv works by inserting a directory of shims at the front of your PATH so that when you call python or pip these shims are the first thing your OS finds. The commands you enter are, then, intercepted and sent to pyenv which decides which version of Python to use for your command based on some rules.
Follow the instructions to install pyenv. Make sure you follow the rest of the post-installation steps under Basic GitHub Checkout even if you use Homebrew to install. When I was installing I found that I had a .bashrc AND .bash_profile. Here is an article on the difference between them and when either file is used. If after following the instructions, you type in pyenv and do not get something like the following, go back and make sure you set the other bash file:
flo at MacBook-Pro in ~ $ pyenv
pyenv 1.2.8
Usage: pyenv <command> [<args>]
Some useful pyenv commands are:
commands List all available pyenv commands
local Set or show the local application-specific Python version
global Set or show the global Python version
shell Set or show the shell-specific Python version
install Install a Python version using python-build
uninstall Uninstall a specific Python version
rehash Rehash pyenv shims (run this after installing executables)
version Show the current Python version and its origin
versions List all Python versions available to pyenv
which Display the full path to an executable
whence List all Python versions that contain the given executable
See `pyenv help <command>' for information on a specific command.
For full documentation, see: https://github.com/pyenv/pyenv#readme
Step 1: Install a specific Python version.
Suppose I'm creating a script that will open the latest xkcd comic in a web browser. I'm going to run it with Python 3.7.0.
flo at MacBook-Pro in ~ $ pyenv install 3.7.0
python-build: use openssl from homebrew
python-build: use readline from homebrew
Downloading Python-3.7.0.tar.xz...
-> https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tar.xz
Installing Python-3.7.0...
python-build: use readline from homebrew
Installed Python-3.7.0 to /Users/flo/.pyenv/versions/3.7.0
Step 2: Create a project directory. Go to the directory.
flo at MacBook-Pro in ~ $ mkdir Documents/comic-creator
flo at MacBook-Pro in ~ $ cd Documents/comic-creator/
Step 3: Set the Python version for the project.
First, look at the files in the folder, even the hidden files (-la will show hidden files).
flo at MacBook-Pro in .../comic-creator $ ls -la
total 0
drwxr-xr-x 2 flo staff 64 Apr 12 21:12 .
drwx------+ 33 flo staff 1056 Apr 12 21:12 ..
Now, set the Python version for the project. Now you can see a hidden file (hidden files start with a dot). When you look inside .python-version, you can see the version we set.
flo at MacBook-Pro in .../comic-creator $ pyenv local 3.7.0
flo at MacBook-Pro in .../comic-creator $ ls -la
total 8
drwxr-xr-x 3 flo staff 96 Apr 12 21:16 .
drwx------+ 33 flo staff 1056 Apr 12 21:12 ..
-rw-r--r-- 1 flo staff 6 Apr 12 21:16 .python-version
flo at MacBook-Pro in .../comic-creator $ cat .python-version
3.7.0
Step 4: Create a virtual environment.
Just as you may have several Python versions installed on your machine, you may also have different versions of Python packages installed. Imagine the dependency graph for one of your projects looks like this:
requests==2.21.0
- certifi [required: >=2017.4.17, installed: 2019.3.9]
- chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
- idna [required: >=2.5,<2.9, installed: 2.8]
- urllib3 [required: >=1.21.1,<1.25, installed: 1.24.1]
In another project, you may be using a different version of requests which depends on a different version of certifi. By using virtual environments, we can keep package installations isolated by project.
A virtual environment is a Python environment such that the Python interpreter, libraries and scripts installed into it are isolated from those installed in other virtual environments, and (by default) any libraries installed in a “system” Python, i.e., one which is installed as part of your operating system. Python venv docs
So, first, you can verify (again) that we correctly set the Python version for the project. Now, create a virtual environment by calling venv and call that new environment venv. You can now see the environment is created.
flo at MacBook-Pro in .../comic-creator $ python --version
Python 3.7.0
flo at MacBook-Pro in .../comic-creator $ python -m venv venv
flo at MacBook-Pro in .../comic-creator $ ls
venv
Step 5: Activate the virtual environment.
Look inside venv. Then, look inside venv/bin. bin stands for binary. In Linux/Unix-like systems, executable programs needed to run the system are found in /bin. Similarly, Python executable programs are stored in bin.
Activate the virtual environment with source.
sourceis a Unix command that evaluates the file following the command executed in the current context... Frequently the "current context" is a terminal window into which the user is typing commands during an interactive session. Thesourcecommand can be abbreviated as just a dot (.) in Bash and similar POSIX-ish shells. Wikipedia
This means that if you open a new Terminal window, you will need to source the activate file again to activate the virtual environment in that window! Also note that you can type in . venv/bin/activate and it will do the exact same thing as source venv/bin/activate.
flo at MacBook-Pro in .../comic-creator $ ls venv/
bin include lib pyvenv.cfg
flo at MacBook-Pro in .../comic-creator $ ls venv/bin/
activate activate.csh activate.fish easy_install easy_install-3.7 pip pip3 pip3.7 python python3
flo at MacBook-Pro in .../comic-creator $ source venv/bin/activate
Let's look at activate:
flo at MacBook-Pro in .../comic-creator using virtualenv: venv $ cat venv/bin/activate
# This file must be used with "source bin/activate" *from bash*
# you cannot run it directly
deactivate () {
# reset old environment variables
if [ -n "${_OLD_VIRTUAL_PATH:-}" ] ; then
PATH="${_OLD_VIRTUAL_PATH:-}"
export PATH
unset _OLD_VIRTUAL_PATH
fi
if [ -n "${_OLD_VIRTUAL_PYTHONHOME:-}" ] ; then
PYTHONHOME="${_OLD_VIRTUAL_PYTHONHOME:-}"
export PYTHONHOME
unset _OLD_VIRTUAL_PYTHONHOME
fi
# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands. Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH:-}" -o -n "${ZSH_VERSION:-}" ] ; then
hash -r
fi
if [ -n "${_OLD_VIRTUAL_PS1:-}" ] ; then
PS1="${_OLD_VIRTUAL_PS1:-}"
export PS1
unset _OLD_VIRTUAL_PS1
fi
unset VIRTUAL_ENV
if [ ! "$1" = "nondestructive" ] ; then
# Self destruct!
unset -f deactivate
fi
}
# unset irrelevant variables
deactivate nondestructive
VIRTUAL_ENV="/Users/flo/Documents/comic-creator/venv"
export VIRTUAL_ENV
_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/bin:$PATH"
export PATH
# unset PYTHONHOME if set
# this will fail if PYTHONHOME is set to the empty string (which is bad anyway)
# could use `if (set -u; : $PYTHONHOME) ;` in bash
if [ -n "${PYTHONHOME:-}" ] ; then
_OLD_VIRTUAL_PYTHONHOME="${PYTHONHOME:-}"
unset PYTHONHOME
fi
if [ -z "${VIRTUAL_ENV_DISABLE_PROMPT:-}" ] ; then
_OLD_VIRTUAL_PS1="${PS1:-}"
if [ "x(venv) " != x ] ; then
PS1="(venv) ${PS1:-}"
else
if [ "`basename \"$VIRTUAL_ENV\"`" = "__" ] ; then
# special case for Aspen magic directories
# see http://www.zetadev.com/software/aspen/
PS1="[`basename \`dirname \"$VIRTUAL_ENV\"\``] $PS1"
else
PS1="(`basename \"$VIRTUAL_ENV\"`)$PS1"
fi
fi
export PS1
fi
# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands. Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH:-}" -o -n "${ZSH_VERSION:-}" ] ; then
hash -r
fi
Step 6: Install dependencies.
If you haven't come across "dependencies", this word is often used to say that something is dependent on something else... Makes sense. In our case, our Python project will depend on installing various libraries that don't come already bundled with Python 3.7.0.
This is what our code looks like:
import json
import sys
import webbrowser
import requests
# url of latest xkcd comic
URL = 'http://xkcd.com/info.0.json'
if __name__ == '__main__':
response = requests.get(URL)
if response.status_code == requests.codes.ok:
content = json.loads(response.text)
print('Comic is located at {}'.format(content['img']))
webbrowser.open(content['img'])
else:
print('Error: \n {}'.format(response.text))
sys.exit()
Create a file comic_popup.py in the project and add this code. If you try to run the code you will get an error. requests module isn't installed. Let's install it.
flo at MacBook-Pro in .../comic-creator using virtualenv: venv $ touch comic_popup.py
flo at MacBook-Pro in .../comic-creator using virtualenv: venv $ ls
comic_popup.py venv
flo at MacBook-Pro in .../comic-creator using virtualenv: venv $ pip install requests
Collecting requests
Using cached https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting urllib3<1.25,>=1.21.1 (from requests)
Using cached https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests)
Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
Using cached https://files.pythonhosted.org/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl
Installing collected packages: chardet, urllib3, idna, certifi, requests
Successfully installed certifi-2019.3.9 chardet-3.0.4 idna-2.8 requests-2.21.0 urllib3-1.24.1
You are using pip version 10.0.1, however version 19.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Step 7: Save packages.
Notice what is printed when you enter pip freeze. This command outputs installed packages in requirements format ({library-name}={version}). In the next line, redirect that output to a file called requirements.txt using >. A single > will overwrite the contents of the file if the file already existed. Using >> would append to the contents of an already existing file.
You don't have to call the file requirements.txt but that is what most Python developers use so follow the convention! More on requirements files here.
You may also notice that requests isn't the only library outputted by pip freeze. The other libraries are libraries that requests depends on so when you install requests you must install the others for requests to work. These other libraries are referred to as transitive dependencies.
flo at MacBook-Pro in .../comic-creator using virtualenv: venv $ pip freeze
certifi==2019.3.9
chardet==3.0.4
idna==2.8
requests==2.21.0
urllib3==1.24.1
You are using pip version 10.0.1, however version 19.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
flo at MacBook-Pro in .../comic-creator using virtualenv: venv $ pip freeze > requirements.txt
You are using pip version 10.0.1, however version 19.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
flo at MacBook-Pro in .../comic-creator using virtualenv: venv $ cat requirements.txt
certifi==2019.3.9
chardet==3.0.4
idna==2.8
requests==2.21.0
urllib3==1.24.1
flo at MacBook-Pro in .../comic-creator using virtualenv: venv $ pip --help
Usage:
pip <command> [options]
Commands:
install Install packages.
download Download packages.
uninstall Uninstall packages.
freeze Output installed packages in requirements format.
list List installed packages.
show Show information about installed packages.
check Verify installed packages have compatible dependencies.
config Manage local and global configuration.
search Search PyPI for packages.
wheel Build wheels from your requirements.
hash Compute hashes of package archives.
completion A helper command used for command completion.
help Show help for commands.
General Options:
-h, --help Show help.
--isolated Run pip in an isolated mode, ignoring environment variables and user configuration.
-v, --verbose Give more output. Option is additive, and can be used up to 3 times.
-V, --version Show version and exit.
-q, --quiet Give less output. Option is additive, and can be used up to 3 times (corresponding to WARNING, ERROR, and CRITICAL logging levels).
--log <path> Path to a verbose appending log.
--proxy <proxy> Specify a proxy in the form [user:passwd@]proxy.server:port.
--retries <retries> Maximum number of retries each connection should attempt (default 5 times).
--timeout <sec> Set the socket timeout (default 15 seconds).
--exists-action <action> Default action when a path already exists: (s)witch, (i)gnore, (w)ipe, (b)ackup, (a)bort).
--trusted-host <hostname> Mark this host as trusted, even though it does not have valid or any HTTPS.
--cert <path> Path to alternate CA bundle.
--client-cert <path> Path to SSL client certificate, a single file containing the private key and the certificate in PEM format.
--cache-dir <dir> Store the cache data in <dir>.
--no-cache-dir Disable the cache.
--disable-pip-version-check
Don't periodically check PyPI to determine whether a new version of pip is available for download. Implied with --no-index.
--no-color Suppress colored output
Step 8: Run the code.
That's it. You should be able to run the code now. You would be able to run it as soon as you install the dependencies it needs but don't forget to save your requirements!
flo at MacBook-Pro in .../comic-creator using virtualenv: venv $ python comic_popup.py
Comic is located at https://imgs.xkcd.com/comics/election_commentary.png
Now, if you save your code to a repo, anyone can pull the code and run it. Add a README.md and include which version of Python to use to run the code. The next developer will set up the right Python version and install the requirements by running pip install -r requirements.txt.
Don't include the .python-version file in the repo because the file is pyenv specific and other developers may have their own way to manage Python versions.
As a rule of thumb, I don't include files that are specific to me like configuration files for various IDEs (Integrated Development Environments) because they clutter up the repository. Ignore these files in your repository by adding and configuring a
.gitignorefile.
That's my development workflow when I start a Python project! I included some developer best practices where I felt it fit. I also explained as much context as I felt appropriate. I encourage you to try out different ways of doing the same thing to see the pros and cons of each.
Feel free to ask any questions! I'd love to chat about best practices and what works for you as well. So many parts of our workflows are by convention or because that's the way we first learned it or we don't know any better. I'd love to hear from you!
Top comments (10)
I think you've described the workflow well! I don't have experience with the builting
venv, I'd always usevirtualenv. I honestly don't know which is the difference.I've since moved on from
pip freezeto Pipenv mostly because it integratespip,pyenvandvirtualenv/venv, it provides a lock file for dependencies and can easily separate runtime dependencies from those in use only in development.I have the feeling there are as many combination of "managing Python packages" as there are stars. I've heard about anaconda/conda also but I've never used it.
Thanks for sharing! I had heard of Pipenv but hadn't looked into it further. Will start playing with it now!
Ugh, pipenv is soooo sweet. Loving the dependency graph feature too!
Glad you're liking it! That's really neat!
These two aliases I set in my shell might be of help (still hope they'll add them at some point):
Thanks for the aliases!
Thanks for the aliases!!!!
I am seconding pipenv! I was introduced recently, and it solves a problem that I will eventually have to deal with: getting my project to run on other peoples' machines easily during development!
Hi John, you tried getting it working with
pipenvand then ended up usingvenv?