DEV Community

Andrew (he/him)
Andrew (he/him)

Posted on • Updated on • Originally published at awwsmm.com

Managing Your Python Project with Git and PyBuilder

#git #pybuilder #python #maven

Introduction

Most of my recent work has been in Java, a language which offers lots of project management tools. The most famous of these is probably Maven. Maven helps you to keep your code and dependencies organized, and makes it easy to compile and package your code.

Maybe it's because Java tends to be a much more verbose language (and therefore needs special tools to help wrangle it into submission), or maybe it's because lots of people use Python for small scripts and helper functions, but I've struggled to find a Maven-like program to use when I'm coding in Python. Recently, I put some effort into getting a new Python project organized and I've found what I think is the best solution for managing a large Python project: PyBuilder + Git.

PyBuilder

PyBuilder is a project management tool meant primarily for Python. It organizes your code with a consistent directory structure (for source code, scripts, and tests), manages unit testing and code coverage, and can package your code for easy sharing or uploading to the Python Package Index (PyPI). Using pydoc, we can also add proper documentation to the list of available features. A full introduction to PyBuilder can be found here.

Git

Git is probably the most popular version control system (VCS) available today. By managing version control through a pseudo-filesystem, rather than the delta-based approach of other VCSes, every copy ("clone") of a Git project is a complete backup of all code committed to that repository. Distributed VCSes like Git are also supremely resistant to intentional falsification and hardware failures, and allow for extremely flexible collaborative environments. A full introduction to Git can be found here.

Why both?

Git doesn't care what your project looks like. It could have a very rigid directory structure, or it could be a directory full of GBs of cat memes. By combining the organizational power and project management of PyBuilder with the version control of Git, we get a Python project that is easy to navigate, easy to manage, and easy to dive into and start contributing to.

Note that both PyBuilder and Git could (and do) have books written about them. This tutorial is not meant to be an exhaustive introduction to either piece of software, but is meant more for someone who is trying to organize their Python project, has some familiarity with Git, and has a desire to ignore all of the pitfalls that I fell into to get to this point. Let's go!

Setting up Git

Before we install PyBuilder, we have to set up Git. I'm running this tutorial on a clean, minimal-install Ubuntu (18.04.1 LTS) VM, so I'll do everything from scratch.

Installing Git

To find out if you have Git installed already (if you forgot or if you're on a new system), run the command

andrew@ubuntuvm:~$ whereis git
Enter fullscreen mode Exit fullscreen mode

If you have Git installed, this should return something like

git: /usr/bin/git /use/share/man/man1/git.1.gz
Enter fullscreen mode Exit fullscreen mode

My new VM doesn't have Git installed, so we'll start there. Ubuntu is a Debian-based Unix-like OS, so it has the Advanced Packaging Tool, apt. Git can be installed easily on Ubuntu via apt:

andrew@ubuntuvm:~$ sudo apt-get install git
Enter fullscreen mode Exit fullscreen mode

And we can verify the installation by getting the Git version:

andrew@ubuntuvm:~$ git --version
git version 2.17.1
Enter fullscreen mode Exit fullscreen mode

Setting Up a New Git Repository

Git repositories can be cloned -- that is, copied from an existing repository -- or created from scratch. For this tutorial, I'll assume you have some Python code on your computer that you want to assemble into a new Git repository. To do that, let's first make a directory where all the code will live, and then initialize a new git repo:

andrew@ubuntuvm:~$ mkdir MyProject
andrew@ubuntuvm:~$ cd MyProject
andrew@ubuntuvm:~/MyProject$ git init
Initialized empty git repository in /home/andrew/MyProject/.git
Enter fullscreen mode Exit fullscreen mode

The .git/ directory contains all of the information that Git needs to manage your files:

andrew@ubuntuvm:~/MyProject$ sudo apt-get install tree 
andrew@ubuntuvm:~/MyProject$ tree .git/

.git/
|-- HEAD
|-- branches
|-- config
|-- description
|-- hooks
|   |-- applypatch-msg.sample
|   |-- commit-msg.sample
|   |-- post-update.sample
|   |-- pre-applypatch.sample
|   |-- pre-commit.sample
|   |-- pre-push.sample
|   |-- pre-rebase.sample
|   |-- prepare-commit-msg.sample
|   `-- update.sample
|-- info
|   `-- exclude
|-- objects
|   |-- info
|   `-- pack
`-- refs
    |-- heads
    `-- tags
Enter fullscreen mode Exit fullscreen mode

But most of the time, you won't need to worry about what's in there, you can interact with Git through git commands. Before we learn about those, though, let's get our project organized with PyBuilder.

Setting up PyBuilder

The recommended way to use PyBuilder is to install it using pip, and within a virtual environment. First, let's set up the virtual environment.

Setting up a Virtual Environment

Since I'm running this tutorial on an Ubuntu system, I already have Python installed (and I'm assuming you do, too, since you're reading this tutorial about how to organize your Python project). If you don't, the Python Software Foundation has instructions on how to download Python so you can use it on your system.

The next step involves a tool called pip, which is installed differently on different OSes and different versions of Python. In my case, I'm going to install pip with the command:

andrew@ubuntuvm:~/MyProject$ sudo apt-get install python3-pip
Enter fullscreen mode Exit fullscreen mode

...and then make sure pip is up-to-date with:

andrew@ubuntuvm:~/MyProject$ python3 -m pip install --user --upgrade pip
Enter fullscreen mode Exit fullscreen mode

Note that I'm using the python3 command and not python, since Ubuntu 18 is set up to use Python 3.6.6 by default. Next, I can use pip to install virtualenv:

andrew@ubuntuvm:~/MyProject$ python3 -m pip install --user virtualenv
Enter fullscreen mode Exit fullscreen mode

If you get a warning like

The script virtualenv is installed in `/home/andrew/.local/bin` which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Enter fullscreen mode Exit fullscreen mode

You can address this issue by adding the noted location to your system's $PATH variable:

andrew@ubuntuvm:~/MyProject$ export PATH=$PATH:/home/andrew/.local/bin
andrew@ubuntuvm:~/MyProject$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:
/usr/local/games:/snap/bin:/home/andrew/.local/bin
Enter fullscreen mode Exit fullscreen mode

Finally, create a virtual environment with

andrew@ubuntuvm:~/MyProject$ virtualenv env
Enter fullscreen mode Exit fullscreen mode

This simply creates the env directory within the MyProject directory. You can enter the virtual environment with the command

andrew@ubuntuvm:~/MyProject$ source env/bin/activate
(env) andrew@ubuntuvm:~/MyProject$
Enter fullscreen mode Exit fullscreen mode

The (env) before the command prompt indicates that you're within the virtual environment. You can exit the virtual environment at any time with the command

(env) andrew@ubuntuvm:~/MyProject$ deactivate
andrew@ubuntuvm:~/MyProject$
Enter fullscreen mode Exit fullscreen mode

Note that (env) no longer appears at the beginning of the command prompt.

Installing PyBuilder

PyBuilder is really easy to install. Enter the virtual environment and pip install:

andrew@ubuntuvm:~/MyProject$ source env/bin/activate
(env) andrew@ubuntuvm:~/MyProject$ pip install pybuilder
Enter fullscreen mode Exit fullscreen mode

We can start a blank PyBuilder project with

(env) andrew@ubuntuvm:~/MyProject$ pyb --start-project
Enter fullscreen mode Exit fullscreen mode

You can hit Enter / Return to accept all of the defaults (these can be changed if you like, but for simplicity's sake, we'll use the defaults here). Note that we don't have to type out the verbose pybuilder, but just pyb. You should now have a MyProject directory that looks like

(env) andrew@ubuntuvm:~/MyProject$ ls
build.py  docs  env  setup.py  src
Enter fullscreen mode Exit fullscreen mode

...where docs, env, and src are all directories. Your src directory should look like:

(env) andrew@ubuntuvm:~/MyProject$ tree src/
src/
|-- main
|   |-- python
|   `-- scripts
`-- unittest
    `-- python

5 directories, 0 files
Enter fullscreen mode Exit fullscreen mode

The next thing to do is to add some source code to src/main/python, some scripts to /src/main/scripts and some unit tests to /src/unittest/python. To make things easy on myself, I like to make softlinks within environments like this:

(env) andrew@ubuntuvm:~/MyProject$ ln -s src/main/python/ source
(env) andrew@ubuntuvm:~/MyProject$ ln -s src/main/scripts/ script
(env) andrew@ubuntuvm:~/MyProject$ ln -s src/unittest/python test
(env) andrew@ubuntuvm:~/MyProject$ ls
build.py  docs  env  script  setup.py  source  src  test
Enter fullscreen mode Exit fullscreen mode

Now I don't need to remember if the directory was src/main/script or src/python/scripts or if the unit tests are in src/unittests or src/python/main/unittest or whatever. Everything is in the main directory and everything is singular (ie. test instead of tests).

Managing a PyBuilder Project

PyBuilder, like many other project management frameworks, is based on the idea of a "build lifecycle", where there are common tasks you'll want to run over and over, like "compile", "run", "document", etc. As an interpreted language, Python doesn't have a "compile" stage in its build lifecycle, but it has most of the main ones. PyBuilder defines lots of different lifecycle stages, but the most common ones are

  • pyb publish and
  • pyb clean

"What about unit tests?" you cry. "WHAT ABOUT CODE COVERAGE?" you scream in anguish. They're covered under publish, don't worry. Before we add any code to the project, let's take a look at build.py:

from pybuilder.core import use_plugin, init

use_plugin("python.core")
use_plugin("python.unittest")
use_plugin("python.install_dependencies")
use_plugin("python.flake8")
use_plugin("python.coverage")
use_plugin("python.distutils")


name = "MyProject"
default_task = "publish"


@init
def set_properties(project):
    pass
Enter fullscreen mode Exit fullscreen mode

build.py defines the plugins, settings, tasks, and so on that will be used throughout the build lifecycle. Let's add a few things here, like a description, an author, and some project properties (it's difficult to find a list of these online, but if you search for self.home_page in PyBuilder's core.py code, that list is pretty close to what's available to use):

from pybuilder.core import use_plugin, init, Author

use_plugin("python.core")
use_plugin("python.unittest")
use_plugin("python.install_dependencies")
use_plugin("python.flake8")
use_plugin("python.coverage")
use_plugin("python.distutils")

name    = "MyProject"
version = "1.0"

summary = "Example PyBuilder / Git project"
url     = "https://github.com/awwsmm/PybGit"

description = """An example PyBuilder / Git project for project management
and file version control. See blog post at http://bit.ly/2QY65wO for a
more through explanation."""

authors      = [Author("Andrew Watson", "andrew.watson@nibrt.ie")]
license      = "None"
default_task = "publish"

@init
def initialize(project):
    project.build_depends_on("mockito")

@init
def set_properties(project):
    pass
Enter fullscreen mode Exit fullscreen mode

Note that we had to add Author to the import list to add a list of authors to the project. Everything else is fairly self explanatory; note in particular that the GitHub URL doesn't need to match the name of the directory in which we've saved the project. We also added a dependency on mockito, which we'll use for some unit testing. What about default_task, though? That defines what PyBuilder should do when we just enter pyb at the command prompt, with no additional arguments. Let's try that now:

(env) andrew@ubuntuvm:~/MyProject$ pyb
PyBuilder version 0.11.17
Build started at 2018-09-25 12:45:38
------------------------------------------------------------
[INFO]  Building MyProject version 1.0
[INFO]  Executing build in /home/andrew/MyProject
[INFO]  Going to execute task publish
[INFO]  Running unit tests
[INFO]  Executing unit tests from Python modules in /home/andrew/MyProject/src/unittest/python
[WARN]  No unit tests executed.
[INFO]  All unit tests passed.
[INFO]  Building distribution in /home/andrew/MyProject/target/dist/MyProject-1.0
[INFO]  Copying scripts to /home/andrew/MyProject/target/dist/MyProject- 1.0/scripts
[INFO]  Writing setup.py as /home/andrew/MyProject/target/dist/MyProject-1.0/setup.py
[INFO]  Collecting coverage information
[WARN]  coverage_branch_threshold_warn is 0 and branch coverage will not be checked
[WARN]  coverage_branch_partial_threshold_warn is 0 and partial branch coverage will not be checked
[INFO]  Running unit tests
[INFO]  Executing unit tests from Python modules in /home/andrew/MyProject/src/unittest/python
[WARN]  No unit tests executed.
[INFO]  All unit tests passed.
Coverage.py warning: No data was collected. (no-data-collected)
[INFO]  Overall coverage is 100%
[INFO]  Overall coverage branch coverage is 100%
[INFO]  Overall coverage partial branch coverage is 100%
------------------------------------------------------------
BUILD FAILED - No data to report.
------------------------------------------------------------
Build finished at 2018-09-25 12:45:38
Build took 0 seconds (619 ms)
Enter fullscreen mode Exit fullscreen mode

We got the error message BUILD FAILED - No data to report. because we don't yet have any files in the project. Let's add a few.

Add the following text to /src/main/python/example_hello.py (aka. source/example_hello.py):

import sys

"""Prints \"Hello World!\" to the given output stream."""
def hello_world(out):
    """
    Prints \"Hello World!\".
    >>> import sys
    >>> hello_world(sys.stdout)
    Hello World!\n
    """
    out.write("Hello World!\n")

"""

Below:

 > Required in order to run the documentation checker on this code.

 > The comment
     pragma: no cover
   means that this block of code will not be checked for unit test
   coverage by the `coverage` module.

"""

if __name__ == "__main__": # pragma: no cover
    import doctest
    doctest.testmod()
Enter fullscreen mode Exit fullscreen mode

Next, we need to add some unit tests for this source code. Add the following text to test/example_hello_tests.py:

from mockito import mock, verify
import unittest

from example_hello import hello_world

class HelloWorldTest(unittest.TestCase):

    def test_correct_message(self):
        out = mock()
        hello_world(out)
        verify(out).write("Hello World!\n")
Enter fullscreen mode Exit fullscreen mode

By convention (and required by PyBuilder), the unit tests for source/abcd.py will be in test/abcd_tests.py. Note that, above, we're using both the standard Python unit test package unittest as well as the mock object package mockito. Mockito isn't strictly necessary for unit testing, but it can make things faster and easier.

We're also importing the method hello_world from the file example_hello (without the .py). PyBuilder knows where to find these files and everything "just works". Let's try building our package again with pyb:

(env) andrew@ubuntuvm:~/MyProject$ pyb
PyBuilder version 0.11.17
Build started at 2018-09-25 14:22:11
------------------------------------------------------------
[INFO]  Building MyProject version 1.0
[INFO]  Executing build in /home/andrew/MyProject
[INFO]  Going to execute task publish
[INFO]  Running unit tests
[INFO]  Executing unit tests from Python modules in /home/andrew/MyProject/src/unittest/python
[INFO]  Executed 1 unit tests
[ERROR] Test has error: unittest.loader._FailedTest.example_hello_tests
------------------------------------------------------------
BUILD FAILED - There were 1 error(s) and 0 failure(s) in unit tests
------------------------------------------------------------
Build finished at 2018-09-25 14:22:11
Build took 0 seconds (318 ms)
Enter fullscreen mode Exit fullscreen mode

What happened? Let's run pyb again with the -v (verbose) flag:

(env) andrew@ubuntuvm:~/MyProject$ pyb -v
PyBuilder version 0.11.17
Build started at 2018-09-25 14:22:21
------------------------------------------------------------
[INFO]  Building MyProject version 1.0
[INFO]  Executing build in /home/andrew/MyProject
[INFO]  Going to execute task publish
[INFO]  Running unit tests
[INFO]  Executing unit tests from Python modules in /home/andrew/MyProject/src/unittest/python
[INFO]  Executed 1 unit tests
[ERROR] Test has error: unittest.loader._FailedTest.example_hello_tests
ImportError: Failed to import test module: example_hello_tests
Traceback (most recent call last):
  File "/usr/lib/python3.5/unittest/loader.py", line 153, in loadTestsFromName
    module = __import__(module_name)
  File "/home/andrew/MyProject/src/unittest/python/example_hello_tests.py", line 1, in <module>
    from mockito import mock, verify
ImportError: No module named 'mockito'


------------------------------------------------------------
BUILD FAILED - There were 1 error(s) and 0 failure(s) in unit tests
------------------------------------------------------------
Build finished at 2018-09-25 14:22:21
Build took 0 seconds (312 ms)
Enter fullscreen mode Exit fullscreen mode

Oh! It's because we forgot to install the dependencies that we included in build.py (namely, Mockito). Let's do that now:

(env) andrew@ubuntuvm:~/MyProject$ pyb install_dependencies
PyBuilder version 0.11.17
Build started at 2018-09-25 14:25:13
------------------------------------------------------------
[INFO]  Building MyProject version 1.0
[INFO]  Executing build in /home/andrew/MyProject
[INFO]  Going to execute task install_dependencies
[INFO]  Installing all dependencies
[INFO]  Processing batch dependency 'mockito'
------------------------------------------------------------
BUILD SUCCESSFUL
------------------------------------------------------------
Build Summary
             Project: MyProject
             Version: 1.0
      Base directory: /home/andrew/MyProject
        Environments:
               Tasks: install_dependencies [3247 ms]
Build finished at 2018-09-25 14:25:16
Build took 3 seconds (3259 ms)
Enter fullscreen mode Exit fullscreen mode

Yes, it really is that easy. We can run pyb again now, and we get a BUILD SUCCESSFUL at the end:

(env) andrew@ubuntuvm:~/MyProject$ pyb
PyBuilder version 0.11.17
Build started at 2018-09-25 14:25:19
------------------------------------------------------------
[INFO]  Building MyProject version 1.0
[INFO]  Executing build in /home/andrew/MyProject
[INFO]  Going to execute task publish
[INFO]  Running unit tests
[INFO]  Executing unit tests from Python modules in /home/andrew/MyProject/src/unittest/python
[INFO]  Executed 1 unit tests
[INFO]  All unit tests passed.
[INFO]  Building distribution in /home/andrew/MyProject/target/dist/MyProject-1.0
[INFO]  Copying scripts to /home/andrew/MyProject/target/dist/MyProject-1.0/scripts
[INFO]  Writing setup.py as /home/andrew/MyProject/target/dist/MyProject-1.0/setup.py
[INFO]  Collecting coverage information
[WARN]  coverage_branch_threshold_warn is 0 and branch coverage will not be checked
[WARN]  coverage_branch_partial_threshold_warn is 0 and partial branch coverage will not be checked
[INFO]  Running unit tests
[INFO]  Executing unit tests from Python modules in /home/andrew/MyProject/src/unittest/python
[INFO]  Executed 1 unit tests
[INFO]  All unit tests passed.
[INFO]  Overall coverage is 100%
[INFO]  Overall coverage branch coverage is 100%
[INFO]  Overall coverage partial branch coverage is 100%
[INFO]  Building binary distribution in /home/andrew/MyProject/target/dist/MyProject-1.0
------------------------------------------------------------
BUILD SUCCESSFUL
------------------------------------------------------------
Build Summary
             Project: MyProject
             Version: 1.0
      Base directory: /home/andrew/MyProject
        Environments:
               Tasks: prepare [248 ms] compile_sources [0 ms] run_unit_tests [40 ms] package [3 ms] run_integration_tests [0 ms] verify [401 ms] publish [516 ms]
Build finished at 2018-09-25 14:25:20
Build took 1 seconds (1222 ms)
Enter fullscreen mode Exit fullscreen mode

Don't forget to always run pyb install_dependencies after you add any new dependencies to the build.py file. Note that PyBuilder ran our unit test that we defined in example_hello_tests.py. As long as you place the tests in the correct directory, PyBuilder knows where to find them and what to do with them. Let's add another, slightly more complicated, example. Create a new file called source/example_primes.py and add the following text to it:

import math
import itertools

"""Returns a list of N primes, beginning with [2, 3, 5...]."""
def first_N_primes(N):
    """
    Returns a list of N primes, beginning with [2, 3, 5...]. Returns an empty
    list if the user supplies a non-positive number. Floating point numbers
    are rounded up.
    >>> first_N_primes(0)
    []
    >>> first_N_primes(3.14)
    [2, 3, 5, 7]
    >>> first_N_primes(-42)
    []
    >>> first_N_primes(10)
    [2, 3, 5, 7, 11, 13, 17, 19, 23, 29]
    """
    if (N <= 0):
        return []

    primes = [2]
    prime  = (x for x in itertools.count(2) if not any ([x%y==0 for y in range(2, int(math.ceil(math.sqrt(x)))+1)]))

    for n in range(1, int(math.ceil(N))):
        primes.append(next(prime))

    return primes

if __name__ == "__main__": # pragma: no cover
    import doctest
    doctest.testmod()
Enter fullscreen mode Exit fullscreen mode

Note that we've again added this "doctest" block at the bottom of this bit of source code... why? It allows us to verify that the documentation we've written (example usage, beginning with >>>) actually works as advertised:

(env) andrew@ubuntuvm:~/MyProject$ python3 source/example_primes.py -v
Trying:
    first_N_primes(0)
Expecting:
    []
ok
Trying:
    first_N_primes(3.14)
Expecting:
    [2, 3, 5, 7]
ok
Trying:
    first_N_primes(-42)
Expecting:
    []
ok
Trying:
    first_N_primes(10)
Expecting:
    [2, 3, 5, 7, 11, 13, 17, 19, 23, 29]
ok
1 items had no tests:
    __main__
1 items passed all tests:
   4 tests in __main__.first_N_primes
4 tests in 2 items.
4 passed and 0 failed.
Test passed.
Enter fullscreen mode Exit fullscreen mode

I don't know about you, but I think that's pretty cool. doctest verifies that our documentation examples actually produce the output that we say they should. If you only want to view the documentation, you can do that with:

(env) andrew@ubuntuvm:~/MyProject$ python3 -i source/example_primes.py
>>> help(first_N_primes)
Enter fullscreen mode Exit fullscreen mode

...or just

(env) andrew@ubuntuvm:~/MyProject$ pydoc ./source/example_primes.py
Enter fullscreen mode Exit fullscreen mode

Note that you can also get nicely-formatted HTML help by adding the -w flag above. Let's add another set of unit tests for example_primes. In test/example_primes_tests.py, add the following lines of code:

import unittest

from example_primes import first_N_primes

class FirstNPrimesTest(unittest.TestCase):

    """ test negative float input """
    def test_negFloatInput(self):
        self.assertEqual(first_N_primes(-14.3), [])

    """ test positive integer input """
    def test_posIntInput(self):
        self.assertEqual(first_N_primes(6), [2, 3, 5, 7, 11, 13])
Enter fullscreen mode Exit fullscreen mode

Running pyb again verifies that we have 3 unit tests (all of which passed) and that we have 100% code coverage! Success!

Python Scripts in PyBuilder

One more thing worth touching on for this intro to PyBuilder is the idea of scripts. A good way to keep code neat is to keep method and class definitions in source/ and runnable scripts which use those methods and classes in script/. Let's write a short script that uses the single method defined in source/example_primes.py. Create a file called script/example_primes_script.py and add the following code to it:

#!/usr/bin/env python
import sys

from example_primes import first_N_primes

N = 10
print("The first",N,"primes are:")

for n in first_N_primes(N):
    print(n)
print()
Enter fullscreen mode Exit fullscreen mode

We then re-publish the code with pyb, and we can install the package locally (within the virtual environment) with the command:

(env) andrew@ubuntuvm:~/MyProject$ pip install target/diest/MyProject-1.0/dist/MyProject-1.0.tar.gz
Enter fullscreen mode Exit fullscreen mode

This makes all of the scripts in the project available for us to use, right on the command line. Simply write:

(env) andrew@ubuntuvm:~/MyProject$ example_primes_script.py
The first 10 primes are:
2
3
5
7
11
13
17
19
23
29
Enter fullscreen mode Exit fullscreen mode

Ta da! PyBuilder, again, knows where the scripts are and makes them available to use. If you change any code, make sure you re-publish with pyb and reinstall using the pip install command above. That will make your code available to use freely within the virtual environment.

Managing Things with Git

As I said earlier, this doesn't attempt to be an introduction to Git by any means, but there are a few commands here that we can review, which will help us to maintain this project. First, let's see what's changed since we initialized everything with git init way back at the beginning of this tutorial:

(env) andrew@ubuntuvm:~/MyProject$ git status
On branch master

Initial commit

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        .coverage
        __pycache__/
        build.py
        env/
        script
        setup.py
        source
        src/
        target/
        test

nothing added to commit but untracked files present (use "git add" to track)
Enter fullscreen mode Exit fullscreen mode

...okay, everything. A good .gitignore file is probably in order, since we don't want our repo to be clogged with the hundreds of files in our virtual environment directory, env. Probably, we only want to commit our source code, unit tests, scripts, and information about our build environment (also maybe our soft links). Let's gitignore everything else. Create a file called .gitignore and add the following lines to it:

.coverage
__pycache__
env
target
Enter fullscreen mode Exit fullscreen mode

If directories are added to .gitignore, that directory and everything in it is ignored when we run git status. Let's see what we have left:

(env) andrew@ubuntuvm:~/MyProject$ git status

On branch master

Initial commit

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        .gitignore
        build.py
        script
        setup.py
        source
        src/
        test

nothing added to commit but untracked files present (use "git add" to track)
Enter fullscreen mode Exit fullscreen mode

Okay! Just code, soft links, and some configuration files. Let's add all of this to our initial commit:

(env) andrew@ubuntuvm:~/MyProject$ git add .gitignore

(env) andrew@ubuntuvm:~/MyProject$ git add *
Enter fullscreen mode Exit fullscreen mode

Note that here you might get a warning about some files being ignored -- that's fine, we purposefully put those files in .gitignore because we wanted to ignore them. Let's commit these files to our local git repo:

(env) andrew@ubuntuvm:~/MyProject$ git commit -am "initial commit"
Enter fullscreen mode Exit fullscreen mode

We've committed all (-a) the files we had staged and added a message (-m) which indicates that this is our first commit to this repo. Next, if you have a GitHub (or other similar account like BitBucket, etc.), you can easily create a remote repo to host your code.

Creating a GitHub Repository

To create a remote repository hosted on GitHub, simply go to your profile page:

Click the "Repositories" link at the top:

Then click the green "New" button over on the right:

Name your repo, give it a description...

...and follow the instructions! We want to "push an existing repository from the command line", because we've already got our directory structure all figured out and added our files, etc. Make sure you've added your ssh key to GitHub so that you can push to your remote repo, and run the following two commands (as the instructions tell you to do):

(env) andrew@ubuntuvm:~/MyProject$ git remote add origin git@github.com:awwsmm/PybGit.git

(env) andrew@ubuntuvm:~/MyProject$ git push -u origin master
Enter fullscreen mode Exit fullscreen mode

That's it! You now have a Git/PyBuilder project with version control, unit tests, documentation, and a nice, organized directory structure:

Oh Wait, One More Thing

Say we forgot to add a file to our remote repo, or we've changed something and we want to update it. No worries. Now that we have our remote repo up and running, it's pretty easy. Let's make a new file called script/example_hello_script.py and add the following code to it:

#!/usr/bin/env python
import sys

"""
Writes "Hello from a script!" to stdout.
"""

sys.stdout.write("Hello from a script!\n")
Enter fullscreen mode Exit fullscreen mode

Nothing special here. But we need to go through a few steps. First, re-publish the project with pyb publish (or just pyb, as we've been doing, since we have the default_task set to "publish" in our build.py file):

(env) andrew@ubuntuvm:~/MyProject$ pyb
Enter fullscreen mode Exit fullscreen mode

Then, re-install the package in our virtual environment:

(env) andrew@ubuntuvm:~/MyProject$ pip install target/dist/MyProject-1.0/dist/MyProject-1.0.tar.gz
Enter fullscreen mode Exit fullscreen mode

Verify that our new script works before we commit it to our local repo:

(env) andrew@ubuntuvm:~/MyProject$ example_hello_script.py
Hello from a script!
Enter fullscreen mode Exit fullscreen mode

It does, cool. If we run git status, we can see that the repository has changed -- we have a new untracked file.

(env) andrew@ubuntuvm:~/MyProject$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

        src/main/scripts/example_hello_script.py

nothing added to commit but untracked files present (use "git add" to track)
Enter fullscreen mode Exit fullscreen mode

Add it to Git (make sure it's tracked) with:

(env) andrew@ubuntuvm:~/MyProject$ git add src/main/scripts/example_hello_script.py
Enter fullscreen mode Exit fullscreen mode

(Note that we can't use our soft links here to make this command shorter.) Commit our changes to the local repository with

(env) andrew@ubuntuvm:~/MyProject$ git commit -am "new hello script"
Enter fullscreen mode Exit fullscreen mode

...and push them to the remote one with

(env) andrew@ubuntuvm:~/MyProject$ git push
Enter fullscreen mode Exit fullscreen mode

We can see that this commit has been pushed to the remote repo (there are now two commits instead of one):

Conclusion

And that's it. Hopefully this guide has helped you to avoid some of the missteps that I made in getting my PyBuilder/Git project up and running. Feel free to comment below or message me privately with any questions!

All the code from this post is available at

GitHub logo awwsmm / PybGit

Example PyBuilder / Git project

PybGit

Example PyBuilder / Git project See the blog post at Dev.To

View on GitHub

Top comments (6)

Collapse
 
glocmike profile image
Michael •

Wow! Comprehensive. I've been trying to wrangle Python packaging and project structure recently too! It's a minefield (boom)!! Had never heard of PyBuilder before, nor pydoc so I'm keen to give it a try.

My first read of an article on dev.to :) and my first comment too :)

Thanks for the great work! Must have taken quite a bit of dark energy to hack this out ;)

Collapse
 
awwsmm profile image
Andrew (he/him) •

I've only just started using them, but they've been really helpful so far. Let me know if you learn any useful tips or tricks!

Collapse
 
glocmike profile image
Michael •

Sure will!

Collapse
 
shostarsson profile image
Rémi Lavedrine •

I wanted to package my python application for a long time but never find a comprehensive way to do that.
Thank you for your article. It is very clear.

I am eager to see if I can do it on my own project.
I am on Ubuntu as well so I can follow your example. :-D

Collapse
 
awwsmm profile image
Andrew (he/him) •

Glad to help! Let me know how it works out :)

Collapse
 
simone010 profile image
Simone Lockman •

beautiful article!

Read next

roalcantara profile image

Simplifying Your Git Workflow with Aliases

Rogério Rodrigues de Alcântara -

noberst profile image

🛠️How to Go from Software Engineer to AI Developer - What it means for YOU (Insider's View)🤖

Namee -

bekbrace profile image

Speed Test : Python Vs Rust

Bek Brace -

reneenoble profile image

Welcome to 14 days of Data Science!

Renee Noble -