OBSOLETION NOTICE

Since I wrote this, Travis CI has stopped offering their free service to open source projects and so I have switched to using Github Actions. This article is left here for historical interest only.

I'm not going to re-write it because there are already a bazillion other tutorials on how to do basic Github Actions stuff and it would be a waste of my time and yours for me to copy them. I have ranted about the amount of low-effort copy-cat material on this platform before. Also, I'm lazy.

Introduction

If you're anything like me, and I know I am, then your code will be used by people using all sorts of different OSes, most of which you don't have access to. So how can you make sure that it works for all those people? Obviously you write tests, you pay attention to test coverage metrics to make sure that you're testing all your code and all the conditions and all the branches it takes, and you hope that the users run the tests before installing the software.

And your users will, if those tests fail, carefully figure out what the problem is, write test cases, and a patch, and send those to you, and will diligently re-run all the tests on their bizarre platform whenever you make a change, just to make sure that you haven't re-broken it.

Yeah right. If you believe that I've got a bridge to sell you.

I mostly develop on a Mac, but most of my users are on Linux, FreeBSD, or Windows, so what I really want is a way of testing my code (automatically, so I don't forget) on those other three platforms. Actually, I want to automatically test it on Mac too, because my Mac has, over the years, had all kinds of things installed on it, and it's all too easy for me to write something that depends on something I installed four years ago and which won't work for anyone else because I forgot about the dependency.

And because I'm not getting paid for writing any of my open source code I want to do this for free. I realise that that sounds like a big ask, but it can be done.

Assumptions

I assume that your code can be run from the command line, that you use git for version control, and that you store your code on Github in a public repository. If any of those don't apply to you then you can probably work around them but I can't help you.

My project

For the purposes of this article I've written a really simple script which creates a file and a symlink to that file, and some thoroughly incomplete tests for it. Note that when you run the test script it will signal success or failure by exiting with code 0 for success or 1 for failure.

Testing on Linux

This was the easiest to find a provider for, presumably because it's the most common OS out there. I went with Travis CI, mostly because they were the first I found. You'll need to sign in to their site using Github, and to authorize it to access your account.

It will then take you to a page listing all your repositories. Confusingly, that list is ... empty.

You need to add your repositories individually, using that little + sign. Once you've done that go back to the home page and you'll see your repositories listed, and also that your most recently updated repository has no builds yet.

Travis looks for a configuration file in your repository telling it how to build the project. Moments after you push that to Github it will notice, and start building your project ...

... and if all goes well, everything will go green. Congratulations, your tests passed. Note that your repository's builds are visible to the public.

You can see all the output from your tests, which will be useful if they failed. Your tests do fail with good diagnostics, right? Travis reports the results of your tests back to Github and the test pass above shows up as a pleasing green tick next to the commit.

Or you might get an angry red cross if there was a failure. You should also get notified by email about failures, but I have found deliverability of those to be a bit dodgy - unsurprising really, auto-generated emails, all looking fairly similar, are quite likely to be mistaken for spam. If you click on the tick (or the cross) you'll see a little pop-up listing all the various automated checks that have happened, with their statuses. Right now there's only one, of course.

The config file linked above is the absolute minimum bare bones needed to work. Travis have lots of helpful stuff pre-configured for many popular languages, including tools to automatically install dependencies, run against multiple versions of an interpreter, and so on. Their online documentation is good, and if the doco leaves you confused, Travis is popular enough that you should be able to easily find examples to crib from.

Next

In part 2 we'll see how to add support for automatic testing on FreeBSD.