Building Gophercises

This article was original posted on my website at calhoun.io, and is based on a talk I gave at the Gotham Go conference in 2018. It won't be identical to the talk, but should cover the same topics and convey the same message.

Slides from this talk can be found here.

What is Gophercises?

Gophercises is a free course I created that is composed of mini-exercises to help budding Gophers (Go developers) practice writing Go code and gain familiarity with different aspects of the language. I'm a firm believer that the only way to become a great developer is to write a lot of programs. Yes, there are other factors involved in being a great developer as well, but if you can't write code to save your life and if you are confused by somewhat simple aspects of the language you are using they will always put an upper bound on your ability. Gophercises is intended to help lift that upper bound through practice.

This write-up isn't about the Gophercises course

While I love the course and could talk about it all day, that isn't the point of this write-up or the original talk. Instead, I want to talk about how we make software overly complex to our dismay, and I intend to use the website powering Gophercises as a counter example of how we can write simpler software that still provides massive value to users.

We make software too complex

Think about the last time someone asked, "How should I design my xxx" where xxx is any basic web application. Chances are you saw responses like:

• "You should use a JSON API and a microservices design so it scales really well and each microservice can be developed independently."
• "Check out Docker and Kubernetes for your deployment. They allow for easy scaling, multi-zone deploys, etc."
• "DON'T use a framework or ORM! Just use the standard library!"

And the list of what you should and shouldn't do goes on and on as we overwhelm people with everything they are supposed to do, or the technologies they are supposed to use.

Worse yet, we tend to make it sound as if these technologies are non-optional. That to not use them would make you a bad developer, or would result in an inferior application that will fall apart at the slightest sign of growth.

Now I'm not saying that all of these technologies are bad, and I'm not trying to pick on any specific technology, but my main issue is that we have this tendency to encourage overly complex software architecture long before there is a need for it. Put more simply, we recommend people build what I would consider the 20th version of an application before they have ever built the 1st version to learn and increment from.

The end result isn't very hard to guess. We now have a whole generation of developers who practice what some refer to as "Hype Driven Development." Developers are building software that is more complex, harder to maintain, takes longer to build, and with all these extra moving pieces can be more prone to bugs.

Now I could probably sit back and not say anything if this were only occurring in projects run by experienced developers who can make educated decisions and know the trade-offs. Similarly it isn't so bad if someone makes a decision to use a technology because they are familiar with it. What really bothers me is when we start giving this advice to developers who don't know any better; developers who may not be beginners, but just don't understand the added complexities they are accepting when they opt to use microservices, SPAs, and whatever else we tell them they should be using.

In this write-up I hope to help prove that you can build useful, scalable software without doing all the things you are supposed to do. To demonstrate this point I will be discussing choices I made when building Gophercises, why I made them, and how they affected my code.

Content Management

We'll start off with my content management system (CMS). Or as others would put it - my lack of a proper content management system.

Long term I expect I'm going to need a more complex backend for my Go courses. I currently have two published courses, and I'm working on a third. At some point having custom backends for each will likely become unscalable and just a pain in the ass. It will be annoying for users who need to create an account on each course page, it will be annoying for me to maintain all these different apps that are incredibly similar. In short, I'll probably need to build a unified backend for all of my courses.

When I build that unified backend, a more complex CMS is going to become a necessity. I'll need a way to add new courses, update videos, mark whether a course is paid or free, and a million other things, but short term that isn't true at all. Short term I only needed a way to push new exercises to Gophercises, and I'm the only person doing that. I don't have non-developers or anyone else managing this content, so why go all out and build a CMS? Why not instead just write some code to seed my database with that information?

app.Exercise{
  Number:      1,
  Title:       "Quiz Game",
  Link:        "/quiz",
  Description: "Create a program to run timed quizzes via ...",
  Topics:      []string{"strings", "channels", "goroutines", "flags"},
  Duration:    36*time.Minute + 52*time.Second,
  Videos: []app.Video{
    app.Video{Name: "Overview", VimeoID: "..."},
    app.Video{Name: "Solution - Part 1", VimeoID: "..."},
    app.Video{Name: "Solution - Part 2", VimeoID: "..."},
  },
  Solutions: []app.Solution{
    app.Solution{Name: "Part 1", Branch: "solution-p1"},
    app.Solution{Name: "Part 2", Branch: "solution-p2"},
  },
},

The real code in my app has a slice of these exercises that are then used to seed the database on every deploy.

Not only is that code incredibly simple and fast to write, I also get the benefit of having a compiler verify that all my data is mostly valid. Now we could argue all day that this isn't a CMS and technically you are correct, but the goal of a CMS is to provide you with a way to get new data into your application, and this code solves that problem. Arguing over other details is mostly irrelevant to me; I'm only concerned with solving real problems.

Gophercises doesn't have a traditional authentication system

Once again, if I had a single unified backend for all of my courses I would need a real authentication system. Users would want a way to log in, change their password, view courses they have access to, add payment methods, and adjust million other little things.

It is easy to think about what could be several iterations from now, and it is easy to use that as a justification for building those features out now, but short term I wanted to ship as quickly as possible. This would allow me to start getting user feedback on both the course, and the website powering the course, allowing me to figure out what truly was working and where the website had shortcomings. All I really needed to do that was an email address in exchange for the course material. With that email address I could send the user a link to log into the course as well as updates about the course and information about any other courses or interesting write-ups (like this one) that I felt they might be interested in.

To make this a reality I built what is essentially a password reset flow. First a user enters their email address into a form. After that I do some stuff on the backend to create an account and do whatever else is necessary, then finally I send them an email with a "login" link. When the user clicks this link I assume they are the owner of that email address and log them in, much like a password reset flow would work. From this point I keep a user logged in for a few weeks much like a standard session would, and if a user loses their email with the login link they can simply enter their email address for a new one.

Some reset flows don't log you in, but allow you to change a password with a token and then log in with the new password, so at a high level they are roughly equivalent, but there are some minor differences in the details.

Now there have definitely been a few complaints about this authentication system, and I'll be the first to admit it can be more annoying that a traditional username/password login form, but there are a few perks many people often forget about:

1. This was MUCH faster to build, and I can reuse most of the code if I ever build a real auth system. That means I got to start creating exercise videos sooner and was able to start gathering feedback on both the course and the course website much sooner than if I had spent time building the auth system upfront.
2. By using this approach users don't have to worry about yet another site leaking their password due to poor security practices. Not that I'm saying I would have done a piss-poor job on the auth system, but users have no real way of verifying that I know what I'm doing or that I didn't cut corners so when you sign up with a password you used somewhere else you are basically making a leap of faith and trusting the developers. My signup process doesn't require quite as much faith (you just have to trust me with an email address).

In summary, this system isn't perfect, but it got me up and running quickly and satisfied my minimum requirements. Rather than focusing on what I thought I was going to need in a future version, I decided this would suffice for now and could be improved in the future.

Deploying

This is probably the first section where we don't look at ways I cut out features, and instead opted to use simpler technologies over industry standards.

While it may feel like the only way to deploy applications these days is with Docker, Kubernetes, etc, the reality is we can deploy Go applications in much simpler ways. In my case I decided to go with something tried and true that I was familiar with - building locally, uploading a binary to my server, and finally sshing into the server to tell it to restart the systemd service. Below is a rough outline of my deploy process (in real code).

# 1. Build the app
$ mage prod

# 2. Upload the binary to the production server
$ rsync -azP prod root@gophercises.com:/path/to/app/prod

# 3. Stop the service on the server
$ ssh root@gophercises.com "sudo service gophercises.com stop"

# 4. Reseed the database with exercise data
$ ssh root@gophercises.com "/path/to/app/prod seed --db /path/to/db"

# 5. Restart the application on the server
$ ssh root@gophercises.com "sudo service gophercises.com restart"

There are no microservices - Gophercises is hosted on a single Digital Ocean droplet (a $5 droplet!).

There is no load balancing, auto-scaling, docker, or really anything else that you might expect. Instead, I simply rely on systemd to keep my service running and this works very well for my use case.

For reference, Gophercises currently has about 10k users, so I suspect this approach would scale fairly well simply by using a more powerful web server.

This obviously won't work for everyone, but again I chose my technologies based on my requirements and experiences, and for me this was the simplest approach even if it isn't industry standard and may be frowned upon.

Serving assets

If you paid close attention to the last slide you might be wonder, "How do you serve changing assets like images, CSS files, and anything else that might not be included in your binary?"

The short answer - I don't serve anything not included in my binary. I instead embed all of these assets into my binary using packr, a library form the Buffalo "framework" (or as Mark prefers - "Web Eco-System"). Below is a brief preview of the code that makes this possible.

// Assets are compiled into the binary with gobuffalo/packr
images := packr.NewBox("../assets/img")


// Assets can be accessed via the Bytes method
imageBytes := images.Bytes(imagePath)


// Packr boxes can also be used as an http.FileSystem and then
// served via the http.FileServer handler
mux.Handle("/img/",
  http.StripPrefix("/img", http.FileServer(images)))

The first snippet creates a new packr box. This is roughly the equivalent of a directory on your filesystem, but can be built into your binary.

The second snippet demonstrates how to access the bytes of a specific file in that packr box. Again, this is very similar accessing it from the local filesystem but is all done from memory.

The third and final snippet demonstrates how packr boxes can be used as an http.FileSystem, making it incredibly easy to serve a packr box's assets in a web server just like you would a directory on the local filesystem. This is, in my opinion, where packr boxes really shine in their simplicity.

Pros and Cons of packr

There are obviously some huge downsides to this approach. For instance, my app has a much larger memory footprint, builds are slower because assets needs to be added to the binary, it is likely less performant than a CDN, and I'm probably missing a few others.

Despite all those downsides, a few very distinct benefits of using packr are what sold me on using it for this application:

1. If I have the correct binary version, I am guaranteed to also have the correct assets for that build. I don't have to worry about weird bugs that stem from the server having an updates binary and not having the correct assets.
2. Uploading and releasing only entails working with a single file, making deployments really simple.

Additional tools - Mage, Cobra, and BoltDB

I also used a few other tools that fit my needs very well and made builds and deploys simpler.

Mage for building

I mentioned that building assets into my binary was slow, so I wanted a way to improve that experience. Rather than just giving up on packr I opted to look for any quick solutions, and found one with mage.

Mage is a build tool where you write all of your build commands in Go. While many of you may know how to create Makefiles and love using them, I don't have much experience writing my own. On the other hand, I DO have quite a bit of experience writing Go code; I write code in Go every day!

Mage allowed me to create build scripts in a language I was comfortable with and in a way that was very simple and lightweight. This lead to me creating two build processes - one for build and one for dev.

# Mage is used for multiple build targets
$ mage
Targets:
  dev    Builds the development binary that reads assets from disk
  prod   Builds the production binary with embedded assets

The first - dev - builds the binary for my local OS with flags telling packr to read the assets from disk rather than bundling them into the binary. These are the choices I use 99.99% of the time in dev, so it made complete sense to just make it the default here.

The second build - prod - builds the production binary. This means building for my production server OS (ubuntu linux) and bundling all the assets into the binary. Builds with this command take a bit more time to complete, but are still relatively fast and eliminate the need to remember all the correct settings for production.

Cobra for subcommands

In a few previous code snippets I showed lines like:

# Cobra allows me to build one binary with many subcommands
$ ./app server --db /path/to/db # starts the server
$ ./app seed --db /path/to/db   # seeds the database

Cobra made it dead simple to add subcommands and flags to my binary. This is important because I don't want to accidentally run the seed command for a binary of one version, then run the server with another version. By bundling this all into a single binary I once again avoid any confusion or potential issues from version mismatching.

BoltDB for the database

I needed a way to keep track of users who sign up, exercises metadata, and anything else that you would traditionally store in a database. While I am familiar with PostgreSQL and it would have made a fine choice, I opted to venture out and use BoltDB for Gophercises.

There were a few motivations for this choice. For starters, BoltDB matches my usage pattern well. BoltDB does well with a heavy read, low write setup and Gophercises only has a few writes going on when users sign up. Aside from that 99% of the DB operations are reads.

BoltDB was also written in pure Go. Normally I don't lean towards a piece of technology solely based on the langauge it was created in, but in this case it meant that I could build a binary with the BoltDB logic embedded into it. That is, I don't have to worry about installing a database on my server. That logic is all built into my binary as long as I import the BoltDB package, once again simplifying things like server setup and deploys.

In fact, between BoltDB, packr, and Cobra I essentially have a single file I could hand off to anyone with a linux server to start a local copy of Gophercises. They won't have the same database file I use so they won't have all the same user data, but they would still have all the seeded exercises and other relevant data.

My build and deploy process focuses on my needs

Before we move on, I want to make it very clear that I'm not recommending that you go out and use the exact same setup I have here. For most of you that would be a bad choice, as your requirements are different from my own.

The point of explaining my build and deploy process was to demonstrate that:

1. You don't NEED Docker, Kubernetes, etc, and
2. You can pick and choose what to use based on what you are familiar with and what suits your current needs, even if they are unorthodox decisions.

Rendering HTML pages

While it may not seem like a popular choice these days, Gophercises does NOT use JavaScript frontend and I instead render all the HTML on the server using the standard library's html/template package.

Long term a JavaScript frontend with an API might make sense. I might have a very complex UI that benefits from a JS framework, mobile applications using the same API, and there are a number of other reasons to consider a JSON API + JS frontend design.

Short term this simply didn't make sense. The current version of Gophercises is only composed of a few unique pages; there isn't an admin portal with custom forms for creating new exercises. Videos are hosted via Vimeo, so even that portion of the application is fairly simple. Really the only reason I even considered using a JavaScript frontend was because it does a great job strictly enforcing a separation of frontend and backend logic, but this can easily be achieved without using a JavaScript frontend and I have much more experience writing Go code than JavaScript. As a result, I opted to do all my HTML rendering on the Go server and instead used the following organization techniques to isolate view specific logic from backend logic:

• Smaller, component-like templates
• Decorator pattern
• Service objects

Sidenote: If you are familiar with Ruby development, chances are you have heard of both the decorator pattern and service objects. The two are very similar in Go, but not quite the same.

Smaller, component-like templates

The basic idea here isn't anything unique or new; rather than creating a mega-template for the a page I instead create templates for individual components, much like you would in something like React. Below is an example.

{{define "exerciseWidget"}}
<div class="panel widget widget-exercise">
  <!-- ... some code omitted for brevity -->
  <div class="col-xs-12">
    <p class="mb0">Exercise {{.Number}}</p>
    <h4 class="m0">{{.Title}}</h4>
  </div>
  <!-- ... -->
  <div class="col-xs-4 text-right text-top">
    <p class="mb0">Length</p>
    <p class="m0 h4 length">{{.Duration}}</p>
  </div>
  <!-- ... -->
</div>
{{end}}

By breaking templates into small components you can easily reuse them across different pages and it becomes much easier to manage each component.

Decorator pattern

The decorator pattern is best explained with an example. Let's start with a shortened version of the Exercise type.

package app

type Exercise struct {
  Duration  time.Duration
  // + other fields
}

Inside our Exercise type we have a Duration field that is of the type time.Duration. Behind the scenes this is basically an integer that stores the durations as a number of nanoseconds. Clearly this wouldn't be useful for end users. Nobody wants to read, "This exercise is 1000000000ns long." You can't easily decide whether you have enough time to watch a video, but with times like "10 minutes" that becomes much easier to do.

Now we don't want to attach this logic to our app.Exercise, as this is UI sepcific logic, but we need to put it somewhere, so what do we do?

Another option would be to add fucntions to your templates, but once again this isn't exactly an ideal solution. Adding logic to your templates can lead to code that is incredibly hard to maintain and test.

Rather than either of these options I opted to create a new package called html, create a new Exercise type in the package, and then embed the app.Exercise in the new type.

package html

type Exercise struct {
  app.Exercise
}

From this point onwards whenever I want to pass an app.Exercise into an HTML template I'll create an html.Exercise and embed the original exercises, then pass that into the template.

var orig app.Exercise
forTemplate := html.Execise{
  Exercise: orig,
}
tpl.Execute(w, forTemplate)

This code works pretty much exactly like the old code would, but it allows us to "override" (for lack of a better term) fields by adding a method with the same name.

package html

type Exercise struct {
  app.Exercise
}

func (e Exercise) Duration() string {
  if e.Exercise.Duration < 0 {
    return "TBD"
  }
  // returns a string like “1:22:03”
  return fmt.Sprintf("%d:%02d:%02d", e.Hours(),
    e.Minutes(), e.Seconds())
}

In most Go code this would cause issues because we need to differentiate between calling a function and referencing the function with parenthesis. That is, in the following example the last two lines of code are not the same.

var e html.Exercise
e.Duration // get the function as a value, but don't call it
e.Duration() // actually call the function

Luckily that isn't true in HTML templates, which means any spots in our templates that previously referenced the Duration field will now call the new Duration method.

{{define "exerciseWidget"}}
<div class="panel widget widget-exercise">
  <!-- ... some code omitted for brevity -->
    <p class="m0 h4 length">{{.Duration}}</p>
  <!-- ... -->
</div>
{{end}}

Now we have a way to isolate our view-specific logic while not polluting our templates, and that is basically what the decorator pattern is.

Service objects

The last thing left to address are HTTP handlers and service objects.

type UserCreator struct {
  userService *db.UserService
  mgClient    *mailgun.Client
}

func (uc *UserCreator) Create(email string) (string, error) {
  token, err := uc.userService.Create(email)
  if err != nil {
    return "", err
  }
  err = uc.mgClient.Welcome(email, token)
  return token, err
}

so I'm not going to elaborate on what they are, but the primary benefits of using this pattern are:

• Code is easy to test and data requirements are clear
• HTTP handlers are simplified because they typically just call one or two service object methods
• Migrating to a JSON API in the future is fairly simple by just reusing service objects

By applying all of these techniques I was able to isolate all of my view specific logic without using a JS frontend.

Gophercises has no tests

This is probably the most controversial decision I made when building Gophercises; I chose not to write any tests.

Let me first just state for the record that I am big fan of tests. I'm currently working on a course that teaches testing in Go and would never advocate to give up testing. Despite that, I still felt it made sense to not test this application.

Why?

1. Gophercises is very simple, so testing it is quick and easy.
2. I don't make frequenty changes to Gophericses. While I will release new versions of the app with each new exercise release, none of these version changes tend to include any logic or feature changes.
3. The current version of the site was intended to provide insight into what does and does not work with the course website.

These first two both synergize very well. When an app is quick to test and isn't changing frequently it means we might actually spend less time manually testing than writing automated tests, but the third reason was also crucial because it suggests that I won't need to write tests in the future for this application. Instead I'll probably scrap a large portion of the code and use what I've learned to build a new and improved course application that includes tests.

It is important to remember that it is okay to write throwaway code to learn from. Not everyone writes PR (pull request) perfect code on their first pass, and not all code we write is easily tested without a refactor. Sometimes the best way forward is to simply accept the fact that your code won't be tested and leverage it as a learning experiment.

Key takeaways

The primary takeaway intended from this write up is that you can build simpler software in Go without getting bogged down by all the things you are "supposed" to do.

I didn't write tests for Gophercises despite the fact that I know I'm supposed to do it. I consciously weighed the pros and cons of this decision and decided that the benefits didn't justify the costs.

Another example is developers coming to Go from a Rails, Django, or Laravel background; for these developers a framework might be a perfect choice for learning and becoming familiar with Go despite most people advocated against the use of frameworks. I'm not trying to imply that they should continue to use them forever - there are many great reasons to consider not using a framework - but if a framework is going to provide a developer with more common ground and make learning and getting up to speed faster in Go then it shouldn't be ignored or shunned as a learning tool.

Stop worrying about what people say you are supposed to be doing. Stop worrying about using the latest and greatest technologies just because everyone is hyping them up. Remember that you can build simpler software in Go, and that what is simple will vary from developer to developer based on past experience.

Want to check out Gophericses?

If you want to check out the course discussed in this article you can do so (for FREE) by going here: Gophercises.com

In the course you will learn all sorts of techniques for writing Go code, but more importantly you will get tons of hands on practice that will help you become an expert at writing software in Go!

Oh, and I'm also running a sale on my premium course, Web Development with Go, to celebrate the release of the 20th and final exercise in Gophercises. The sale will end on June 4th, and in the course you will learn to build a real, production-ready web application in Go from the ground up building things like a complete authentication system from scratch.