Why Can't We Settle on a Single Web App Structure in Go?

This article is part of a larger series on Go application structure that was originally posted on calhoun.io, where I write about Go, web dev, testing, and more. I will be porting the entire series over to Dev.to, but I appreciate you checking out on my website anyway 😀

Getting started in Go can be hard. The language itself is pretty easy to pick up, but figuring out how to structure your application can become overwhelming early on. At least it was a big time sink for me coming from a Ruby on Rails background where all of those early decisions were made for me.

As I progressed I kept wondering why I had to make all of these decisions myself. I would second guess myself, considering a framework (but everyone kept telling me not to touch those), and generally just feeling like I wasn't being productive.

Reading tutorials didn't help much either. They either felt so simple and contrived that they ignored hundreds of potential issues and questions, or it felt like I was learning how to draw an owl.

In a way, this seems a bit ridiculous. As I stated before, frameworks like Ruby on Rails take care of all of these early decisions for you, so why can't we settle on some form of web application structure in Go? Why are we instead constantly being told that it depends, or given vague advice that doesn't always feel actionable?

The goal of this article is it explore why exactly there isn't a single app structure that is recommended to new Gophers, and the overall goal of this series is help newcomers to Go understand what choices are available to them so they can get started a little easier.

Can't we just learn from other languages?

First, let's talk about languages like Ruby and Python. Why does it feel like they have this problem solved? Why can't we learn from them?

One of the biggest things working in favor of Ruby, Python, and many other languages is the adoption of frameworks in those languages. Ruby has Ruby on Rails. Python has Django (and a few others). In these languages, the actual programming language itself isn't suggesting any specific app structure for a web application. It is the framework that makes those suggestions, but in languages like Ruby it is so synonymous with "Rails" that people just assume this is how all Ruby web applications should be written.

If you were a Ruby developer and you really wanted to, there is nothing stopping you from using the standard library or something like Sinatra to build a web server without using Rails. Sure, it would take more setup than with Rails, but it would work and you could structure your web application however you wanted. At this point we are in the same position as we are with Go - there isn't a suggested app structure to follow and we need to figure one out on our own.

Most Ruby developers will suggest that you just use Rails; similarly, most Python developers will suggest a framework like Django. So why are Go developers suggesting you use the standard library rather than a framework?

The values of your programming language matter

Every language has a set of values. They might not always be explicitly stated, but they exist and will shape both the language and code that is written in it.

Understanding how a language's values can affect code written in it is easiest to grasp when comparing two languages with very different values. Since I know more about Ruby than most other languages, I will compare Ruby and Go.

Ruby is a dynamic language where development speed and flexibility are given a fairly high priority. Go, on the other hand, tends to prioritize clarity, readability, and explicitness over development speed. What does this look like in practice?

One example that sticks out to me is what type of reflection and metaprogamming is allowed in each language. In Go the reflect package is fairly limited, and developers are encouraged to avoid it. "Reflection is never clear" is even one of the Go Proverbs. As a result, metaprogramming isn't really possible in Go outside of code generation.

Sidenote: Metaprogramming is basically act of writing code that will analyze other code and generate new code. In languages like Go this might be akin to code generation, but in languages like Ruby you can dynamically define methods on types and do a wild array of other things **while your code is running 🤯* which makes it both powerful and insanely complicated to read and understand.*

Ruby has a slightly different take on reflection and metaprogramming. In fact, if you take a tutorial that teaches you Ruby, it is very likely that you will learn how to add new methods to the string type within the first few hours of that tutorial!

In Ruby you can even create a catch-all method (method_missing) that will be triggered anytime you try to call a method that isn't defined and then you can dynamically create that method at runtime if you want. In Go this simply isn't possible, and the reflect library is unlikely to ever allow it as it can lead to incredibly confusing code, so why is this done in Ruby?

Development speed!

Frameworks like Rails use metaprogramming to quickly create a new "language" within ruby that is flexible enough to work with a variety of different use cases, while not needing to be compiled or generated on a case-by-case basis. For instance, if you had an articles table in your Rails application with blog posts like this one, Rails makes it very easy to generate methods that create SQL queries for you:

Article.published.with_author.order(:published_at)

This particular query would probably end up running something like:

SELECT * FROM articles WHERE (published_at != null) AND (author != null) ORDER BY published_at DESC

Metaprogramming can be a little scary sounding at first, but once you get the hang of it, it is actually pretty cool and allows you to do some crazy things very quickly. For example, I once created a ruby admin framework that would dynamically scan all of your Rails database models and relationships and create a complete admin dashboard with links to joined resources, editing ability, and more. The craziest part was all you had to do to add these features to an application was install the library and add a single line of code to your application:

mount Upmin::Engine => '/admin'

That's it. After that it would dynamically look at your Rails code when your application started up create dynamic methods, instantiate all the pages, declare routes, etc. Talk about magic!

Absolutely none of this will ever exist in Go. I can't imagine a PR ever being accepted that allows developers to do anything this crazy in Go. You might be able to use code generation and some SQL database analysis to generate code in a similar way (SQLBoiler does something like this to generate an ORM), but in practice this is very different from metaprogramming in Ruby. For instance, you can actually read generated code and understand what it is doing, meanwhile in Ruby you might not be able to even find where a method gets defined in the source code because it is buried in a metaprogramming jumble.

Metaprogramming allows for fast development, but it does so at the cost of clarity, readability, and explicitness, all of which are core values of Go.

Frameworks aren't always as extreme as this metaprogramming example, but at their core frameworks tend to be about giving up explicitness in exchange for faster development speed. That is, developers using a framework agree to adhere to some restrictions imposed by the framework in exchange for having the framework do some of the work for them. This in turn allows them to create applications faster, but it isn't always as clear how things are happening. It can also lead to giving up control in some situations, as most frameworks don't allow you to customize everything.

Frameworks aren't commonly suggested in Go because these tradeoffs are at odds with Go's core values. I'm not saying there isn't a place for frameworks in Go - there is - but I don't ever see frameworks gaining the adoption you see in languages like Python or Ruby because most Go developers choose Go not for its development speed, but because they are willing to spend extra time writing coding if it means their code is clearer and more explicit. This is exactly why the try proposal was ultimately rejected.

Context is king

Rather than focusing on a single application structure, you are likely to have far more success in Go if you focus on understanding the context of your application.

Context can refer to a wide variety of factors; the size of your team, your experience as a programmer, your deployment environment, etc. Basically anything that might make your application structure decisions different from project to project is part of your context.

There is a great example of this in Go Time episode #94 where we discuss Structuring your Go apps.

Go Time 94: Structuring your Go apps – Listen on Changelog.com

In the episode the panel is talking about how we each start a new Go application, and Johnny Boursiquot and I gave different answers. I said I normally start out with just a main.go and see where it goes, meanwhile Johnny said he almost always starts out with an folder structure similar to this:

appName/
  cmd/
    appName/
        main.go

And then Johnny will try to keep that main.go very thin - it just does initialization, configuration, and calls out to other go packages that contain all of the real application logic.

So why are Johnny and I starting out so differently? Am I doing it wrong?

Let's take a look at our contexts; when I start a new project, I very frequently am creating what I'd call a throwaway program. A quick example to help a student having trouble with a concept. A silly program to press the spacebar in random intervals for a friend who is upset with WoW classic queues 😂. Or even a simple little program that helps my wife clean up duplicate files imported from her camera. The point is, most of these programs are going to be less than a few hundred lines of code and will be a single .go file. Starting off with anything more than that is probably overkill, so in my personal context, this makes a lot of sense.

On the other hand, I suspect that Johnny is frequently building applications in the context of his work at Heroku; other team members are likely to get involved in the project, having consistent app structure across all their apps has real value, and the way he starts his projects makes complete sense.

This is just one example using a really simple aspect, but there are countless others.

• Are you a big company that values correctness over moving fast and breaking things? Chances are you will focus on testability a lot more than a startup with your app structure.
• Does your team mind testing with a real (not prod) database, or do you prefer more unit testing? This can affect what app structures work out for you.
• Are you a relatively new developer, or an seasoned veteran? This would have a drastic effect on what structure you should start with.
• Are you deploying with a variety of microservices, or as a single monolithic app? Yep, that changes your app structure too.

This list could literally go on forever, as there are an infinite number of factors that could differentiate applications. And therein lies the problem - there isn't a one size fits all structure because developers all have different needs.

Sidenote: Not shockingly, Peter Bourgon makes a point to bring up context in Go Time #102 - On application design and brings up a few of these example contexts. If you haven't already, I recommend checking that episode out as well.

In summary...

In summary, we can't settle on a single application structure in Go because there are just too many contexts to consider, BUT that doesn't mean we can't give new Gophers better advice. And that is the goal of this series.

I want to spend the next few articles in this series exploring some of the app structures you might use in your next project. I'll try to present them in the order I tend to see developers progress on their own, as this also tends to map directly to their complexity. The current list of structures are:

• Flat structure
• Model, view, controller (MVC)
• Domain driven design (or something resembling it)

Along with exploring how each potential app structure looks, we will also talk about the pros and cons of each. When we get to MVC we will also spend some time discussing mistakes that can make it feel like MVC won't work in Go (eg cyclical dependency mistakes).

As an added bonus, I hope to also dive into things like globals that don't exactly alter how you organize code, but do alter the overall structure of your code in many ways. Globals will still pop up in some of the other articles in the series, but we will really focus on them at the end of this series.

By the end of this series I hope you will have a good grasp on a few app structures you can try out as you venture into building web applications with Go.

Want to learn Go?

Interested in learning or practicing Go? Check out my FREE course - Gophercises - Programming Exercises for Budding Gophers.

I also have some premium courses that cover Web Dev with Go and Testing with Go that you can check out as well.