Load config from file & environment variables in Golang with Viper

When developing and deploying a backend web application, we usually have to use different configurations for different environments such as development, testing, staging, and production.

Today we will learn how to use Viper to load configurations from file or environment variables.

Here's:

• Link to the full series playlist on Youtube
• And its Github repository

Why file and environment variables

Reading values from file allows us to easily specify default configuration for local development and testing.

While reading values from environment variables will help us override the default settings when deploying our application to staging or production using docker containers.

Why Viper

Viper is a very popular Golang package for this purpose.

• It can find, load, and unmarshal values from a config file.
• It supports many types of files, such as JSON, TOML, YAML, ENV, or INI.
• It can also read values from environment variables or command-line flags.
• It gives us the ability to set or override default values.
• Moreover, if you prefer to store your settings in a remote system such as Etcd or Consul, then you can use viper to read data from them directly.
• It works for both unencrypted and encrypted values.
• One more interesting thing about Viper is, it can watch for changes in the config file, and notify the application about it.
• We can also use viper to save any config modification we made to the file.

A lot of useful features, right?

What will we do

In the current code of our simple bank project, we’re hard-coding some constants for the dbDriver, dbSource in the main_test.go file, and also one more constant for the serverAddress in the main.go file.

const (
    dbDriver      = "postgres"
    dbSource      = "postgresql://root:secret@localhost:5432/simple_bank?sslmode=disable"
    serverAddress = "0.0.0.0:8080"
)

So in this tutorial, we will learn how to use Viper to read these configurations from file and environment variables.

Install Viper

OK, let’s start by installing Viper! Open your browser and search for golang viper.

Then open its Github page. Scroll down a bit, copy this go get command and run it in the terminal to install the package:

❯ go get github.com/spf13/viper

After this, in the go.mod file of our project, we can see Viper has been added as a dependency.

Create config file

Now I’m gonna create a new file app.env to store our config values for development. Then let’s copy these variables from main.go file and paste them to this config file. Since we’re using dot env format, we must change the way we declare these variables. It should be similar to how we declare environment variables:

• Each variable should be declared on a separate line.
• The variable's name is uppercase and its words are separated by an underscore.
• The variable value is followed by the name after an equal symbol.

DB_DRIVER=postgres
DB_SOURCE=postgresql://root:secret@localhost:5432/simple_bank?sslmode=disable
SERVER_ADDRESS=0.0.0.0:8080

And that’s it! This app.env file is now containing the default configurations for our local development environment. Next, we will use Viper to load this config file.

Load config file

Let’s create a new file config.go inside the util package.

Then declare a new type Config struct in this file. This Config struct will hold all configuration variables of the application that we read from file or environment variables. For now, we only have 3 variables:

• First, the DBDriver of type string,
• Second, the DBSource, also of type string.
• And finally the ServerAddress of type string as well.

type Config struct {
    DBDriver      string `mapstructure:"DB_DRIVER"`
    DBSource      string `mapstructure:"DB_SOURCE"`
    ServerAddress string `mapstructure:"SERVER_ADDRESS"`
}

In order to get the value of the variables and store them in this struct, we need to use the unmarshaling feature of Viper. Viper uses the mapstructure package under the hood for unmarshaling values, so we use the mapstructure tags to specify the name of each config field.

In our case, we must use the exact name of each variable as being declared in the app.env. For example, the DBDriver’s tag name should be DB_DRIVER, the DBSource’s tag name should be DB_SOURCE, and similar for the ServerAddress, should be SERVER_ADDRESS.

OK, next I’m gonna define a new function LoadConfig(), which takes a path as input, and returns a config object or an error. This function will read configurations from a config file inside the path if it exists, or override their values with environment variables if they’re provided.

func LoadConfig(path string) (config Config, err error) {
    viper.AddConfigPath(path)
    viper.SetConfigName("app")
    viper.SetConfigType("env")

    ...
}

First, we call viper.AddConfigPath() to tell Viper the location of the config file. In this case, the location is given by the input path argument.

Next, we call viper.SetConfigName() to tell Viper to look for a config file with a specific name. Our config file is app.env, so its name is app.

We also tell Viper the type of the config file, which is env in this case, by calling viper.SetConfigFile() and pass in env. You can also use JSON, XML or any other format here if you want, just make sure your config file has the correct format and extension.

Now, besides reading configurations from file, we also want viper to read values from environment variables. So we call viper.AutomaticEnv() to tell viper to automatically override values that it has read from config file with the values of the corresponding environment variables if they exist.

// LoadConfig reads configuration from file or environment variables.
func LoadConfig(path string) (config Config, err error) {
    viper.AddConfigPath(path)
    viper.SetConfigName("app")
    viper.SetConfigType("env")

    viper.AutomaticEnv()

    err = viper.ReadInConfig()
    if err != nil {
        return
    }

    err = viper.Unmarshal(&config)
    return
}

After that, we call viper.ReadInConfig() to start reading config values. If error is not nil, then we simply return it.

Otherwise, we call viper.Unmarshal() to unmarshals the values into the target config object. And finally just return the config object and any error if it occurs.

So basically the load config function is completed. Now we can use it in the main.go file.

Use LoadConfig in the main function

Let’s remove all of the previous hardcoded values. Then in the main() function, let’s call util.LoadConfig() and pass in "." here, which means the current folder because our config file app.env is in the same location as this main.go file.

If there’s an error, then we just write a fatal log saying cannot load configuration. Else, we just change these variables to config.DBDriver, config.DBSource, and config.ServerAdress.

func main() {
    config, err := util.LoadConfig(".")
    if err != nil {
        log.Fatal("cannot load config:", err)
    }

    conn, err := sql.Open(config.DBDriver, config.DBSource)
    if err != nil {
        log.Fatal("cannot connect to db:", err)
    }

    store := db.NewStore(conn)
    server := api.NewServer(store)

    err = server.Start(config.ServerAddress)
    if err != nil {
        log.Fatal("cannot start server:", err)
    }
}

And we’re done! Let’s try to run this server.

❯ make server
go run main.go
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:   export GIN_MODE=release
 - using code:  gin.SetMode(gin.ReleaseMode)

[GIN-debug] POST   /accounts                 --> github.com/techschool/simplebank/api.(*Server).createAccount-fm (3 handlers)
[GIN-debug] GET    /accounts/:id             --> github.com/techschool/simplebank/api.(*Server).getAccount-fm (3 handlers)
[GIN-debug] GET    /accounts                 --> github.com/techschool/simplebank/api.(*Server).listAccount-fm (3 handlers)
[GIN-debug] Listening and serving HTTP on 0.0.0.0:8080

It’s working! The server is listening on localhost port 8080, just like what we specified in the app.env file.

Let’s open Postman and send an API request. I’m gonna call the list accounts API.

Oops, we’ve got a 500 Internal Server Error. It’s because our web server cannot connect to Postgres database on port 5432. Let’s open the terminal and check if Postgres is running or not:

❯ docker ps

There are no containers running. If we run docker ps -a, we can see that Postgres container has been exited. So we have to start it by running:

❯ docker start postgres12

OK, now the database is up and running. Let’s go back to Postman and send the request again.

This time, it’s successful. We’ve got a list of accounts here. So our code to load configurations from file is working well.

Let’s try overriding those configurations with environment variables. I will set the SERVER_ADDRESS variable to localhost port 8081 before calling make server.

❯ SERVER_ADDRESS=0.0.0.0:8081 make server
go run main.go
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:   export GIN_MODE=release
 - using code:  gin.SetMode(gin.ReleaseMode)

[GIN-debug] POST   /accounts                 --> github.com/techschool/simplebank/api.(*Server).createAccount-fm (3 handlers)
[GIN-debug] GET    /accounts/:id             --> github.com/techschool/simplebank/api.(*Server).getAccount-fm (3 handlers)
[GIN-debug] GET    /accounts                 --> github.com/techschool/simplebank/api.(*Server).listAccount-fm (3 handlers)
[GIN-debug] Listening and serving HTTP on 0.0.0.0:8081

Here we can see that the server is now listening on port 8081 instead of 8080 as before. Now if we try to send the same API request in Postman to port 8080, we will get a connection refused error:

Only when we change this port to 8081, then the request will be successful:

So we can conclude that Viper has successfully overridden the values it read from the config file with environment variables. That’s very convenient when we want to deploy the application to different environments such as staging or production in the future.

Use LoadConfig in the test

Now before we finish, let’s update the main_test.go file to use the new LoadConfig() function.

First, remove all of the hard-coded constants. Then in the TestMain() function, we call util.LoadConfig().

But this time, the main_test.go file is inside the db/sqlc folder, while the app.env config file is at the root of the repository, so we must pass in this relative path: "../..". These 2 dots .. basically mean go to the parent folder.

func TestMain(m *testing.M) {
    config, err := util.LoadConfig("../..")
    if err != nil {
        log.Fatal("cannot load config:", err)
    }

    testDB, err = sql.Open(config.DBDriver, config.DBSource)
    if err != nil {
        log.Fatal("cannot connect to db:", err)
    }

    testQueries = New(testDB)

    os.Exit(m.Run())
}

If the error is not nil, then we just write a fatal log. Otherwise, we should change these 2 values to config.DBDriver and config.DBSource.

And that’s it! We’re done. Let’s run the whole package test.

All passed! And that wraps up this lecture about reading configuration from file and environment variables.

I highly recommend you to check out the documentation and try some other interesting features of viper such as live watching, or reading from remote system.

Happy coding and I will see you in the next lecture!

If you like the article, please subscribe to our Youtube channel and follow us on Twitter for more tutorials in the future.

If you want to join me on my current amazing team at Voodoo, check out our job openings here. Remote or onsite in Paris/Amsterdam/London/Berlin/Barcelona with visa sponsorship.