How we define and load configuration from files.

Background

When we write applications, we basically use configuration files, from various shells to nginx, etc., all have their own configuration files. Although this is not too difficult, the configuration items are relatively cumbersome, and parsing and verifying them can be troublesome. In this article, I will tell how we simplify the definition and parsing of configuration files.

Scenario

If we want to write a Restful API service, the configuration items will probably have the following contents.

• Host, the listening IP, if not filled, the default is 0.0.0.0
• Port, the port to listen to, required, can only be a number, greater than or equal to 80, less than 65535
• LogMode, logging mode, only file or console can be selected
• Verbose, see whether to output detailed logs, optional, default is false
• MaxConns, the maximum number of concurrent connections allowed, default 10000
• Timeout, timeout setting, default 3s
• CpuThreshold, sets the threshold for CPU usage to trigger load shedding, default 900, 1000m means 100%

Previously we used json for the configuration file, but there was a problem with json that we couldn't add comments, so we switched to yaml format.

Now let's see how easy it is to define and parse such a configuration file with go-zero ~

Defining the configuration

First, we need to define the above configuration requirements into a Go structure, as follows.

RestfulConf struct {
    Host string `json:",default=0.0.0.0"`
    Port int `json:",range=[80,65535)"`
    LogMode string `json:",options=[file,console]"`
    Verbose bool `json:",optional"`
    MaxConns int `json:",default=10000"`
    Timeout time.Duration `json:",default=3s"`
    CpuThreshold int64 `json:",default=900,range=[0:1000]"`
}

As you can see, we have certain definitions and restrictions for each configuration item, some of which are defined as follows.

• default, the default value is used if the configuration is not filled, you can see that 3s in it will be automatically resolved into time.Duration type
• optional, you can leave the item blank, if not, use the zero value of its type
• range, which restricts the number type to the given range
• options, which restricts the configured value to only one of the given values

And, some attributes can be used together, such as.

• default and range can be used together to both add a range restriction and provide a default value
• default and options can be used together to add option restrictions and provide a default value

Configuration files

Because we give a lot of default values when defining the configuration, as well as using optional to specify as optional, so our configuration file has relatively few configuration items that can use the default values without writing explicitly, as follows.

# Because many of them have default values, you only need to write the ones
# that need to be specified and the ones that don't have default values
Port: 8080
LogMode: console
# You can read the values of environment variables
MaxBytes: ${MAX_BYTES}

Here is a note, if the value of the configuration chars are all numbers, and you define the configuration type is string, for example, some people often use 123456 to test the password, but the password will generally be defined as string, the configuration should be written as follows (just an example, passwords are generally not recommended to be written in the configuration file)

Password: "123456"

Here the double quotes can not be missed, otherwise type mismatch will be reported, because yaml parser will parse 123456 as int.

Loading configuration files

We have the configuration definition (config.go) and the configuration file (config.yaml), the next step is to load the configuration file, which can be done in three ways.

• must be loaded successfully, otherwise the program exits, we generally use so, if the configuration is not correct, the program will not be able to continue

// If error, exit the program directly
var config RestfulConf
conf.MustLoad("config.yaml", &config)

The default code generated by go-zero's goctl tool also uses MustLoad to load the configuration file

• Load the configuration and handle the error on your own

// handle errors on your own
var config RestfulConf
// For simplicity, LoadConfig will be changed to Load later,
// LoadConfig has been marked as Deprecated.
if err := conf.LoadConfig("config.yaml", &config); err ! = nil {
    log.Fatal(err)
}

• Load the configuration and read the environment variables

// Automatically read environment variables
var config RestfulConf
conf.MustLoad(configFile, &config, conf.UseEnv())

Here why we need to explicitly specify conf.UseEnv(), because if default to use, you may need to escape when writing specific characters in the configuration, so the default does not read the environment variables.

Implementation principle

We usually use encoding/json or the corresponding yaml library when implementing yaml/json parsing, but for go-zero, we need to have more precise control over unmarshal, which leads us to customize yaml/json parsing ourselves, the complete code is here.

Configuration file code: https://github.com/zeromicro/go-zero/tree/master/core/conf

yaml/json parsing code: https://github.com/zeromicro/go-zero/tree/master/core/mapping

This is also a good example of how reflect can be used and how unit tests can be used to ensure correctness in complex scenarios.

Summary

I always recommend the idea of Fail Fast, we do the same on loading configuration files, once there is an error, immediately exit, so that ops will find the problem in time when deploying services, because the process can not get up running.

All the configuration items of go-zero services are loaded and automatically verified in this way, including the configuration of many of the tools I wrote are also based on this, hope it helps!

Project address

https://github.com/zeromicro/go-zero

Welcome to use go-zero and star to support us!