How to securely store passwords?

Hello everyone, welcome back to the backend master class!

In this lecture, we’re gonna learn how to securely store users’ password in the database.

Here's:

• Link to the full series playlist on Youtube
• And its Github repository

How to store password

As you already know, we should never ever store naked passwords! So the idea is to hash it first, and only store that hash value.

Basically, the password will be hashed using brypt hashing function to produce a hash value.

Besides the input password, bcrypt requires a cost parameter, which will decide the number of key expansion rounds or iterations of the algorithm.

Bcrypt also generates a random salt to be used in those iterations, which will help protect against the rainbow table attack. Because of this random salt, the algorithm will give you a completely different output hash value even if the same input password is provided.

The cost and salt will also be added to the hash to produce the final hash string, which looks something like this:

In this hash string, there are 4 components:

• The first part is the hash algorithm identifier. 2A is the identifier of the bcrypt algorithm.
• The second part is the cost. In this case, the cost is 10, which means there will be 2^10 = 1024 rounds of key expansion.
• The third part is the salt of length 16 bytes, or 128 bits. It is encoded using base64 format, which will generate a string of 22 characters.
• Finally, the last part is the 24 bytes hash value, encoded as 31 characters.

All of these 4 parts are concatenated together into a single hash string, and it is the string that we will store in the database.

So that’s the process of hashing users’ password!

But when users login, how can we verify that the password that they entered is correct or not?

Well, first we have to find the hashed_password stored in the DB by username.

Then we use the cost and salt of that hashed_password as the arguments to hash the naked_password users just entered with bcrypt. The output of this will be another hash value.

Then all we have to do is to compare the 2 hash values. If they’re the same, then the password is correct.

Alright, now let’s see how to implement these logics in Golang.

Implement functions to hash and compare passwords

In the previous lecture, we have generated the code to create a new user in the database. And hashed_password is one of the input parameters of the CreateUser() function.

type CreateUserParams struct {
    Username       string `json:"username"`
    HashedPassword string `json:"hashed_password"`
    FullName       string `json:"full_name"`
    Email          string `json:"email"`
}

func (q *Queries) CreateUser(ctx context.Context, arg CreateUserParams) (User, error) {
    row := q.db.QueryRowContext(ctx, createUser,
        arg.Username,
        arg.HashedPassword,
        arg.FullName,
        arg.Email,
    )
    var i User
    err := row.Scan(
        &i.Username,
        &i.HashedPassword,
        &i.FullName,
        &i.Email,
        &i.PasswordChangedAt,
        &i.CreatedAt,
    )
    return i, err
}

Also, in this createRandomUser() function of the unit test in db/sqlc/user_test.go, we’re using a simple "secret" string for the hash_password field, which doesn’t reflect the real correct values this field should hold.

func createRandomUser(t *testing.T) User {
    arg := CreateUserParams{
        Username:       util.RandomOwner(),
        HashedPassword: "secret",
        FullName:       util.RandomOwner(),
        Email:          util.RandomEmail(),
    }

    ...
}

So today we’re gonna update it to use a real hash string.

Hash password function

First, let’s create a new file password.go inside the util package. In this file, I’m gonna define a new function: HashPassword().

It will take a password string as input, and will return a string or an error. This function will compute the bcrypt hash string of the input password.

// HashPassword returns the bcrypt hash of the password
func HashPassword(password string) (string, error) {
    hashedPassword, err := bcrypt.GenerateFromPassword([]byte(password), bcrypt.DefaultCost)
    if err != nil {
        return "", fmt.Errorf("failed to hash password: %w", err)
    }
    return string(hashedPassword), nil
}

In this function, we call bcrypt.GenerateFromPassword(). It requires 2 input parameters: the password of type []byte slice, and a cost of type int.

So we have to convert the input password from string to []byte slice.

For cost, I use the bcrypt.DefaultCost value, which is 10.

The output of this function will be the hashedPassword and an error. If the error is not nil, then we just return an empty hashed string, and wrap the error with a message saying: "failed to hash password".

Otherwise, we convert the hashedPassword from []byte slice to string, and return it with a nil error.

Compare passwords function

Next, we will write another function to check if a password is correct or not: CheckPassword().

This function will take 2 input arguments: a password to check, and the hashedPassword to compare. It will return an error as output.

Basically, this function will check if the input password is correct when comparing to the provided hashedPassword or not.

As the standard bcrypt package has already implemented this feature, all we have to do is to call bcrypt.CompareHashAndPassword() function, and pass in the hashedPassword and naked password, after converting them from string to []byte slices.

// CheckPassword checks if the provided password is correct or not
func CheckPassword(password string, hashedPassword string) error {
    return bcrypt.CompareHashAndPassword([]byte(hashedPassword), []byte(password))
}

And that’s it. We’re done!

Write unit test for HashPassword and CheckPassword functions

Now let’s write some unit tests to make sure these 2 functions work as expected.

I’m gonna create a new file password_test.go inside the util package. Then let’s define function TestPassword() with a testing.T object as input.

First I will generate a new password as a random string of 6 characters. Then we get the hashedPassword by calling HashPassword() function with the generated password.

We require no errors to be returned, and the hashedPassword string should be not empty.

func TestPassword(t *testing.T) {
    password := RandomString(6)

    hashedPassword, err := HashPassword(password)
    require.NoError(t, err)
    require.NotEmpty(t, hashedPassword)

    err = CheckPassword(password, hashedPassword1)
    require.NoError(t, err)
}

Next we call CheckPassword() function with the password and hashedPassword parameters.

As this is the same password we used to create the hashedPassword, this function should return no errors, which means correct password.

Let’s also test the case where an incorrect password is provided!

I will generate a new random wrongPassword string, and call CheckPassword() again with this wrongPassword argument. This time, we expect an error to be returned, since the provided password is incorrect.

func TestPassword(t *testing.T) {
    password := RandomString(6)

    hashedPassword, err := HashPassword(password)
    require.NoError(t, err)
    require.NotEmpty(t, hashedPassword)

    wrongPassword := RandomString(6)
    err = CheckPassword(wrongPassword, hashedPassword)
    require.EqualError(t, err, bcrypt.ErrMismatchedHashAndPassword.Error())
}

To be exact, we use require.EqualError() to compare the output error. It must be equal to the bcrypt.ErrMismatchedHashAndPassword error.

OK, the test is now completed. Let’s run it!

It passed! Awesome!

Update the existing code to use HashPassword function

So the HashPassword() function is working properly. Let’s go back to the user_test.go file and use it in the createRandomUser() function.

Here I’m gonna create a new hashedPassword value by calling util.HashPassword() function with a random string of 6 characters.

We require no errors, then change the "secret" constant to hashedPassword instead:

func createRandomUser(t *testing.T) User {
    hashedPassword, err := util.HashPassword(util.RandomString(6))
    require.NoError(t, err)

    arg := CreateUserParams{
        Username:       util.RandomOwner(),
        HashedPassword: hashedPassword,
        FullName:       util.RandomOwner(),
        Email:          util.RandomEmail(),
    }

    ...
}

Alright, let’s run the whole db package test!

All passed!

Now if we open the database in Table Plus and check the users table, we can see that the hashed_password column is now containing the correct bcrypt hashed string.

It looks just like the example that I shown you in the beginning of this video.

Make sure all hashed passwords are different

One thing we want to make sure of is: if the same password is hashed twice, 2 different hash values should be produced.

So let’s go back to the TestPassword() function. I’m gonna change the hashPassword variable’s name to hashedPassword1.

Then let’s duplicate the hash password code block, and change the variable’s name to hashedPassword2.

func TestPassword(t *testing.T) {
    password := RandomString(6)

    hashedPassword1, err := HashPassword(password)
    require.NoError(t, err)
    require.NotEmpty(t, hashedPassword1)

    err = CheckPassword(password, hashedPassword1)
    require.NoError(t, err)

    wrongPassword := RandomString(6)
    err = CheckPassword(wrongPassword, hashedPassword1)
    require.EqualError(t, err, bcrypt.ErrMismatchedHashAndPassword.Error())

    hashedPassword2, err := HashPassword(password)
    require.NoError(t, err)
    require.NotEmpty(t, hashedPassword2)
    require.NotEqual(t, hashedPassword1, hashedPassword2)
}

What we expect to see is: the value of hashedPassword2 should be different from the value of hashedPassword1. So here I use require.NotEqual() to check this condition.

OK, let’s rerun the test.

It passed! Excellent!

To really understand why it passed, we have to open the implementation of the bcrypt.GenerateFromPassword() function.

func GenerateFromPassword(password []byte, cost int) ([]byte, error) {
    p, err := newFromPassword(password, cost)
    if err != nil {
        return nil, err
    }
    return p.Hash(), nil
}

func newFromPassword(password []byte, cost int) (*hashed, error) {
    if cost < MinCost {
        cost = DefaultCost
    }
    p := new(hashed)
    p.major = majorVersion
    p.minor = minorVersion

    err := checkCost(cost)
    if err != nil {
        return nil, err
    }
    p.cost = cost

    unencodedSalt := make([]byte, maxSaltSize)
    _, err = io.ReadFull(rand.Reader, unencodedSalt)
    if err != nil {
        return nil, err
    }

    p.salt = base64Encode(unencodedSalt)
    hash, err := bcrypt(password, p.cost, p.salt)
    if err != nil {
        return nil, err
    }
    p.hash = hash
    return p, err
}

As you can see here, in the newFromPassword() function, a random salt value is generated, and it is used in the bcrypt() function to generate the hash.

So now you know, because of this random salt, the generated hash value will be different everytime.

Implement the create user API

Next step, I’m gonna use the HashPassword() function that we’ve written to implement the create user API for our simple bank.

Let’s create a new file user.go inside the api package.

This API will be very much alike the create account API that we’ve implemented before, so I’m just gonna copy it from the api/account.go file.

Then let’s change this struct to createUserRequest.

The first parameter is username. It is a required field.

And let’s say we don’t allow it to contain any kind of special characters, so here I’m gonna use the alphanum tag, which is already provided by the validator package. It basically means that this field should contain ASCII alphanumeric characters only.

The second field is password. It is also required. And normally we don’t want the password to be too short because it would be very easy to hack. So here let’s use the min tag to say that the length of the password should be at least 6 characters.

type createUserRequest struct {
    Username string `json:"username" binding:"required,alphanum"`
    Password string `json:"password" binding:"required,min=6"`
    FullName string `json:"full_name" binding:"required"`
    Email    string `json:"email" binding:"required,email"`
}

The third field is full_name of the user. There’s no specific requirements for this field, except that it is required.

Then, the last field is email, which is very important because it would be the main communication channel between the users and our system. We can use the email tag provided by validator package to make sure that the value of this field is a correct email address.

There are many other useful built-in tags that were already implemented by the validator package, you can check them out in its documentation or github page.

Now let’s go back to the code to complete this createUser() function.

func (server *Server) createUser(ctx *gin.Context) {
    var req createUserRequest
    if err := ctx.ShouldBindJSON(&req); err != nil {
        ctx.JSON(http.StatusBadRequest, errorResponse(err))
        return
    }

    hashedPassword, err := util.HashPassword(req.Password)
    if err != nil {
        ctx.JSON(http.StatusInternalServerError, errorResponse(err))
        return
    }

    arg := db.CreateUserParams{
        Username:       req.Username,
        HashedPassword: hashedPassword,
        FullName:       req.FullName,
        Email:          req.Email,
    }

    ...   
}

Here we use the ctx.ShouldBindJSON() function to bind the input parameters from the context into the createUserRequest object.

If any of the parameters are invalid, we just return 400 Bad Request status to the client. Otherwise, we will use them build the db.CreateUserParams object.

There are 4 fields that need to be set: Username, HashedPassword, Fullname, and Email.

So first, we compute the hashedPassword by calling util.HashPassword() function and pass in the input request.Password value.

If this function returns a not nil error, then we just return a status 500 Internal Server Error to the client.

Else, we will build the CreateUserParams object, where Username is request.Username, HashedPassword is the computed hashedPassword, FullName is request.FullName, and Email is request.Email.

func (server *Server) createUser(ctx *gin.Context) {
    ...

    user, err := server.store.CreateUser(ctx, arg)
    if err != nil {
        if pqErr, ok := err.(*pq.Error); ok {
            switch pqErr.Code.Name() {
            case "unique_violation":
                ctx.JSON(http.StatusForbidden, errorResponse(err))
                return
            }
        }
        ctx.JSON(http.StatusInternalServerError, errorResponse(err))
        return
    }

    ctx.JSON(http.StatusOK, user)
}

Then we call server.store.CreateUser() with this input argument. It will return the created user object or an error.

Just like in the create account API, if error is not nil, then there are some possible scenarios. Keep in mind that, in the users table, we have 2 unique constraints:

• One is for the primary key username,
• And the other is for the email column.

We don’t have a foreign key in this table, so here we only need to keep the unique_violation code name to return status 403 Forbidden in case an user with the same username or email already exists.

Finally, if no errors occur, we just return status 200 OK with the created user to the client.

OK, so now the createUser API handler is completed. The last step we must do is to register a route for it in the api/server.go file.

Here, in this NewServer() function, I’m gonna add a new route with method POST. Its path should be /users, and its handler function is server.createUser

// NewServer creates a new HTTP server and set up routing.
func NewServer(store db.Store) *Server {
    server := &Server{store: store}
    router := gin.Default()

    if v, ok := binding.Validator.Engine().(*validator.Validate); ok {
        v.RegisterValidation("currency", validCurrency)
    }

    router.POST("/users", server.createUser)

    router.POST("/accounts", server.createAccount)
    router.GET("/accounts/:id", server.getAccount)
    router.GET("/accounts", server.listAccounts)

    router.POST("/transfers", server.createTransfer)

    server.router = router
    return server
}

And that’s it! We’re done!

Test the create user API

Let’s open the terminal and run make server to start the server.

I’m gonna use Postman to test the new API.

Let’s select method POST and fill in the URL: http://localhost:8080/users

For the request body, let’s choose raw, and select JSON format. I'm gonna use this JSON data:

{
    "username": "quang1",
    "full_name": "Quang Pham",
    "email": "quang@email.com",
    "password": "secret"
}

OK, let’s send this request!

It’s successful! We’ve got the created user object here with all correct field values.

Let’s open the database to find this user.

Here it is! So our API is working well in the happy case.

Now let’s see what happens if I send this same request the second time.

We’ve got a 403 Forbidden status code. And the reason is that the unique constraint for username is violated.

We’re trying to create another user with the same username, So clearly it should not be allowed!

Now let’s try changing the username to quang2, but keep the email value the same, and send the request again.

We still got 403 Forbidden. But this time, the error is because the email unique constraint is violated. Exactly what we expected!

If I change the email to quang2@email.com, then the request will be successful, since this email doesn’t belong to any other users.

OK, now let’s try an invalid username, such as quang#2:

This time, the status code is 400 Bad Request. And the reason is: the field validation for username failed on the alphanum tag. There’s a special character # in the username, which is not alphanumeric.

Next, let’s try an invalid email. I’m gonna change the username to quang3, and email to quang3email.com, without the @ character.

We’ve got 400 Bad Request status again. And the error is: field validation for email failed on the email tag, which is exactly what we want.

OK now let’s fix the email address, and change the password to a very short value, such as "123". Then send the request one more time.

This time, we’ve got an error because the password field validation failed on the min tag. It doesn’t satisfy the minimum length constraint of 6 characters.

API should not expose hashed password

Before we finish, there’s one more thing I want to tell you. Let’s fix the password value and send the request again.

Now it’s successful. But you can notice that the hashed_password value is also returned, which doesn’t seem right, because the client will never need to use this value for anything.

And it might raise some security concerns, as this piece of sensitive information is being transmitted in the public.

It would be better to remove this field from the response body.

To do that, I’m gonna declare a new createUserResponse struct in the api/user.go file. It will contain almost all fields of the db.User struct, except for the HashedPassword field that should be removed.

type createUserResponse struct {
    Username          string    `json:"username"`
    FullName          string    `json:"full_name"`
    Email             string    `json:"email"`
    PasswordChangedAt time.Time `json:"password_changed_at"`
    CreatedAt         time.Time `json:"created_at"`
}

Then here, at the end of the createUser() handler function, we build a new createUserResponse object, where Username is user.Username, FullName is user.FullName, Email is user.Email, PasswordChangedAt is user.PasswordChangedAt, and CreatedAt is user.CreatedAt.

func (server *Server) createUser(ctx *gin.Context) {
    ...

    user, err := server.store.CreateUser(ctx, arg)
    if err != nil {
        if pqErr, ok := err.(*pq.Error); ok {
            switch pqErr.Code.Name() {
            case "unique_violation":
                ctx.JSON(http.StatusForbidden, errorResponse(err))
                return
            }
        }
        ctx.JSON(http.StatusInternalServerError, errorResponse(err))
        return
    }

    rsp := createUserResponse{
        Username:          user.Username,
        FullName:          user.FullName,
        Email:             user.Email,
        PasswordChangedAt: user.PasswordChangedAt,
        CreatedAt:         user.CreatedAt,
    }
    ctx.JSON(http.StatusOK, rsp)
}

Finally, we return the response object instead of user. And we’re done!

Let’s restart the server. Then go back to Postman, update the username and email to new values, and send the request.

It’s successful. And now there’s no hashed_password field in the response body anymore. Perfect!

So that brings us to the end of this lecture. I hope you have learned something useful.

Thank you for reading, and see you in the next one!

If you like the article, please subscribe to our Youtube channel and follow us on Twitter or Facebook for more tutorials in the future.

If you want to join me on my current amazing team at Voodoo, check out our job openings here. Remote or onsite in Paris/Amsterdam/London/Berlin/Barcelona with visa sponsorship.