Introduction
Users of our application might make mistakes while filling out the registration form. Or they might change their names. To not bore our users with not-so-important details at the registration stage, we omitted some fields such as thumbnail, github_link, birth_date, and phone_number. We need to provide an interface for our users to update these fields. For picture uploads, we will use AWS S3. Also, as developers, it is important for us to quickly know how and when to increase or reduce our application's infrastructure. We will use Golang's expvar and httpsnoop to provide such an interface.
Source code
The source code for this series is hosted on GitHub via:
go-auth
This repository accompanies a series of tutorials on session-based authentication using Go at the backend and JavaScript (SvelteKit) on the front-end.
It is currently live here (the backend may be brought down soon).
To run locally, kindly follow the instructions in each subdirectory.
Implementation
Step 1: Setting up AWS S3 client
In order to use AWS S3 to handle our system's file uploads, we need to build some foundations. Using AWS in Golang, as in many other languages, requires installing its SDK. Parts of the SDK are general and required while others are need-based. Therefore, let's first install the core SDK and the config modules:
~/Documents/Projects/web/go-auth/go-auth-backend$ go get github.com/aws/aws-sdk-go-v2
~/Documents/Projects/web/go-auth/go-auth-backend$ go get github.com/aws/aws-sdk-go-v2/config
Then the AWS service API client for S3:
~/Documents/Projects/web/go-auth/go-auth-backend$ go get github.com/aws/aws-sdk-go-v2/service/s3
NOTE: Before you proceed, kindly Get your AWS access keys. It's needed for the next steps.
Now, let's add AWS configurations to our app's config type and API client to the application type. Then initialize the client:
// cmd/api/main.go
import (
...
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/credentials"
"github.com/aws/aws-sdk-go-v2/service/s3"
...
)
type config struct {
...
awsConfig struct {
AccessKeyID string
AccessKeySecret string
Region string
BucketName string
BaseURL string
s3_key_prefix string
}
}
type application struct {
...
S3Client *s3.Client
}
func main() {
...
sdkConfig := aws.Config{
Region: cfg.awsConfig.Region,
Credentials: credentials.NewStaticCredentialsProvider(
cfg.awsConfig.AccessKeyID, cfg.awsConfig.AccessKeySecret, "",
),
}
s3Client := s3.NewFromConfig(sdkConfig)
...
app := &application{
...
S3Client: s3Client,
}
...
}
The above additions added some details to the config type. Using them, we built an AWS SDK config instance which was later used to initialize an AWS S3 API client.
The application won't compile yet since we haven't really loaded these credentials in the config. We will do that in cmd/api/config.go:
// cmd/api/config.go
...
func updateConfigWithEnvVariables() (*config, error) {
...
// AWS configs
flag.StringVar(&cfg.awsConfig.AccessKeyID, "aws-access-key", os.Getenv("AWS_ACCESS_KEY_ID"), "AWS Access KeyID")
flag.StringVar(&cfg.awsConfig.AccessKeySecret, "aws-access-secret", os.Getenv("AWS_SECRET_ACCESS_KEY"), "AWS Access Secret")
flag.StringVar(&cfg.awsConfig.Region, "aws-region", os.Getenv("AWS_REGION"), "AWS region")
flag.StringVar(&cfg.awsConfig.BucketName, "aws-bucketname", os.Getenv("AWS_S3_BUCKET_NAME"), "AWS bucket name")
flag.Parse()
cfg.awsConfig.BaseURL = fmt.Sprintf(
"https://%s.s3.%s.amazonaws.com",
cfg.awsConfig.BucketName,
cfg.awsConfig.Region,
)
cfg.awsConfig.s3_key_prefix = "media/go-auth/"
...
}
The details were loaded from our app's .env file. Also, we are hard-coding s3_key_prefix in this case but you can make it dynamic if you wish.
Now, we can use these configurations to upload and delete files.
Step 2: Uploading and deleting files from AWS S3
Our design decision will be to have different endpoints to upload and delete files from S3. But before the endpoints, we'll abstract away the upload and delete logic. The logic will live in cmd/api/s3_utils.go:
// cmd/api/s3_utils.go
package main
import (
"context"
"crypto/rand"
"encoding/base32"
"fmt"
"net/http"
"strings"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/aws/aws-sdk-go-v2/service/s3/types"
)
func (app *application) uploadFileToS3(r *http.Request) (*string, error) {
file, handler, err := r.FormFile("thumbnail")
if err != nil {
app.logError(r, err)
return nil, err
}
defer file.Close()
b := make([]byte, 16)
_, err = rand.Read(b)
if err != nil {
app.logError(r, err)
return nil, err
}
// Encode bytes in base32 without the trailing ==
s := base32.StdEncoding.WithPadding(base32.NoPadding).EncodeToString(b)
fileName := fmt.Sprintf("%s_%s", s, handler.Filename)
key := fmt.Sprintf("%s%s", app.config.awsConfig.s3_key_prefix, fileName)
_, err = app.S3Client.PutObject(context.Background(), &s3.PutObjectInput{
Bucket: aws.String(app.config.awsConfig.BucketName),
Key: aws.String(key),
Body: file,
})
if err != nil {
app.logError(r, err)
return nil, err
}
s3_url := fmt.Sprintf("%s/%s", app.config.awsConfig.BaseURL, key)
return &s3_url, nil
}
func (app *application) deleteFileFromS3(r *http.Request) (bool, error) {
thumbnailURL := r.FormValue("thumbnail_url")
var objectIds []types.ObjectIdentifier
word := "media"
substrings := strings.Split(thumbnailURL, word)
key := fmt.Sprintf("%s%s", word, substrings[len(substrings)-1])
objectIds = append(objectIds, types.ObjectIdentifier{Key: aws.String(key)})
_, err := app.S3Client.DeleteObjects(context.Background(), &s3.DeleteObjectsInput{
Bucket: aws.String(app.config.awsConfig.BucketName),
Delete: &types.Delete{Objects: objectIds},
})
if err != nil {
app.logError(r, err)
return false, err
}
return true, nil
}
In uploadFileToS3, we used Go's request.FormFile() to retrieve the file coming from the request's FormData by name. This is a way to get uploaded files from .FormFile(). The method returns three items: the file, handler and err. The file holds the uploaded file itself while handler holds the file details such as name, size and so on. You can check this article for ways to handle FormData in Golang. Next, I don't want files with the same name to be overwritten so with each file, we prepend encoded randomly generated bytes to the filename. Therefore, the filenames have texts prepended to them. Then, we used AWS S3 API Client's PutObject to upload the file.
As for the deleteFileFromS3, we require that users supply their images' URLs. In our app, the URL will be automatically extracted from the users. Using the URL, we trimmed off its beginning until media is seen. For example, if an image URL is https://bucket_name.s3.origin.amazonaws.com/media/go-auth/name_of_image.png, after trimming, we will be left with media/go-auth/name_of_image.png. Natively, AWS S3 supports bulk deletion of objects, we supplied the shortened URL as the object's objectId and used DeleteObjects to delete it. That's it!
Now, the handlers will be dead simple as almost all the major things have been abstracted!
// cmd/api/upload_image_s3.go
package main
import (
"errors"
"net/http"
)
func (app *application) uploadFileToS3Handler(w http.ResponseWriter, r *http.Request) {
_, status, err := app.extractParamsFromSession(r)
if err != nil {
switch *status {
case http.StatusUnauthorized:
app.unauthorizedResponse(w, r, err)
case http.StatusBadRequest:
app.badRequestResponse(w, r, errors.New("invalid cookie"))
case http.StatusInternalServerError:
app.serverErrorResponse(w, r, err)
default:
app.serverErrorResponse(w, r, errors.New("something happened and we could not fullfil your request at the moment"))
}
return
}
s3URL, err := app.uploadFileToS3(r)
if err != nil {
app.badRequestResponse(w, r, err)
return
}
env := envelope{"s3_url": s3URL}
err = app.writeJSON(w, http.StatusOK, env, nil)
if err != nil {
app.serverErrorResponse(w, r, err)
}
app.logSuccess(r, http.StatusOK, "Image uploaded successfully")
}
The upload handler should be very familiar. We only allowed authenticated users to upload files and after a successful process, returned the URL of the uploaded file.
// cmd/api/delete_image_s3.go
package main
import (
"errors"
"net/http"
)
func (app *application) deleteFileOnS3Handler(w http.ResponseWriter, r *http.Request) {
_, status, err := app.extractParamsFromSession(r)
if err != nil {
switch *status {
case http.StatusUnauthorized:
app.unauthorizedResponse(w, r, err)
case http.StatusBadRequest:
app.badRequestResponse(w, r, errors.New("invalid cookie"))
case http.StatusInternalServerError:
app.serverErrorResponse(w, r, err)
default:
app.serverErrorResponse(w, r, errors.New("something happened and we could not fullfil your request at the moment"))
}
return
}
_, err = app.deleteFileFromS3(r)
if err != nil {
app.badRequestResponse(w, r, err)
return
}
app.successResponse(w, r, http.StatusNoContent, "Image deleted successfully.")
}
deleteFileOnS3Handler is almost the same aside from the fact that we didn't return the file's URL but a success message instead!
Step 3: User profile update
Now the handler that updates users' data:
// cmd/api/update_user.go
package main
import (
"errors"
"net/http"
"goauthbackend.johnowolabiidogun.dev/internal/data"
"goauthbackend.johnowolabiidogun.dev/internal/types"
"goauthbackend.johnowolabiidogun.dev/internal/validator"
)
func (app *application) updateUserHandler(w http.ResponseWriter, r *http.Request) {
userID, status, err := app.extractParamsFromSession(r)
if err != nil {
switch *status {
case http.StatusUnauthorized:
app.unauthorizedResponse(w, r, err)
case http.StatusBadRequest:
app.badRequestResponse(w, r, errors.New("invalid cookie"))
case http.StatusInternalServerError:
app.serverErrorResponse(w, r, err)
default:
app.serverErrorResponse(
w,
r,
errors.New("something happened and we could not fullfil your request at the moment"),
)
}
return
}
db_user, err := app.models.Users.Get(userID.Id)
if err != nil {
app.badRequestResponse(w, r, err)
return
}
var input struct {
FirstName *string `json:"first_name"`
LastName *string `json:"last_name"`
Thumbnail *string `json:"thumbnail"`
PhoneNumber *string `json:"phone_number"`
BirthDate types.NullTime `json:"birth_date"`
GithubLink *string `json:"github_link"`
}
err = app.readJSON(w, r, &input)
if err != nil {
app.badRequestResponse(w, r, err)
return
}
if input.FirstName != nil {
db_user.FirstName = *input.FirstName
}
if input.LastName != nil {
db_user.LastName = *input.LastName
}
if input.Thumbnail != nil {
db_user.Thumbnail = input.Thumbnail
}
if input.PhoneNumber != nil {
db_user.Profile.PhoneNumber = input.PhoneNumber
}
if input.BirthDate.Valid {
db_user.Profile.BirthDate = input.BirthDate
}
if input.GithubLink != nil {
db_user.Profile.GithubLink = input.GithubLink
}
v := validator.New()
if data.ValidateUser(v, db_user); !v.Valid() {
app.failedValidationResponse(w, r, v.Errors)
return
}
updated_user, err := app.models.Users.Update(db_user)
if err != nil {
app.serverErrorResponse(w, r, err)
return
}
err = app.writeJSON(w, http.StatusOK, updated_user, nil)
if err != nil {
app.serverErrorResponse(w, r, err)
}
app.logSuccess(r, http.StatusOK, "User updated successfully")
}
Since this handler will allow PATCH HTTP method, users are allowed to supply any field they want updated using the Update method on the UserModel:
// internal/data/user_queries.go
...
func (um UserModel) Update(user *User) (*User, error) {
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
var userOut User
var userPOut UserProfile
tx, err := um.DB.BeginTx(ctx, nil)
if err != nil {
return nil, err
}
query_user := `
UPDATE
users
SET
first_name = COALESCE($1, first_name),
last_name = COALESCE($2, last_name),
thumbnail = COALESCE($3, thumbnail)
WHERE
id = $4 AND is_active = true
RETURNING id, email, password, first_name, last_name, is_active, is_staff, is_superuser, thumbnail, date_joined
`
args_user := []interface{}{user.FirstName, user.LastName, user.Thumbnail, user.ID}
err = tx.QueryRowContext(ctx, query_user, args_user...).Scan(&userOut.ID,
&userOut.Email, &userOut.Password.hash, &userOut.FirstName, &userOut.LastName, &userOut.IsActive, &userOut.IsStaff, &userOut.IsSuperuser, &userOut.Thumbnail, &userOut.DateJoined)
if err != nil {
log.Printf("User: %v", err)
return nil, err
}
query_user_profile := `
UPDATE
user_profile
SET
phone_number = NULLIF($1, ''),
birth_date = $2::timestamp::date,
github_link = NULLIF($3, '')
WHERE
user_id = $4
RETURNING id, user_id, phone_number, birth_date, github_link
`
args_profile_user := []interface{}{
user.Profile.PhoneNumber,
user.Profile.BirthDate.Time,
user.Profile.GithubLink,
user.ID,
}
err = tx.QueryRowContext(ctx, query_user_profile, args_profile_user...).Scan(&userPOut.ID, &userPOut.UserID, &userPOut.PhoneNumber, &userPOut.BirthDate, &userPOut.GithubLink)
if err != nil {
log.Printf("Profile: %v", err)
return nil, err
}
if err = tx.Commit(); err != nil {
return nil, err
}
userOut.Profile = userPOut
return &userOut, nil
}
Though the method appears lengthy, it is easy to decipher considering each line's familiarity.
Let's now add these handlers to the routes:
// cmd/api/routes.go
...
func (app *application) routes() http.Handler {
...
router.HandlerFunc(http.MethodPatch, "/users/update-user/", app.updateUserHandler)
// Uploads
router.HandlerFunc(http.MethodPost, "/file/upload/", app.uploadFileToS3Handler)
router.HandlerFunc(http.MethodDelete, "/file/delete/", app.deleteFileOnS3Handler)
return app.recoverPanic(router)
}
Before we go, we need an endpoint to "instrument" our application.
Step 4: Getting the app's metrics
To get application's metrics, we'll use primarily expvar and, just for recording HTTP Status codes, httpsnoop. To start with, this endpoint should be heavily protected as hackers can take advantage of the data it exposes to attack, using a Denial of Service attack, our application. As a result of this, we will write a middleware that only allows superuser(s), who in most applications should be only one person, to access the endpoint:
// cmd/api/middleware.go
...
func (app *application) authenticateAndAuthorize(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
userID, status, err := app.extractParamsFromSession(r)
if err != nil {
switch *status {
case http.StatusUnauthorized:
app.unauthorizedResponse(w, r, err)
case http.StatusBadRequest:
app.badRequestResponse(w, r, errors.New("invalid cookie"))
case http.StatusInternalServerError:
app.serverErrorResponse(w, r, err)
default:
app.serverErrorResponse(
w,
r,
errors.New("something happened and we could not fullfil your request at the moment"),
)
}
return
}
// Get session from redis
_, err = app.getFromRedis(fmt.Sprintf("sessionid_%s", userID.Id))
if err != nil {
app.unauthorizedResponse(w, r, errors.New("you are not authorized to access this resource"))
return
}
db_user, err := app.models.Users.Get(userID.Id)
if err != nil {
app.badRequestResponse(w, r, err)
return
}
if !db_user.IsSuperuser {
app.unauthorizedResponse(w, r, errors.New("you are not authorized to access this resource"))
return
}
next.ServeHTTP(w, r)
})
}
To know more about middleware, kindly go through this article.
In the middleware, we only allowed authenticated users who have is_superuser set to true to access the endpoint.
Next, we will register the endpoint:
// cmd/api/routes.go
...
import (
"expvar"
...
)
func (app *application) routes() http.Handler {
...
// Metrics
router.Handler(http.MethodGet, "/metrics/", app.authenticateAndAuthorize(expvar.Handler()))
...
}
We simply wrapped the default metrics hander, expvar.Handler() with the newly created middleware. Since the default data exposed by this endpoint ain't enough, we will register more data such as database connection information in cmd/api/main.go:
// cmd/api/main.go
...
import (
...
"expvar"
...
)
func main() {
...
expvar.NewString("version").Set(version)
expvar.Publish("goroutines", expvar.Func(func() interface{} {
return runtime.NumGoroutine()
}))
expvar.Publish("database", expvar.Func(func() interface{} {
return db.Stats()
}))
expvar.Publish("timestamp", expvar.Func(func() interface{} {
return time.Now().Unix()
}))
...
}
The application's version, number of goroutines, database statistics, and current timestamp in Unix format were added. Next, we need to get requests and response metrics. A middleware will also help here. Using this opportunity, we can bring in httpsnoop just for HTTP status codes and how many times they were returned:
...
import (
...
"expvar"
...
"github.com/felixge/httpsnoop"
)
...
func (app *application) metrics(next http.Handler) http.Handler {
totalRequestsReceived := expvar.NewInt("total_requests_received")
totalResponsesSent := expvar.NewInt("total_responses_sent")
totalProcessingTimeMicroseconds := expvar.NewInt("total_processing_time_μs")
totalResponsesSentByStatus := expvar.NewMap("total_responses_sent_by_status")
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
totalRequestsReceived.Add(1)
metrics := httpsnoop.CaptureMetrics(next, w, r) // Only place `httpsnoop` is needed
totalResponsesSent.Add(1)
totalProcessingTimeMicroseconds.Add(metrics.Duration.Microseconds())
totalResponsesSentByStatus.Add(strconv.Itoa(metrics.Code), 1)
})
}
...
Now, we can wrap our entire routes with this middleware:
// cmd/api/routes.go
func (app *application) routes() http.Handler {
...
return app.metrics(app.recoverPanic(router))
}
With that, all the features of our backend system have been added.
NOTE: The code in the repo has an additional feature which ensures that if our application is abruptly interrupted, it will wait for backend tasks and pending requests to be fulfilled before stopping. You can check that out.
In the next one, we will build out the remaining front-end codes.
Outro
Enjoyed this article? Consider contacting me for a job, something worthwhile or buying a coffee ☕. You can also connect with/follow me on LinkedIn and Twitter. It isn't bad if you help share this article for wider coverage. I will appreciate it...
Top comments (0)