DEV Community

Cover image for Golang Tutorial: Upload Bulk Files to aws S3 Using Golang
Kuldeep Singh
Kuldeep Singh

Posted on • Originally published at programmingeeksclub.com

Golang Tutorial: Upload Bulk Files to aws S3 Using Golang

AWS S3 or Amazon S3 is a storage system from Amazon to store and retrieve files from anywhere on the web. It is a highly scalable, reliable, fast, inexpensive data storage system from Amazon. It provides easy to use developer kit to store and retrieve files.

In this tutorial you will learn how to upload bulk files to S3 bucket aws golang. We will use the Go AWS SDK to upload files to AWS S3.

The AWS SDK for Go requires Go 1.15 or later versions. You can view your current version of Go by running the following command.

go version
Enter fullscreen mode Exit fullscreen mode

For information about installing or upgrading your version of Go, see https://golang.org/doc/install

Before you can use the AWS SDK for Go V2, you must have an Amazon account. See How do I create and activate a new AWS account? for details

Download the AWS Golang SDK for using in your codebase using below command

go get github.com/aws/aws-sdk-go/aws
Enter fullscreen mode Exit fullscreen mode

We will cover this tutorial step by step to upload bulk files to AWS S3 Bucket server using Golang with example code.

Load require packages

We will create main.go file and import necessary packages we need to upload the files into AWS S3 Bucket.

package main

import (
    "bytes"
    "fmt"
    "log"
    "net/http"
    "os"

    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/credentials"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/s3"
)
Enter fullscreen mode Exit fullscreen mode

Custom struct for receiver

We will be using a Session struct so we can use same it as receiver, and we will pass this receiver to our main upload function,

Via using a custom struct we’re eliminating the duplication from our code. We’ll be using only single session for our all bulk files to upload into s3 bucket aws golang.

type Session struct {
    S3Session *session.Session
}
Enter fullscreen mode Exit fullscreen mode

Set up main configuration and AWS session

We will setup configuration using AWS S3 REGION and SECRET ID and SECRET KEY and create single AWS session to upload multiple files to AWS S3. Then we will call method upload() and pass AWS session instance and file details to upload file to AWS S3 server.

func main() {
    paths := []string{"", ""}
    credential := credentials.NewStaticCredentials(
        os.Getenv("SECRET_ID"),
        os.Getenv("SECRET_KEY"),
        "",
    )

    awsConfig := aws.Config{
        Region:      aws.String(os.Getenv("REGION")),
        Credentials: credential,
    }

    s, err := session.NewSession(&awsConfig)
    if err != nil {
        log.Println("failed to create S3 session:", err.Error())
        return
    }

    se := Session{s}

    err = se.upload(paths)
    if err != nil {
        log.Println(err.Error())
        return
    }

}
Enter fullscreen mode Exit fullscreen mode

Create Upload Method to Upload Bulk Files to AWS S3

We will create method upload() to upload files to AWS S3 server. We will open the file one by one and store into buffer and then put the file to AWS S3 using PutObject() method from S3. We will also specify options in the PutObjectInput when uploading the file. We will also enable AES256 encryption on files using ServerSideEncryption options of s3.PutObjectInput struct.

func (s Session) upload(paths []string) error {
    for _, path := range paths {
        upFile, err := os.Open(path)
        if err != nil {
            log.Printf("failed %s, error: %v", path, err.Error())
            continue
        }
        defer upFile.Close()

        upFileInfo, err := upFile.Stat()
        if err != nil {
            log.Printf("failed to get stat %s, error: %v", path, err.Error())
            continue
        }
        var fileSize int64 = upFileInfo.Size()
        fileBuffer := make([]byte, fileSize)
        upFile.Read(fileBuffer)

        // uploading
        _, err = s3.New(s.S3Session).PutObject(&s3.PutObjectInput{
            Bucket:               aws.String(os.Getenv("BUCKET_NAME")),
            Key:                  aws.String(path),
            ACL:                  aws.String("public-read"), // could be private if you want it to be access by only authorized users
            Body:                 bytes.NewReader(fileBuffer),
            ContentLength:        aws.Int64(int64(fileSize)),
            ContentType:          aws.String(http.DetectContentType(fileBuffer)),
            ContentDisposition:   aws.String("attachment"),
            ServerSideEncryption: aws.String("AES256"),
            StorageClass:         aws.String("INTELLIGENT_TIERING"),
        })

        if err != nil {
            log.Printf("failed to upload %s, error: %v", path, err.Error())
            continue
        }

        url := "https://%s.s3-%s.amazonaws.com/%s"
        url = fmt.Sprintf(url, os.Getenv("BUCKET_NAME"), os.Getenv("REGION"), path)
        fmt.Printf("Uploaded File Url %s\n", url)
    }

    return nil
}
Enter fullscreen mode Exit fullscreen mode

you can checkout more article like this on my personal blog

Full Code

Top comments (0)