AWS S3 or Amazon S3 is a storage system from Amazon to store and retrieve files from anywhere on the web. It is a highly scalable, reliable, fast, inexpensive data storage system from Amazon. It provides easy to use developer kit to store and retrieve files.
In this tutorial you will learn how to upload bulk files to S3 bucket aws golang. We will use the Go AWS SDK to upload files to AWS S3.
The AWS SDK for Go requires Go 1.15 or later versions. You can view your current version of Go by running the following command.
go version
For information about installing or upgrading your version of Go, see https://golang.org/doc/install
Before you can use the AWS SDK for Go V2, you must have an Amazon account. See How do I create and activate a new AWS account? for details
Download the AWS Golang SDK for using in your codebase using below command
go get github.com/aws/aws-sdk-go/aws
We will cover this tutorial step by step to upload bulk files to AWS S3 Bucket server using Golang with example code.
Load require packages
We will create main.go file and import necessary packages we need to upload the files into AWS S3 Bucket.
package main
import (
"bytes"
"fmt"
"log"
"net/http"
"os"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
)
Custom struct for receiver
We will be using a Session struct so we can use same it as receiver, and we will pass this receiver to our main upload function,
Via using a custom struct we’re eliminating the duplication from our code. We’ll be using only single session for our all bulk files to upload into s3 bucket aws golang.
type Session struct {
S3Session *session.Session
}
Set up main configuration and AWS session
We will setup configuration using AWS S3 REGION and SECRET ID and SECRET KEY and create single AWS session to upload multiple files to AWS S3. Then we will call method upload() and pass AWS session instance and file details to upload file to AWS S3 server.
func main() {
paths := []string{"", ""}
credential := credentials.NewStaticCredentials(
os.Getenv("SECRET_ID"),
os.Getenv("SECRET_KEY"),
"",
)
awsConfig := aws.Config{
Region: aws.String(os.Getenv("REGION")),
Credentials: credential,
}
s, err := session.NewSession(&awsConfig)
if err != nil {
log.Println("failed to create S3 session:", err.Error())
return
}
se := Session{s}
err = se.upload(paths)
if err != nil {
log.Println(err.Error())
return
}
}
Create Upload Method to Upload Bulk Files to AWS S3
We will create method upload() to upload files to AWS S3 server. We will open the file one by one and store into buffer and then put the file to AWS S3 using PutObject() method from S3. We will also specify options in the PutObjectInput when uploading the file. We will also enable AES256 encryption on files using ServerSideEncryption options of s3.PutObjectInput struct.
func (s Session) upload(paths []string) error {
for _, path := range paths {
upFile, err := os.Open(path)
if err != nil {
log.Printf("failed %s, error: %v", path, err.Error())
continue
}
defer upFile.Close()
upFileInfo, err := upFile.Stat()
if err != nil {
log.Printf("failed to get stat %s, error: %v", path, err.Error())
continue
}
var fileSize int64 = upFileInfo.Size()
fileBuffer := make([]byte, fileSize)
upFile.Read(fileBuffer)
// uploading
_, err = s3.New(s.S3Session).PutObject(&s3.PutObjectInput{
Bucket: aws.String(os.Getenv("BUCKET_NAME")),
Key: aws.String(path),
ACL: aws.String("public-read"), // could be private if you want it to be access by only authorized users
Body: bytes.NewReader(fileBuffer),
ContentLength: aws.Int64(int64(fileSize)),
ContentType: aws.String(http.DetectContentType(fileBuffer)),
ContentDisposition: aws.String("attachment"),
ServerSideEncryption: aws.String("AES256"),
StorageClass: aws.String("INTELLIGENT_TIERING"),
})
if err != nil {
log.Printf("failed to upload %s, error: %v", path, err.Error())
continue
}
url := "https://%s.s3-%s.amazonaws.com/%s"
url = fmt.Sprintf(url, os.Getenv("BUCKET_NAME"), os.Getenv("REGION"), path)
fmt.Printf("Uploaded File Url %s\n", url)
}
return nil
}
you can checkout more article like this on my personal blog
Top comments (0)