Interacting with AWS S3 using Spring Boot application

#springboot #java #aws #programming

When it comes to storing objects (files, images..), there are multiple options to choose from. Should I go with keeping files on same/different server as that of application or in database as a BLOB or persist path as record on DB? What about scalability, maintenance, resiliency, decoupling etc.?

With increasing popularity of cloud computing services like storage, fortunately we don't have to worry about all of these anymore. AWS Simple Storage Service (S3) is one of such popular services that you can use to resolve your data storage concerns.

Introduction to AWS S3:

Amazon S3 (Simple Storage Service) is a scalable, high-speed, low-cost web-based service designed for online backup and archiving of data and application programs. It allows to upload, store, and download any type of files up to 5 TB in size. The subscriber has control over the accessibility of data, i.e. privately/publicly accessible.

AWS provides SDKs to manage your objects programmatically. Let's look at steps to get started.

First and foremost thing that you need is AWS account.
Access keys to gain access to storage using AWS storage APIs. You can either use AWS root account's or IAM user account's access keys. Checkout below image to see where these keys are available.
Go to Security Credentials and click on Create Access Key.

Search for S3 and create a bucket with all default values for now.

You need to select AWS S3 region of your choice where you would like to store the data. Please be aware that region affects your AWS bill. More details are available here.

That's enough setup for AWS S3 bucket, let's move on to Spring boot part of this article and get hands dirty with some coding.

I will be using IntelliJ IDEA for this project with Spring Initiliazr plugin & Maven as build tool.

Create a new project and choose "Spring Initiliazr" from Generators section.

Specify project properties as per your need and create project.

Add AWS S3 SDK dependency in pom.xml. There are few other dependencies needed for this project. GitHub repository link is shared at the end of this article.

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk-s3</artifactId>
    <version>1.12.239</version>
</dependency>

Add below parameters in application.properties under resources folder.

We will be dividing this into 3 parts.

Configuration class which will be responsible to authenticate application with AWS S3 client.
Service class responsible to interact with S3 bucket.
Controller class containing APIs to perform file operations.

Configuration class:

package com.springbootstorage.awss3.configuration;

import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import lombok.Getter;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class S3ClientConfiguration {

    @Value("${access.key.id}")
    @Getter
    private String accessKeyId;

    @Value("${access.key.secret}")
    @Getter
    private String accessKeSecret;

    @Value("${s3.region.name}")
    @Getter
    private String bucketRegion;

    @Bean
    public AmazonS3 getAwsS3Client() {
        BasicAWSCredentials basicAWSCredentials = new BasicAWSCredentials(accessKeyId, accessKeSecret);
        return AmazonS3ClientBuilder
                .standard()
                .withCredentials(new AWSStaticCredentialsProvider(basicAWSCredentials))
                .withRegion(bucketRegion)
                .build();
    }
}

Service Class:

package com.springbootstorage.awss3.service;

import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.model.*;
import lombok.Data;
import org.apache.commons.io.FileUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;

import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;

@Data
@Service
public class S3FileService {

    @Value("${s3.bucket.name}")
    private String s3BucketName;
    private final String DOWNLOAD_PATH = "D:\\S3Download\\";
    private final String UPLOAD_PATH = "E:\\";

    @Autowired
    private AmazonS3 amazonS3;

    // List all objects in the bucket
    public List<String> listAllObjects(String s3BucketName) {

        List<String> listOfObjects = new ArrayList<>();

        ObjectListing objectListing = amazonS3.listObjects(s3BucketName);
        for (S3ObjectSummary objectSummary : objectListing.getObjectSummaries()) {
            listOfObjects.add("FileName: " + objectSummary.getKey() +
                    " | " + "LastModified: " + objectSummary.getLastModified() +
                    " | " + "Size: " + objectSummary.getSize());
        }

        return listOfObjects;
    }

    // Downloading object from the bucket
    public String downloadObject(String s3BucketName, String objectName) {
        S3Object s3Object = amazonS3.getObject(s3BucketName, objectName);
        S3ObjectInputStream inputStream = s3Object.getObjectContent();
        try {
            FileUtils.copyInputStreamToFile(inputStream, new File( DOWNLOAD_PATH + objectName));
            return DOWNLOAD_PATH + objectName +  " Downloaded Successfully!";
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
    }

    // Upload object to the bucket
    public PutObjectResult uploadObject(String s3BucketName, String objectName, File objectToUpload) {
        return amazonS3.putObject(s3BucketName, objectName, objectToUpload);
    }

    public void deleteObject(String bucketName, String objectName) {
        amazonS3.deleteObject(bucketName, objectName);
    }
}

Controller class:

package com.springbootstorage.awss3.controller;

import com.amazonaws.services.s3.model.PutObjectResult;
import com.springbootstorage.awss3.service.S3FileService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.*;

import java.io.File;
import java.util.List;

@RestController
@RequestMapping("/api/v1")
public class S3Controller {

    @Autowired
    private S3FileService s3FileService;

    // List all files in the bucket
    @GetMapping("/listAllFiles")
    public String listAllFilesInBucket() {
        List<String> files = s3FileService.listAllObjects(s3FileService.getS3BucketName());
        StringBuilder allFiles = new StringBuilder();
        allFiles.append("****************************************\n");
        allFiles.append("List of Files available in bucket ").append(s3FileService.getS3BucketName());
        allFiles.append("\n****************************************\n");
        for (String filename : files)
            allFiles.append(files).append("\n");
        return allFiles.toString();
    }

    @PostMapping("/upload/{fileToUpload}")
    public PutObjectResult uploadFiletoBucket(@PathVariable String fileToUpload) {
        return s3FileService.uploadObject(s3FileService.getS3BucketName(), fileToUpload, new File(s3FileService.getUPLOAD_PATH()+ fileToUpload));
    }

    @PostMapping("/download/{fileToDownload}")
    public String downloadFileFromBucket(@PathVariable String fileToDownload) {
        return s3FileService.downloadObject(s3FileService.getS3BucketName(), fileToDownload);
    }

    @PostMapping("/delete/{fileToDelete}")
    public String deleteFileFromBucket(@PathVariable String fileToDelete) {
        s3FileService.deleteObject(s3FileService.getS3BucketName(), fileToDelete);
        return fileToDelete + " has been deleted successfully from " + s3FileService.getS3BucketName();
    }
}

Lets start Spring boot application and test this API for Listing, uploading, downloading and deleting objects on S3 bucket now.

We have below 4 endpoints to perform these operations.

http://localhost:9090/api/v1/listAllFiles
http://localhost:9090/api/v1/upload/
http://localhost:9090/api/v1/download/
http://localhost:9090/api/v1/delete/

Upload objects to S3 bucket.

curl -XPOST http://localhost:9090/api/v1/upload/sample1.jpg
curl -XPOST http://localhost:9090/api/v1/upload/sample2.jpg
curl -XPOST http://localhost:9090/api/v1/upload/employees.json