DEV Community

Cover image for Step-by-Step Guide: Uploading and Downloading Binary Files to AWS S3 v3 using Node.js and Codehooks.io
Jones - codehooks.io
Jones - codehooks.io

Posted on • Updated on

Step-by-Step Guide: Uploading and Downloading Binary Files to AWS S3 v3 using Node.js and Codehooks.io

Are you looking for an API to upload and download binary files to AWS S3 v3 using Node.js? Look no further! In this step-by-step guide, we will walk you through the process of achieving this seamlessly with a Codehooks.io API and easy deployment to the cloud.

The complete source code is available at GitHub

Set Up AWS S3

To begin, make sure you have an AWS account and an S3 bucket created. Note down the bucket name and region for future reference.

Then get your secure access credential to use with the AWS S3 API, read more about how to in this excellent Medium guide from @shamnad.p.s.

Create a codehooks.io project and install required Packages

If you're new to Codehooks.io you can read the getting started guide here.

Install the official Codehooks command line interface (CLI) first.

npm install codehooks -g
Enter fullscreen mode Exit fullscreen mode

Then, after logging in, create a new project, in this example we'll name our project s3project. Choose any project title you prefer.

coho create s3project
cd s3project
Enter fullscreen mode Exit fullscreen mode

In your project directory, initialise and install the necessary packages by running the following commands:

npm init es6 -y
npm install codehooks-js
npm install @aws-sdk/client-s3
Enter fullscreen mode Exit fullscreen mode

Writing the Code

Now, let's dive into the code to handle the upload and download files to AWS S3.

Import the required packages:

import { app } from 'codehooks-js'
import { S3Client, GetObjectCommand, PutObjectCommand } from '@aws-sdk/client-s3'
import { PassThrough } from 'stream'
Enter fullscreen mode Exit fullscreen mode

Configure the AWS SDK with your AWS credentials and region:

Use the CLI or the Admin UI to add your secret environment variables. Note the --encrypted flag to prevent reading the actual data content of the variables.

coho set-env AWS_ACCESS_KEY_ID 'YOUR_KEY' --encrypted
coho set-env AWS_SECRET_ACCESS_KEY 'YOUR_SECRET' --encrypted
coho set-env AWS_BUCKET 'YOUR_BUCKET' --encrypted
coho set-env AWS_REGION 'YOUR_REGION' --encrypted
Enter fullscreen mode Exit fullscreen mode

Use the CLI command coho info to inspect and verify that the variables are set correctly.

With the necessary secrets in place, we can create the code to access these in the application as regular process.env values.

const { AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION, AWS_BUCKET } = process.env;

const s3config = {
    region: AWS_REGION,
    AWS_ACCESS_KEY_ID,
    AWS_SECRET_ACCESS_KEY
}

const s3client = new S3Client(s3config);
Enter fullscreen mode Exit fullscreen mode

The next step is to create an API that process file uploads from any client and store files in an AWS S3 bucket.

API to Upload a binary file to AWS S3:

The API to upload a data stream to AWS S3 uses the PutObjectCommand with a stream that we can pipe data from the client into.

// API to POST a binary data stream
app.post('/upload/single', async (req, res) => {
    try {
        // get size, type and filename from destructured header values
        const { 'content-length': ContentLength, 'content-type': ContentType, filename } = req.headers;

        const input = {
            "Bucket": AWS_BUCKET,
            "Key": `tmp/${filename}`,  // emulate file system /bucketname/tmp/filename
            "Body": new PassThrough(), // stream to pipe data through
            "ContentLength": ContentLength,
            "ContentType": ContentType
        };
        // pipe binary request data to S3 stream
        req.pipe(input.Body); 
        // create put command
        const command = new PutObjectCommand(input); 
        // execute put object command
        const response = await s3client.send(command); 
        // return data to client
        res.json(response);
    } catch (error) {
        // some error occured, return 400 status to client
        res.status(400).end(error.message)
    }
})
Enter fullscreen mode Exit fullscreen mode

Let's test the upload API with curl by uploading a test file from a local filesystem. Use coho info --examples to inspect your project API endpoint. In this example it's: https://s3project-kvqx.api.codehooks.io/dev/upload/single

curl --location 'https://s3project-kvqx.api.codehooks.io/dev/upload/single' \
--header 'x-apikey: XXXXX' \
--header 'filename: einstein.jpg' \
--header 'Content-Type: image/jpeg' \
--data '@./Einstein_tongue.jpg'
Enter fullscreen mode Exit fullscreen mode

API to Download a binary file from AWS S3:

The API to download a data stream from AWS S3 uses the GetObjectCommand with a stream that can pipe data back to the client.

// API to GET a binary data stream from AWS S3
app.get('/download/:file', async (req, res) => {
    try {
        // filename from route
        const { file } = req.params;
        const input = {
            "Bucket": AWS_BUCKET,
            "Key": `tmp/${decodeURI(file)}` // decode filename and store in /bucket/tmp/file
        };
        // Create get command
        const command = new GetObjectCommand(input);
        // Send get command
        const response = await s3client.send(command);
        // set content-type
        res.set('content-type', response.ContentType)
        // stream data back to client
        response.Body.pipe(res.writable)
    } catch (error) {
        // Woops
        res.status(400).end(error.message)
    }
})
Enter fullscreen mode Exit fullscreen mode

Easy deployment to the cloud:

After finishing the code for the AWS S3 API, we simply deploy the application to the cloud with this command:

coho deploy
Enter fullscreen mode Exit fullscreen mode

The command should output the following on success.

Project: s3project-kvqx  Space: dev
Deployed Codehook successfully! 🙌 
Enter fullscreen mode Exit fullscreen mode

Now that we understand the basic operations required to upload and download binary files to AWS S3, let's go ahead and create a simple web client that uses the API.

Happy days 🙌

Client web app to test the API

Let's create a simple (and ugly) web page to upload and download a binary file to AWS S3 through the Codehooks API.

<html>
<body>
    <h2>AWS S3 upload and download API demo</h2>
    <label> upload binary file to read here </label>
    <input type="file" id="inputFile">
    <div id="mydiv"></div>
    <script src="test.js"></script>
</body>
</html>
Enter fullscreen mode Exit fullscreen mode

Screen shot of the ugly web page:

web page example to upload binary file to AWS S3

Note that the web page refers to a JavaScript test.js which contains the following code.

/*
* Web client to upload a binary file to a Codehooks.io AWS S3 API
*/

var URL = 'https://<YOUR_PROJECT_ID_HERE>.api.codehooks.io';
var APIKEY = 'YOUR_API_TOKEN_HERE';

async function uploadFile(file, cb) {
    var myHeaders = new Headers();
    myHeaders.append("x-apikey", APIKEY);
    myHeaders.append("filename", file.name);
    myHeaders.append("content", file.type);
    myHeaders.append("content-type", file.type);
    myHeaders.append("content-length", file.size);

    var requestOptions = {
        method: 'POST',
        headers: myHeaders,
        body: file, // file stream
        redirect: 'follow'
    };

    try {
        var response = await fetch(`${URL}/dev/upload/single`, requestOptions)
        console.log(response.status, response.statusText);
        var result = await response.text();
        console.log(result);
        cb(result);
    } catch (error) {
        console.error(error)
    }
}

// HTML5 file event
function fileChange(theEvent) {
    var theFile = theEvent.target.files[0]; // target the loaded file
    uploadFile(theFile, (result) => {
        var mydiv = document.getElementById("myDiv");
        var aTag = document.createElement('a');
        aTag.setAttribute('href',`${URL}/dev/download/${theFile.name}`);
        aTag.setAttribute('target','_blank');
        aTag.innerText = theFile.name;
        mydiv.appendChild(aTag);
        mydiv.appendChild(document.createElement("br"));
        if (theFile.type.startsWith('image')) {
            var imgTag = document.createElement('img');
            imgTag.setAttribute('src',`${URL}/dev/download/${theFile.name}`);
            imgTag.setAttribute('width','200px');
            mydiv.appendChild(imgTag);
            mydiv.appendChild(document.createElement("br"));
        }
    });    
}
// get html element for file: <input type="file" id="inputFile">
document.getElementById("inputFile").addEventListener("change", fileChange, false); // listener for file upload button
Enter fullscreen mode Exit fullscreen mode

You'll find the complete example source code for AWS S3 upload and download of binary files at our Github account here

Remember to replace AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION and AWS_BUCKET with your actual AWS credentials, bucket and region.

Links

If you have read this far and even tried out the example code in your own project, you're all set to conquer the world with apps and API's that uses the power of AWS S3 for binary file persistence and much more.

Happy coding!

Top comments (0)