Node.Js - Stream from S3 with Partial Content support (206)

In this post I will show you how you can stream videos from a S3 bucket with Partial Content support (HTTP 206).

To give a bit of context, I have a service where users can upload their files, which are stored in a AWS S3 bucket. For security, the files in the bucket can not be accessed directly, only through my API.

When a user requests a file I stream the entire file directly from S3.

This approach is good enough for the majority of use cases when the user simply wants to download a file. But things don't work so well when the user wants to stream the file, and that's usually the case for videos - if the user wants to display the video on their webpage using the HTML5 video tag it will have limited functionality, for example, the user won't be able to use the "seeking" feature of the video player.

To solve that the server needs to support Partial Content requests.

HTTP 206 Partial Content

In the context of video streaming, we don't want to download the entire file at once. We want to be able to tell the server what portion of the video we want. When the server is able to fulfill such a request it will respond with the status code 206.

To request a portion of the file the Range header is used by the client, specifying the content range. The range is specified in bytes.

Example:
Range: bytes=0-10000

Note that the video player does this automatically for us.

In the response, the server will include the Content-Range header to inform the client the range of the content being sent and Content-Length informing the size of the file.

Example:
Content-Range: bytes 0-10000/10001
Content-Length: 10001

Show me the code!

In this example I'm using the version 3 of the AWS SDK and will take advantage of the Range property when retrieving the file from S3.

So here's how the controller looks like!

const { S3 } = require('@aws-sdk/client-s3');
const s3Config = require('...YOUR_S3_CONFIG_FILE...')

// #1
const s3AwsClient = new S3({
    credentials: {
        accessKeyId: s3Config.accessKeyId,
        secretAccessKey: s3Config.secretAccessKey
    },
    region: s3Config.region,
    endpoint: 'https://' + s3Config.endpoint
});

async (req, res, next) => {
  // #2
  const s3Options = {
    Bucket: s3Config.bucket,
    Key: objectKey // key to your object in S3
  };

  // #3
  const { ContentLength } = await s3AwsClient.headObject(s3Options);
  const videoSize = ContentLength;

  // #4
  const FILE_PORTION_SIZE = 5000000; //bytes = 5MB
  const requestedRange = req.headers.range || '';
  const start = Number(requestedRange.replace(/\D/g, ''));
  const end = Math.min(start + FILE_PORTION_SIZE, videoSize - 1);
  const contentLength = end - start + 1;

  // #5
  res.status(206);
  res.setHeader('Cache-Control', 'no-cache');
  res.setHeader('Accept-Ranges', 'bytes');
  res.setHeader('Content-Range', `bytes ${start}-${end}/${videoSize}`);
  res.setHeader('Content-Length', contentLength);

  // #6
  const streamRange = `bytes=${start}-${end}`;
  const s3Object = await s3AwsClient.getObject({
    ...s3Options,
    Range: streamRange
  });

  // #7  
  s3Object.Body.pipe(res);
}

Let's analyse what's happening in the code above:

#1
Create the S3 client. If the code above doesn't work you can find more information here.

#2
s3Options contain the bucket name and the key of the file you want to access.

#3
The headObject function allows us to get information about the file. In our case we are interested in the ContentLength of the file.

#4
In this step we calculate the ranges of the file portion we want to return. After this we will have start, end and contentLength. Note that in the FILE_PORTION_SIZE variable we define the size we want to return. You can change this to your needs.

#5
Since we want to support Partial content requests, we need to set the correct response code (206), Content-Range and Content-Length headers.

#6
Now it's time to get the object from S3! For that we use getObject function. Note that we are passing the range here, if you forget this you will get the entire file.

#7
The property Body of the object returned by getObject is a stream, so we will pipe it to the response (res) object.

And this is all you need!

Using the video stream

  <video controls>
    <source
      src="url-to-your-video.mp4"
      type="video/mp4"
    />
  </video>

The HTML5 video tag does all the work for you. Once it recognises that the server supports Partial Content, it will provide the correct Range headers and make the sequential requests to get the video.

BONUS! Are you using CloudFlare?

After implementing this everything worked perfectly on my local machine, but once it was deployed to the server it wouldn't work. The video would unexpectedly stop and the video player would reload randomly.

All the requests to my server are proxied through CloudFlare, and I was suspicious that the problem could be here.

And I was right, after some googling I discovered a thread where this issue was discussed. In summary, CloudFlare ignores the Range headers in the requests to cache them. As we explained above, the server expects this header to be able to calculate the range of the next portion of the video.

To go around this issue you need to create a page exception in CloudFlare to not cache the request. To do this go to page rules and add an exception. This is what I added:

URL: *app.rested.dev*.mp4*
Cache Level: Bypass

This has the downside that the requests won't be cached, but you can stream your videos without issues!

Conclusion

I hope this will help you with the implementation of Partial Content support in your. Keep in mind that this is not the most sophisticated implementation, but is good enough to get you started!

If you are curious about my service you can visit rested.dev.

And that's all for today! Thank you for reading!