In this post, you are going to see how I created a Serverless Video Transcoding Pipeline using AWS MediaConvert using Nodejs.
There will be two parts to this project this is the first part where I will be showing you how I built the backend for this using AWS Serverless.
Let’s get started by creating a blank folder with serverless.yml
file which will be the core file to deploy our Serverless stack to AWS.
Creating Serverless.yml file
service: video-transcoding-pipeline
provider:
name: aws
region: ${file(./env.yml):${opt:stage}.REGION}
runtime: nodejs14.x
versionFunctions: false
tracing:
lambda: true
functions:
- ${file(./lambdaFunctions.yml)}
resources:
- ${file(./permissions.yml)}
- ${file(./db.yml)}
- ${file(./s3.yml)}
As you can see here that we are importing a bunch of yml files which we will be creating next, we are also setting the region which is imported from the env file of the project.
To know more about serverless.yml file check out “What is a serverless.yml file?” section here.
Creating S3 buckets
Resources:
MediaInputBucket:
Type: AWS::S3::Bucket
Properties:
CorsConfiguration:
CorsRules:
- AllowedHeaders: ["*"]
AllowedMethods: [GET, PUT, POST]
AllowedOrigins: ["*"]
MediaOutputBucket:
Type: AWS::S3::Bucket
Properties:
CorsConfiguration:
CorsRules:
- AllowedHeaders: ["*"]
AllowedMethods: [GET, PUT, POST]
AllowedOrigins: ["*"]
AccessControl: PublicRead
Now we will be creating the s3.yml
file which will be responsible for creating the S3 buckets, we are creating two buckets here.
The MediaInputBucket
is the input bucket where the video file will be uploaded to get transcoded.
The MediaOutputBucket
is the output bucket where the transcoded video will be saved by AWS MediaConvert.
- CorsRules : This config is used to set the Cors for the buckets so we can interact with the buckets through the client side (these can be changed according to the need).
- AccessContro l: This is giving the public access to the bucket so transcoded videos can be played publicly.
To check more configurations provided for S3 bucket creation, check out the official documentation.
Creating a DynamoDB Table
Resources:
VideoStatus:
Type: AWS::DynamoDB::Table
Properties:
TableName: ${file(./env.yml):${opt:stage}.VIDEO_STATUS_TABLE}
PointInTimeRecoverySpecification:
PointInTimeRecoveryEnabled: true
AttributeDefinitions:
- AttributeName: id
AttributeType: S
KeySchema:
- AttributeName: id
KeyType: HASH
BillingMode: PAY_PER_REQUEST
Here a DynamoDB table is being created, this table will be used to store the AWS MediaConvert Job status (more on this later).
Also, you can see that the table name is also being imported from the env file, so let’s create this file now.
Creating the env.yml file
prod:
MEDIA_INPUT_BUCKET: !Ref MediaInputBucket
MEDIA_OUTPUT_BUCKET: !Ref MediaOutputBucket
REGION: us-east-2
VIDEO_STATUS_TABLE: VideoStatusTable
ACCOUNT_ID: [REPLACE_THIS_WITH_YOUR_ACCOUNT_ID]
MEDIA_ENDPOINT: [REPLACE_THIS_WITH_YOUR_ENDPOINT]
MEDIA_CONVERT_ROLE: !GetAtt MediaConvertRole.Arn
Here we are creating a bunch of env variables under the prod stage name.
- MEDIA_ENDPOINT : This is the endpoint for MediaConvert which you can get from your AWS Console by going under the Account section in the MediaConvert dashboard.
- MEDIA_CONVERT_ROLE : This is the IAM role for AWS MediaConvert.
Creating permissions.yml file
Now it’s time to create the permissions.yml
file, there will be two roles created in this file, one will be used by all the Lambda functions and another one will be used by AWS MediaConvert.
Let’s break down this file as it is a bit long.
Creating policy for interacting with DynamoDB
Resources:
LambdaRole:
Type: "AWS::IAM::Role"
Properties:
RoleName: "LambdaRole-${opt:stage}"
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: "sts:AssumeRole"
Policies:
- PolicyName: "LambdaRolePolicy-${opt:stage}"
PolicyDocument:
Version: "2012-10-17"
Statement:
- Action:
- "dynamodb:PutItem"
- "dynamodb:UpdateItem"
- "mediaconvert:*"
Effect: Allow
Resource: "*"
This policy will allow the lambda functions to interact with the DynamoDB table.
Creating policy for interacting with AWS MediaConvert
Policies:
- PolicyName: 'MediaConvertLambdaPolicy-${opt:stage}'
PolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: PassRole
Effect: Allow
Action:
- 'iam:PassRole'
Resource: !GetAtt MediaConvertRole.Arn
- Sid: MediaConvertService
Effect: Allow
Action:
- 'mediaconvert:*'
Resource:
- '*'
- Sid: MediaInputBucket
Effect: Allow
Action:
- 's3:*'
Resource:
- '*'
This policy will allow the Lambda functions to interact with AWS MediaConvert, to read more about how these permissions work check out this official documentation by AWS.
Creating policy to write CloudWatch Log streams
Policies:
- PolicyName: 'CloudWatchLogsPolicy-${opt:stage}'
PolicyDocument:
Version: '2012-10-17'
Statement:
- Action:
- 'logs:CreateLogGroup'
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
Effect: Allow
Resource:
- >-
arn:aws:logs:${file(./env.yml):${opt:stage}.REGION}:${file(./env.yml):${opt:stage}.ACCOUNT_ID}:*
This is straightforward as we are allowing the lambda log to be created in the same region and AWS account where we are deploying the stacks.
Now we will create the second role which will be attached to the MediaConvert.
Creating IAM role for AWS MediaConvert
MediaConvertRole:
Type: AWS::IAM::Role
Properties:
RoleName: "MediaConvertRole-${opt:stage}"
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service:
- "mediaconvert.amazonaws.com"
- "mediaconvert.us-east-2.amazonaws.com"
Action:
- sts:AssumeRole
Policies:
- PolicyName: "MediaConvertPolicy"
PolicyDocument:
Statement:
- Effect: "Allow"
Action:
- "s3:*"
Resource:
- "*"
- Effect: "Allow"
Action:
- "cloudwatch:*"
- "logs:*"
Resource:
- "*"
This role will allow AWS MediaConvert to interact with S3 and also be able to write AWS CloudWatch logs to the AWS account.
That was a lot to take in but now you are done with creating core yml files, now there is only one yml file left that will create all the lambda functions which are needed, so let’s start with that.
Creating lambdaFunctions.yml file
startJob:
handler: resolvers/job/startJob.handler
name: ${opt:stage}-startJob
timeout: 600
role: LambdaRole
description: Lambda function to start the media convert job
environment:
VIDEO_STATUS_TABLE: ${file(./env.yml):${opt:stage}.VIDEO_STATUS_TABLE}
MEDIA_INPUT_BUCKET: ${file(./env.yml):${opt:stage}.MEDIA_INPUT_BUCKET}
MEDIA_OUTPUT_BUCKET: ${file(./env.yml):${opt:stage}.MEDIA_OUTPUT_BUCKET}
MEDIA_ENDPOINT: ${file(./env.yml):${opt:stage}.MEDIA_ENDPOINT}
REGION: ${file(./env.yml):${opt:stage}.REGION}
MEDIA_CONVERT_ROLE: ${file(./env.yml):${opt:stage}.MEDIA_CONVERT_ROLE}
events:
- s3:
bucket: ${file(./env.yml):${opt:stage}.MEDIA_INPUT_BUCKET}
event: s3:ObjectCreated:*
existing: true
getSignedUrl:
handler: resolvers/getSignedUrl.handler
name: ${opt:stage}-getSignedUrl
timeout: 600
role: LambdaRole
description: Lambda function to get the signed url to upload the video
environment:
MEDIA_INPUT_BUCKET: ${file(./env.yml):${opt:stage}.MEDIA_INPUT_BUCKET}
events:
- http:
path: getSignedUrl
method: post
cors: true
updateJobStatus:
handler: resolvers/job/updateJobStatus.handler
name: ${opt:stage}-updateJobStatus
timeout: 600
role: LambdaRole
description: Lambda function to update the media convert job status in the DB
environment:
VIDEO_STATUS_TABLE: ${file(./env.yml):${opt:stage}.VIDEO_STATUS_TABLE}
REGION: ${file(./env.yml):${opt:stage}.REGION}
events:
- cloudwatchEvent:
event:
source:
- 'aws.mediaconvert'
detail-type:
- 'MediaConvert Job State Change'
There are three lambda functions that are being created here.
- startJob : This lambda function will be responsible to start the AWS MediaConvert job and it will be called whenever any file will be uploaded to the input S3 bucket which you created earlier.
- getSignedUrl : This lambda function will return the signed URL to upload the video file to the input bucket from the client side.
- updateJobStatus : This lambda function will be updating the MediaConvert job status to the DynanmoDB table and it will be called whenever the job status gets changed in MediaConvert.
Now you are done with creating all the required yml files, let’s move on to creating the resolvers for the lambda functions.
getSignedUrl Lambda Resolver
This lambda function will be called first to get the signed URL back and then that signed URL will be used to upload the video file to S3 from the client side so we are uploading the video from the backend.
Adding validations
const {
fileName,
metaData
} = JSON.parse(event.body)
if (!fileName || !fileName.trim()) {
return sendResponse(400, {
message: 'Bad Request'
})
}
Here you are getting the file name and the metadata from the client side and you are also checking that the file name must exist otherwise 400 Status code is being returned.
The sendResponse
is a utility function that is just sending the response to the API request, you can find it in the source code.
Creating the signed URL
const params = {
Bucket: process.env.MEDIA_INPUT_BUCKET,
Key: fileName,
Expires: 3600,
ContentType: 'video/*',
Metadata: {
...metaData
}
}
const response = s3.getSignedUrl('putObject', params)
Here parameters are being created and getSignedUrl
API call is made to get the signed URL, ContentType
is set to video/* because only videos will be uploaded to the S3 bucket from the client side.
Now when the file will get uploaded to the S3 bucket by the client application using this signed URL, startJob
lambda function will be triggered which will start the AWS MediaConvert job, let’s see what this lambda function looks like.
startJob Lambda Resolver
The first thing that I wanna show you is what are the imports that are added in this lambda resolver.
Imports
const {
sendResponse
} = require('../../utilities/index')
const AWS = require('aws-sdk')
AWS.config.mediaconvert = {
endpoint: `https://${process.env.MEDIA_ENDPOINT}.mediaconvert.${process.env.REGION}.amazonaws.com`
}
const MediaConvert = new AWS.MediaConvert({
apiVersion: '2017-08-29'
})
const s3 = new AWS.S3()
const params = require('./mediaParams.js')
const dbClient = new AWS.DynamoDB.DocumentClient()
Notice here that I am updating the endpoint for the MediaConvert config, also there is a file named mediaParams.js
which is being imported here.
This file will hold the configuration for starting the MediaConvert job, so we will now create this file first.
Creating mediaParams.js config file
module.exports = {
Settings: {
TimecodeConfig: {
Source: 'ZEROBASED'
},
OutputGroups: [
{
Name: 'Apple HLS',
Outputs: [
{
ContainerSettings: {
Container: 'M3U8',
M3u8Settings: {}
},
VideoDescription: {
Width: '',
Height: '',
CodecSettings: {
Codec: 'H_264',
H264Settings: {
MaxBitrate: '',
RateControlMode: 'QVBR',
SceneChangeDetect: 'TRANSITION_DETECTION'
}
}
},
AudioDescriptions: [
{
CodecSettings: {
Codec: 'AAC',
AacSettings: {
Bitrate: 96000,
CodingMode: 'CODING_MODE_2_0',
SampleRate: 48000
}
}
}
],
OutputSettings: {
HlsSettings: {}
},
NameModifier: 'hgh'
}
],
OutputGroupSettings: {
Type: 'HLS_GROUP_SETTINGS',
HlsGroupSettings: {
SegmentLength: 10,
MinSegmentLength: 0,
DestinationSettings: {
S3Settings: {
AccessControl: {
CannedAcl: 'PUBLIC_READ'
}
}
}
}
}
},
{
CustomName: 'Thumbnail Creation Group',
Name: 'File Group',
Outputs: [
{
ContainerSettings: {
Container: 'RAW'
},
VideoDescription: {
Width: 1280,
Height: 720,
CodecSettings: {
Codec: 'FRAME_CAPTURE',
FrameCaptureSettings: {
FramerateNumerator: 1,
FramerateDenominator: 5,
MaxCaptures: 5,
Quality: 80
}
}
}
}
],
OutputGroupSettings: {
Type: 'FILE_GROUP_SETTINGS',
FileGroupSettings: {
DestinationSettings: {
S3Settings: {
AccessControl: {
CannedAcl: 'PUBLIC_READ'
}
}
}
}
}
}
],
Inputs: [
{
AudioSelectors: {
'Audio Selector 1': {
DefaultSelection: 'DEFAULT'
}
},
VideoSelector: {},
TimecodeSource: 'ZEROBASED'
}
]
},
AccelerationSettings: {
Mode: 'DISABLED'
},
StatusUpdateInterval: 'SECONDS_60',
Priority: 0
}
As you can see that there are a lot of parameters added here but most of these values are static in this project, you will only be modifying transcoded video width/height and bitrate (many many more configurations can be made dynamic according to the requirement).
Fetching the metadata from the uploaded file
const fileKey = event.Records[0].s3.object.key
const {
metaData
} = await fetchMetaData(fileKey)
Here you are getting the uploaded file key (which will be received in the lambda trigger attached to the S3 bucket) and calling fetchFromS3
function.
Creating fetchFromS3 function
async function fetchMetaData (key) {
try {
const params = {
Bucket: MEDIA_INPUT_BUCKET,
Key: key
}
const response = await s3.headObject(params).promise()
return { metaData: response.Metadata }
} catch (err) {
throw new Error(err)
}
}
Creating the params for starting the MediaConvert job
const input = `s3://${MEDIA_INPUT_BUCKET}/${fileKey}`
const output = `s3://${MEDIA_OUTPUT_BUCKET}/`
params.Role = MEDIA_CONVERT_ROLE
params.Settings.OutputGroups[0].OutputGroupSettings.HlsGroupSettings.Destination = output
params.Settings.OutputGroups[1].OutputGroupSettings.FileGroupSettings.Destination = output
params.Settings.Inputs[0].FileInput = input
params.Settings.OutputGroups[0].Outputs[0].VideoDescription.Width = metaData.videowidth || 1920
params.Settings.OutputGroups[0].Outputs[0].VideoDescription.Height = metaData.videoheight || 1080
params.Settings.OutputGroups[0].Outputs[0].VideoDescription.CodecSettings.H264Settings.MaxBitrate = metaData.videobitrate || 6000000
const response= await MediaConvert.createJob(params).promise()
We are setting the IAM role for MediaConvert and other settings with metadata as we discussed previously.
Creating initial entry for the created job in the DB
const vodObj = {
TableName: VIDEO_STATUS_TABLE,
Item: {
id: response.Job.Id,
createdAt: new Date().toISOString(),
vodStatus: 'SUBMITTED'
},
ConditionExpression: 'attribute_not_exists(id)'
}
await dbClient.put(vodObj).promise()
We are taking the created job id and making it a Sort key in the DynamoDB table and we are also setting the initial job status to SUBMITTED.
Now it is time to work on the last lambda function resolver.
updateJobStatus Lambda Resolver
try {
const { VIDEO_STATUS_TABLE, REGION } = process.env
const { jobId, status, outputGroupDetails } = event.detail
const params = {
TableName: VIDEO_STATUS_TABLE,
Key: {
id: jobId
},
ExpressionAttributeValues: {
':vodStatus': status
},
UpdateExpression: 'SET vodStatus = :vodStatus',
ReturnValues: 'ALL_NEW'
}
if (status !== 'INPUT_INFORMATION') {
if (status === 'COMPLETE') {
const splitOutput = outputGroupDetails[0].outputDetails[0].outputFilePaths[0].split('/')
params.ExpressionAttributeValues[':outputPath'] = `https://${splitOutput[2]}.s3.${REGION}.amazonaws.com/${splitOutput[3]}`
params.UpdateExpression += ', outputPath = :outputPath'
}
await dbClient.update(params).promise()
}
} catch (err) {
return sendResponse(500, { message: 'Internal Server Error' })
}
This will be the final lambda function resolver you will need, this lambda will be called whenever the status of the MediaConvert job changes and it will update the new status to the DynamoDB table using the job id we stored earlier.
There are three main stages of a job progression –
-
SUBMITTED : This is the initial job status when it gets started and this is being stored by
startJob
lambda function. - PROGRESSING : This is the status when the job is going on and it will be set through this lambda function.
- COMPLETE : This is the final status when the job gets successfully completed.
If you want to read more about the different stages of a job, you can check here.
AND we are done, pat yourself on the back if you have reached this far, there are a lot of improvements that can be done in this project.
Improvements
- MediaConvert endpoint can be fetched using
describeEndpoints
API, read more here. - More configuration can be added to the AWS MediaConvert startJob parameters.
- Multi-part upload can be implemented to upload larger video files.
- Job status can be pushed to SNS topic to use in other places.
- AWS CloudFront distribution can be used to distribute the transcoded video.
Conclusion
Today you saw how we can create a video transcoder using AWS MediaConvert with Serverless and Nodejs, you can play around with it and have fun adding new things, there will be a part 2 to this series where i will be showing how to make the Frontend for this
Find the full source code here.
Check more posts:
What is AWS Athena?
DynamoDB VS MongoDB: Detailed Comparison
AWS DynamoDB Pricing Model and Features Explained
The post How I Built a Video Transcoder using AWS MediaConvert appeared first on DevsWisdom.
Top comments (0)