Running Puppeteer on Amazon Web Services' (AWS) serverless computing platform, Lambda, is it a bit of a challenge. Puppeteer and it's bundled version of Chromium easily exceeds the lambda size limitation. There are a few options you opt for instead:
- Run or maintain your own farm of Google Chrome / Chromium instances
- Use a Google Chrome / Chromium-as-a-Service offering like Browserless
- Use an NPM module,
npm i @serverless-chrome/lambdato handle this for you
I'm going to be walking you through how I built "Get Me the GIF" (referred to as GMTG hereafter) with Puppeteer, Serverless, Chromium, and Nodejs.
The Idea
The idea behind GMTG is an easy one. When Twitter user's tweet GIFs out, Twitter will show these GIFs as videos instead of GIFs.
I'm somewhat of a GIF collector. You never know when a GIF is more useful in communicating an idea than words are. When I see GIFs on Twitter that are funny, I like to save those for later. Twitter for the web and Android make this impossible. One day, I had more free time than usual and decided to open up Chrome DevTools and explore the code behind how a tweet is displayed in the browser. Long story short, it's relatively easy to capture and download these GIFs (videos). I wrote a little extra code to use FFMPEG to convert the video I capture from Twitter into a GIF I can add to my collection.
I'm not going to highlight all the pieces I used, but just a few pieces of code I thought were useful in helping me reach my end goal, getting all of those GIFs.
The Flow
You give GMTG a URL to a tweet containing the GIF, like:
https://twitter.com/EvanHalley/status/1130891914675445760
It should give you a GIF.
The overall flow:
- Open the Tweet
- Intercept the network request that rendered the MP4 preview frame
- Parse out identifier of the MP4 preview frame
- Build the URL to access the MP4 and download it
- Use FFMPEG to convert the MP4 to a GIF
- Make the GIF available to the user
Getting Chrome to Run in AWS Lambda
Note: For this project, I am using the Serverless Framework to debug and deploy my serverless app to AWS Lambda. I'm not going to dive into Serverless, but checkout this great Serverless tutorial.
As mentioned earlier, getting a Puppeteer + Chrome based Nodejs app running in AWS Lambda is difficult because of the deployment package size limits. In order to get around this limitation, I used a NPM module, serverless-chrome.
Without jumping to far into the details, serverless-chrome handles everything needed to get Chrome up and running in a serverless environment and manages to get around the deployment package size limitations.
The aim of this project is to provide the scaffolding for using Headless Chrome during a serverless function invocation. Serverless Chrome takes care of building and bundling the Chrome binaries and making sure Chrome is running when your serverless function executes. In addition, this project also provides a few example services for common patterns (e.g. taking a screenshot of a page, printing to PDF, some scraping, etc.)
Once you have a Nodejs, Puppeteer, and Serverless project bootstrapped, you can easily add serverless-chrome:
npm install --save @serverless-chrome/lambda
In your source code, connect to a Chrome instance running in a serverless environment:
const launchChrome = require("@serverless-chrome/lambda");
async function getChrome() {
let chrome = await launchChrome();
let response = await request
.get(`${chrome.url}/json/version`)
.set("Content-Type", "application/json");
console.log(JSON.stringify(response.body));
let endpoint = response.body.webSocketDebuggerUrl;
return {
endpoint,
instance: chrome
};
}
The code snippet above, calls launchChrome() to start a Chrome process. Once it's launched, we can query the Chrome instance to find the URL to the Chrome DevTools Protocol (CDP) socket. Puppeteer uses this URL to connect to Chrome.
Making a GET request to this URL + /json/version returns:
{
"Browser": "HeadlessChrome/78.0.3904.97",
"Protocol-Version": "1.3",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/78.0.3904.97 Safari/537.36",
"V8-Version": "7.8.279.23",
"WebKit-Version": "537.36 (@021b9028c246d820be17a10e5b393ee90f41375e)",
"webSocketDebuggerUrl": "ws://127.0.0.1:9222/devtools/browser/0fbe3418-968a-4d57-9b53-0cf20d590eec"
}
We are after webSocketDebuggerUrl.
Once we have a URL to the CDP socket, connecting with Puppeteer is straighforward:
const puppeteer = require("puppeteer");
...
let cdpSocketUrl = 'ws://127.0.0.1:9222/devtools/browser/0fbe3418-968a-4d57-9b53-0cf20d590eec';
let browser = await puppeteer.connect({
browserWSEndpoint: cdpSocketUrl
});
There is a caveat to know about when using serverless-chrome. It has not been updated in over a year, which means the latest prebuilt version of Chromium (69.0.3497.81) it uses is over a year old. This means it is pretty much only guaranteed to work with older versions Puppeteer.
Intercepting Requests with Puppeteer
Puppeteer has a handy API for intercepting ALL network requests the browser makes when loading a web page. You can intercept these requests and either continue or abort them. There are some really useful cases where this level of control is desired, such as capturing a webpage screenshot, but not processing any of the image or javascript. In my case, I just wanted to identify the MP4 thumbnail network request.
I discovered that the URL to the MP4 thumbnail looks like:
https://pbs.twimg.com/tweet_video_thumb/1234567890.jpg
The link to the MP4 looks like:
https://video.twimg.com/tweet_video/1234567890.mp4
Using Puppeteer I am able to write request interception code that looks for this URL.
const VIDEO_THUMBNAIL_PREFIX = 'https://pbs.twimg.com/tweet_video_thumb/';
let videoUrl = null;
page = await browser.newPage();
await page.setRequestInterception(true);
page.on('request', request => {
if (request.url().startsWith(VIDEO_THUMBNAIL_PREFIX) && request.url().endsWith('.jpg')) {
let thumbnailUrl = request.url();
let assetId = thumbnailUrl.replace(VIDEO_THUMBNAIL_PREFIX, '')
.replace('.jpg', '');
videoUrl = VIDEO_URL_PREFIX + assetId + '.mp4';
}
request.continue();
});
await page.goto(tweetUrl);
Once I had the URL to the video thumbnail, I'm easily able to build a URL to the video so I can download it later.
Converting the Video
FFMPEG is one of the most popular command line utilities for transcoding a wide area of video, audio, and still images. It's written in C. However, like many things nowadays, you can instrument it with JavaScript. I discovered a GitHub Gist that was tackling a similar problem, converting a video into a GIF.
Using traditional command line FFMPEG, you could execute the operation with:
ffmpeg -i input_video.mp4 output.gif
Using a Nodejs library, fluent-ffmpeg, the same operation looks like:
const ffmpegPath = require('@ffmpeg-installer/ffmpeg').path;
const ffmpeg = require('fluent-ffmpeg');
ffmpeg.setFfmpegPath(ffmpegPath);
ffmpeg('input_video.gif')
.on('end', () => {
// done, do something with output.gif
})
.on('error', err => {
// oh noe error!
})
.save('output.gif');
The use of fluent-ffmpeg requires the use of another Nodejs libray, @ffmpeg-installer/ffmpeg. It installs a version of ffmpeg compatible with the underlying platform (Linux, Mac, or Windows). You then tell fluent-ffmpeg where it has been installed and it takes over from there.
The End
I've deployed GMTG to AWS Lambda. You can make HTTP GET calls to it. The value of the URL parameter should be a URL to a Tweet containing a GIF (short movies probably work as well).
https://1ehn2dwcfk.execute-api.us-east-1.amazonaws.com/dev/?url=
In this example, we will extract the GIF out of this Tweet
Using the following GMTG API call:
The GIF:
The source code has been uploaded to GitHub.
https://github.com/evanhalley/get-me-the-gif
Let me know if you have any questions by leaving a comment or on Twitter, @EvanHalley.
If you like reading articles about all the cool things you can do with Puppeteer, please sign up for my email newsletter, The PuppetHero Digest.
This article was originally posted at EvanHalley.dev
Top comments (0)