DEV Community

Cover image for Setting Up a Vercel "Log Drain" with AWS
Mannie Schumpert
Mannie Schumpert

Posted on • Updated on

Setting Up a Vercel "Log Drain" with AWS

If you're hosting apps on Vercel, you might have noticed that they have a number of quality logging integrations in their Marketplace. But if those don't suit your needs, or if you're already using AWS services (especially Lambda and CloudWatch) and want to keep things lean and simple, this article is for you!

This might seem a bit involved, but the code is simple and it really only takes about 15 minutes! 😎
Also, searching API logs from Vercel in CloudWatch is a breeze! 🏄🏻 I'll tell you more about that at the end of the article.

1. Create a Lambda to process your logs

In your AWS console, go to Services -> Lambda -> Functions, and click "Create Function". Name your function as appropriate, but leave everything else in this first section at the defaults.
Image description

Under "Advanced Settings", you need to select "Enable function URL", choose "NONE" for "Auth type" (we'll be handling authentication ourselves), and check the CORS button.
Image description

Now click "Create Function". Once you're redirected to your new lambda, find "Function URL" in the upper right under "Function Overview", and copy it to your clipboard. You'll need this for the next step.

⛔️ Don't close this tab! You've got more work to do here very soon.

2. Create a new Log Drain in Vercel

In a different browser tab, go to your Vercel dashboard, then go to Settings (this is settings for your whole team in Vercel), then click "Log Drains". Here you'll see the "Add a Log Drain" form.

  1. Choose your desired sources. (I just wanted to log Next.js API requests, so I only chose "Lambda").
  2. For delivery format, we will use "NDJSON" format which is easy for us to process in our logging lambda. (More on that later.)
  3. Choose "all projects" or specific projects as you wish. Note that you can easily filter logs by project later from the CloudWatch console. (See end of article for details.) Image description
  4. Under "Endpoint" paste your lambda function URL. (You'll now see a new panel with a "Verify" button. ✋ Don't do this yet!) Image description
  5. Click the toggle for "Custom Headers". We will be authenticating requests to our logging lambda via an "authentication" header.
  6. Enter "authentication" in the Header Key. Then create a token (using something like the LastPass generator), and paste it into "Header Value", prepending "Bearer ". (This is an authentication header convention.) Now click "Add" to add the header to the Log Drain. Image description Image description

⛔️ Don't close this tab!

3. Verify your logging URL

Vercel requires that you verify your ownership of your logging URL, and to do this, you'll need to modify your lambda to return a header that Vercel specifies.
Image description

Go back to your lambda console in AWS, and replace index.mjs with this:

export const handler = async(event) => {
    const response = {
        statusCode: 200,
        body: JSON.stringify('Hello from Lambda!'),
        headers: {
            'x-vercel-verify': '<vercel-verification-string>'
        },
    };
    return response;
};
Enter fullscreen mode Exit fullscreen mode

🟢 Be sure to click "Deploy" to deploy your changes to your live lambda.

Now go back to Vercel and click "Verify". The verification panel should now go away and you'll see a blue check mark next to your URL.

🙌 We're almost ready to "Add Log Drain", but not quite yet!

4. Complete your lambda, adding authentication and logging functions

Back in your lambda console tab, you'll need to add the authentication token you generated earlier to your lambda's environment variables. Go to "Configuration -> Environment Variables" then click "Edit". Add your token with the key "AUTH_TOKEN" and click "Save".
Image description

Now, replace your function's index.mjs with the following:

/* global atob */

export const handler = async(event, context) => {
    // Return unauthorized if missing auth header
    if (!event.headers.authentication || event.headers.authentication !== `Bearer ${process.env.AUTH_TOKEN}`) {
        console.log('🔴 Unauthorized request');
        return {
            statusCode: 401,
        };
    }
    /**
     * We convert the request body from Base64,
     * then split the resulting string by new lines.
     * (The NDJSON format sends line-delimited JSON logs.)
     */
    const logs = atob(event.body).split('\n');

    // We create a log entry for each item received.
    logs.forEach((log) => {
        /**
         * The split function will leave an empty item
         * at the end of the array. We don't want to log this.
         */
        if (log !== ''){
          console.log(log);
        }
    });

    return {
        statusCode: 200,
    };
};
Enter fullscreen mode Exit fullscreen mode

🟢 Again, be sure to click "Deploy" to deploy your changes to your live lambda.

Your lambda is now complete! 🕺🏻

FINAL STEP: Activate the Log Drain in Vercel

Finally, go back to your Vercel tab and click the "Add Log Drain" button.

Done! 🎉

(Note that Vercel will show a dialog with information about securing your logging lambda, but we've already handled this, so you can ignore these instructions unless you prefer the authentication method they suggest.)


🚨 Bonus: Working with API logs in CloudWatch and Logs Insights

To see the logs you've ingested from Vercel, go to your lambda's console, click "Monitor" then click "View CloudWatch logs". To view all logs, click "Search all log streams", then apply filters as needed to find the logs you're looking for.

The way we've set up our logging, with one entry per log sent by Vercel, logs can be filtered as though they are JSON objects using CloudWatch filter syntax. For example, to view all GET requests, add this to the search/filter input:
{$.proxy.method = "GET"}

Or to filter by project, use:
{$.projectName = "my-nextjs-project"}

In Log Insights, you can create some valuable visualizations, like how many requests missed the cache per hour:

fields @message, @timestamp
|  filter(proxy.vercelCache = "MISS")
|  stats count() by bin(1h)
Enter fullscreen mode Exit fullscreen mode

...or stats for requests for specific project grouped by request method:

fields @message, @timestamp
|  filter(projectName = "my-nextjs-project")
|  stats count() by proxy.method
Enter fullscreen mode Exit fullscreen mode

I hope this helps!

Top comments (4)

Collapse
 
athesto profile image
Gustavo Adolfo Mejía Sánchez

Awesome explanation, It's what I need. Just a little feedback

There is an error in the code. When you set the custom header you named it as authentication, but in the code you are asking for event.headers.authorization

Again Thanks for the excellent post :D

Collapse
 
mannieschumpert profile image
Mannie Schumpert

Thanks for the heads up!

Collapse
 
cjsingh profile image
Charanjit Singh

Thank you for the post. If you want to just copy paste and not do to and fro tab switching and checking if everything is set up correctly. You can use following script:


/* global atob */


export const handler = async (event, context) => {

  // Check if authorization header is present

  if (!event.headers.authentication || event.headers.authentication !== `Bearer ${process.env.AUTH_TOKEN}`) {
    console.log('🔴 Unauthorized request');
    return {
      statusCode: 401,
    };
  }


  if (event.body === undefined) {
    // No body i.e. check request

    return {
      statusCode: 200,
      headers: {
        'x-vercel-verify': process.env.INTEGRATION_SECRET
      }
    };
  }

  /**
   * We convert the request body from Base64,
   * then split the resulting string by new lines.
   * (The NDJSON format sends line-delimited JSON logs.)
   */
  try{

    const logs = atob(event.body).split('\n');

    // We create a log entry for each item received.
    logs.forEach((log) => {
      /**
       * The split function will leave an empty item
       * at the end of the array. We don't want to log this.
      */
     if (log !== '') {
       console.log(log);
      }
    });
  } catch (e) {
    console.log('🔴 Error decoding body', e);
    // Continue to return 200 to avoid retries
  }

  return {
    statusCode: 200,
    headers: {
      'x-vercel-verify': process.env.INTEGRATION_SECRET
    }
  };
};

Enter fullscreen mode Exit fullscreen mode

and add INTEGRATION_SECRET and AUTH_TOKEN in Lambda env variables

INTEGRATION_SECRET is provided by vercel and AUTH_TOKEN is your random password from lastpass

Collapse
 
sebdevgee123 profile image
Sebastian Goos

Thanks for sharing this!

After setting up based on the tutorial we receive an error on the final setup "Add Log Drain".

Cannot validate endpoint url: either the url is not reachable or the x-vercel-verify header in the response did not contain the expected value

The verification in step #3 just works fine and we can still see the blue verified tick/checkmark.

Any hint of what we are doing wrong?