How I Built An Open Source Serverless Newsletter Platform

This is the last week before the newsletter service Revue shuts down.

It's the service I've been using for the past 7 months to publish my newsletter. It worked decently well, had some nice features that shared the weekly summary with me, and it integrated directly with Twitter so people could sign up from my profile. Best of all, it was free!

But on January 18th, it goes away. So I had to come up with another option for publishing my newsletter. I don't want to have an interruption in service for all my lovely readers, that's not fair to you.

I did some digging on newsletter services like MailChimp, Sendinblue, and ConvertKit, and they would work except I wasn't looking to start paying a premium. Most of the newsletter services I looked into supported up to 300 readers on their free tier - and I'm fortunate enough to have more than that subscribed to the Serverless Picks of the Week newsletter.

I've had a full career as a software engineer and some would say I know my way around serverless development so I decided to write my own newsletter platform. Serverless services are inexpensive, they scale automatically to meet demand, and are one of the fastest ways to build software.

I also recently finished an automation that cross-posts my blogs that I could use as a foundation to trigger newsletter publishing. Seemed like a no-brainer to me!

If you want a tl;dr - you can skip straight to the open source repo and get started.

Finding An Email Engine

The first problem I had to solve in order to build a custom newsletter application is sending of the actual email. There are plenty of libraries that send emails, but in the spirit of assemble vs build I wanted to use a managed service.

Sending emails that don't go to spam is difficult and requires expertise and ongoing maintenance - something I do not have. So I explored two options: Amazon SES and SendGrid. There are others out there, but I had the most familiarity with these two.

By default I gravitate toward AWS services. Everything else I build is hosted and managed in AWS, so why not this?

I looked at some of the features and they didn't quite align with what I was looking for. It has reputation management and decent deliverability insights and metrics, but a big feature I was looking for and didn't find was campaign management.

I track each issue of my newsletter as a separate marketing campaign. This allows me to get stats on an issue by issue basis, like how many people am I sending this issue to, how many of them were opened, how many bounced, and how many links were clicked in inside the issue.

With that information in my pocket, I took a look at SendGrid. They had exactly what I needed. They offer the ability to create a Single Send, which is a one-time email message delivered to a contact list. Sounds exactly like a newsletter issue to me!

SendGrid also offers a wonderful free tier - it manages up to 2,000 contacts and allows you to send up to 6,000 Single Send messages a month. Perfect for the size of my newsletter and it's projected growth for the year!

With that, I landed on SendGrid as the email management engine behind the serverless app I was going to build.

Building a Workflow

I've had a cursory knowledge of SendGrid the past few years. We use it at its most basic level at work for sending user notifications. But that was the extent of my exposure.

As I started building out my app with features I have never used before, I realized quickly that you can't publish a Single Send through SendGrid programmatically. It requires a human to manually push the schedule/send button.

This annoyed me a bit at first, but after thinking about it I realized this is a great safety feature. I'd hate to publish an email to thousands of people with a glaring typo or major formatting error. This led me to the following workflow:

Workflow getting a newsletter from creation to publish

I write all my content in Markdown. When I commit a new blog post or newsletter, it is compiled into html by the Hugo framework in the CI pipeline of my AWS Amplify project. When a build finishes successfully, Amplify publishes an event to EventBridge, which is where the workflow picks it up.

1. Write the newsletter in Markdown and push to main branch in GitHub repo
2. Amplify build compiles into html and publishes content to my site
3. A Lambda function is triggered that loads the Markdown file from my repo
4. A Step Function workflow is kicked off that parses the content, runs it through a transform for SendGrid, and creates the Single Send
5. I receive an email with a link to review the generated Single Send and hit the Approve button
6. SendGrid takes care of the scheduled delivery to all my readers

It's a pretty hands-off workflow that saves me tons of time every week. With Revue, I had to preview the html content locally on my site, copy and paste the content into Revue, then reorganize the content and resubmit images and embedded information.

This process wasn't particularly difficult, but it did take a while and was prone to error.

By automating this process through code, it saves me all that time and guarantees I don't forget anything week over week.

Transforming the Data

SendGrid has a neat feature called dynamic templates that allows you to create a parameterized email message using Handlebars.

Using dynamic templates, you can pass in raw data and have it formatted into consistent, structured html. SendGrid has a nice user experience around building these templates that provides a drag and drop interface for newsletter components that you can extend with code. It also provides a test bed so you can see merged templates as you build.

I opted to host my newsletter template in SendGrid. Using the builder, I can make quick adjustments to the format and see the changes reflected on the fly. In my newsletter app, the template is loaded via the SendGrid API and the data is merged via the Handlebars npm package.

However, to merge the data into the dynamic template, the content must be in a specific format. A Lambda function is executed as part of the Step Function workflow that will convert the newsletter markdown into a formatted json object with the following format:

{
  "metadata": {
    "number": 1,
    "title": "My first newsletter!",
    "date": "2023-01-01",
    "url": "https://readysetcloud.io/newsletter/1"
  },
  "content": {
    "sections": [
      {
        "header": "Serverless Superhero",
        "text": "<p>Congrats to our serverless superhero!</p>"
      }
    ]
  }
}

By structuring the newsletter content as a json object, we enable the Handlebars package to substitute values into the message template. This results in easy maintainability in the future by isolating the data separately from the formatting. If I need to adjust the styling of a particular component, no problem! I don't need to touch the data and add risk to my change.

Once the data is merged into the template, the newsletter app uses the SendGrid API to create a Single Send. After that, it sends me an email for review!

Other Cool Features

Revue offered a nice feature that I was sad to lose. It would send me a summary of how well my newsletter did after a few days. That got me thinking about some other features that I'd like to have managed for me that I was taking on myself.

• Sponsor support
• Automated reminders
• Issue summary
• Subscriber count tracking

The newsletter app has all of these built-in so you can pick up and go without having to think about it.

Sponsors

I often have a sponsor for issues of my newsletter. But I'm terrible at reminding them about an upcoming ad. Sponsors often want to run different ads week over week, so a reminder to get a fresh ad back to me is always appreciated so they don't have to scramble the day before the issue releases.

This app tracks all the different sponsors of your newsletter and manages a json based calendar for issues. Five days prior to a new issue being published, the app looks up the next sponsor and sends the point of contact an email reminding them of the upcoming ad. When they get back to me with the ad they want to run, I add that in the newsletter metadata and that's that!

To get the ad to show up in the newsletter content, I added a short code that injects it anywhere you like. By simply adding the short code somewhere in the content, the automation will pick it up and substitute the formatted ad with enriched content about the sponsor.

Newsletter Metrics

Week over week, I like to track the individual stats of how my newsletter performed. SendGrid monitors a slew of metrics along with click tracking, which leads to some valuable insights into how readers are interacting with my content.

The problem is that the newsletter isn't my full time job and I often forget to check on the stats. By the time I check them, I run out of time to take action on the next newsletter for better engagement.

I have a Step Function workflow that runs on a schedule to check stats for me every week. It loads up the latest newsletter, uses the SendGrid API to get the metrics and the current subscriber count, then it sends me an email with a summary and direct link to the detailed analysis page.

These come in on Fridays for me. By the time I sit down at my computer for the day I have an email in my inbox waiting to tell me how my last newsletter performed four days ago and report the subscriber trend compared to last week!

Summary

This was a fun project that solved a number of headaches. Just like my cross-posting automation, this solution is saving me a significant amount of time every week.

I've said it many times before, take out the human element every chance you get. Humans make mistakes. I am certainly no exception. I no longer forget to send an email reminder to a sponsor or to check the stats of my newsletter. I also don't accidentally fail to format the email version of my newsletter correctly when I copy and paste something wrong.

This platform is open source and I'd love for you to try it out and make updates as you see fit. It does require you to make a SendGrid account, but that's the only external thing you have to do. You can find all setup instructions in the README.

With the free tier of SendGrid and the low cost of the AWS serverless implementation that drives it, this solution will cost nothing to run until you reach over 2,000 subscribers (or send more than 6,000 messages a month).

Thank you for following along and good luck on your newsletter journey!

Happy coding!