loading...
Cover image for Hacker News with Serverless Stack
Lambda Store

Hacker News with Serverless Stack

svenanderson profile image Sven Anderson ・7 min read

In this article, I will implement a simple version Hacker News using serverless technologies. By the end of the article we will have the following web app: http://serverless-hackernews.s3-website-us-east-1.amazonaws.com/

If you want to check the code here the repo.

I have two objectives:

  • To show how easy to write a full serverless stack web application.
  • Many people thinks Redis is a cache. I want to show Redis can be used as a system of record database in many usecases. And the good thing is that you do not need a cache in front of it :)

To keep the blog simple and understandable I skipped some functionalities of Hacker News such as:

  • No user registration. Anyone can submit a link anonymously.
  • I did not add comments section.
  • I have not implemented API rate limiting.

So our requirements will be:

  • News API: Endpoint which will list the top news items as a list.
  • Vote API: Endpoint that anyone vote up a news item.
  • Submit API: Anyone can submit a news with a URL and title.
  • Static web site

Here the stack that I will use:

  • Client side: React
  • Static hosting: AWS S3
  • Backend (the API server): AWS Lambda + API Gateway
  • Deployment: AWS SAM
  • Database: Lambda Store (Redis)

Alt Text

Data Model

We will use Redis with Lambda Store. as it is the only Serverless Redis service. If you want to learn more about it read my blog post where I introduce lambda store.

I have created a Redis database using the console. I have entered the endpoint, password and the port as environment variable into the SAM template as here. In my function code, I created my Redis client as below:

const redis = require("redis");

const client = redis.createClient ({
    host : process.env.ENDPOINT,
    port : process.env.PORT,
    password: process.env.PASSWORD
});

One Redis database should be enough for my usecase. I will use two data structures:

  • SortedSet to keep the votes together with id's of news items.
  • Hash to keep the data (url and title) of each news item.

Alt Text

Submitting a new entry (submit.js)

When a user submits an entry (url+title), the request comes to submitFunction (AWS Lambda). We create a unique id using 'uuid' node.js library. Then we create one entry in the sortedset with default score 100 but also hash entry to keep the details like url and title. See the code here or below:


const redis = require("redis");

const client = redis.createClient ({
    host : process.env.ENDPOINT,
    port : process.env.PORT,
    password: process.env.PASSWORD
});

const {promisify} = require('util');
const hsetAsync = promisify(client.hset).bind(client);
const zaddAsync = promisify(client.zadd).bind(client);

exports.submitHandler = async function(event, context) {
    if (event.httpMethod !== 'POST') {
     throw new Error(`postMethod only accepts POST method, you tried: ${event.httpMethod} method.`);
    }
    console.info('received:', event);

    // Get id and name from the body of the request
    const body = JSON.parse(event.body)
    const url = body["url"];
    const title = body["title"];
    const { v4: uuidv4 } = require('uuid');
    const id = uuidv4();

    client.on("error", function(err) {
        throw err;
    });

    await zaddAsync("news", 100, id );
    await hsetAsync(id, "url", url, "title", title);

    return {
        statusCode: 200,
        headers: {
            "Access-Control-Allow-Origin": "*",
            "Access-Control-Allow-Headers": "Content-Type",
            "Access-Control-Allow-Methods": "OPTIONS,POST"
        },
        body: JSON.stringify(id)
    };

}

Listing the news entries (news.js)

First we get the id's of top 50 news entries with the highest score (votes) from sortedset. Then we get the title and url from our hash and return the result in JSON format. See the code here or below:


const redis = require("redis");

const client = redis.createClient ({
    host : process.env.ENDPOINT,
    port : process.env.PORT,
    password: process.env.PASSWORD
});

const { promisify } = require('util');
const hgetallAsync = promisify(client.hgetall).bind(client);
const zrevrangeAsync = promisify(client.zrevrange).bind(client);

exports.newsHandler = async (event) => {
    if (event.httpMethod !== 'GET') {
        throw new Error(`news only accept GET method, you tried: ${event.httpMethod}`);
    }
    console.info('received:', event);

    let n = await zrevrangeAsync("news", 0, 50, "WITHSCORES");
    let result = []
    const promises = []  
    for (let i = 0; i < n.length-1; i += 2) {
        let id = n[i]
        let p = hgetallAsync(id).then( item => {
            if(item) {
                item["id"] = id
                item["score"] = n[i+1]
                result[i/2] =item
            }
        })
        promises.push(p);
    }

    let response
    await Promise.all(promises).then(res => {
        response = {
            statusCode: 200,
            headers: {
                "Access-Control-Allow-Origin": "*",
                "Access-Control-Allow-Headers": "Content-Type",
                "Access-Control-Allow-Methods": "OPTIONS,GET"
            },
            body: JSON.stringify(result)
        };
    });
    return response;
}

Voting an entry (vote.js)

When a user clicks on upvote icon, we call the voteFunction. The vote function simply increments the score of the item in the SortedSet. See the code here

Frontend

I implemented front end using React. I created the project using create-react-app. Also I used react-router to route urls to components (/submit -> submit component). I used the fetch api to call my lambda functions. Thanks to the rich documentation of React, it was a smooth development experience. You can check the code here

Deployment

Backend Deployment

I used AWS SAM to deploy my lambda functions. If you are not familiar with AWS SAM, see my related blog post. The good thing was that I was able to run and test my functions locally without deploying to the cloud. See my template file which creates 3 functions as well as the API gateway endpoints below:

# This is the SAM template that represents the architecture of your serverless application
# https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-template-basics.html

# The AWSTemplateFormatVersion identifies the capabilities of the template
# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/format-version-structure.html
AWSTemplateFormatVersion: 2010-09-09
Description: >-
  serverless-hackernews

# Transform section specifies one or more macros that AWS CloudFormation uses to process your template
# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/transform-section-structure.html
Transform:
- AWS::Serverless-2016-10-31

Globals:
  Function:
    Runtime: nodejs12.x
    Timeout: 180
    Environment:
      Variables:
        ENDPOINT: "YOUR_LAMBDA_STORE_ENDPOINT"
        PORT: "YOUR_DB_PORT"
        PASSWORD: "YOUR_DB_PASSWORD"
  Api:
    Cors:
      AllowMethods: "'OPTIONS,POST,GET'"
      AllowHeaders: "'Content-Type'"
      AllowOrigin: "'*'"

# Resources declares the AWS resources that you want to include in the stack
# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/resources-section-structure.html
Resources:
  submitFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: src/handlers/submit.submitHandler
      Description: The function to submit a new news item.
      Events:
        Api:
          Type: Api
          Properties:
            Path: /submit
            Method: POST

  voteFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: src/handlers/vote.voteHandler
      Description: The function to vote a news item.
      Events:
        Api:
          Type: Api
          Properties:
            Path: /vote
            Method: POST


  newsFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: src/handlers/news.newsHandler
      Description: The function to list the news.
      Events:
        Api:
          Type: Api
          Properties:
            Path: /
            Method: GET

Outputs:
  WebEndpoint:
    Description: "API Gateway endpoint URL for Prod stage"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/"

Frontend Deployment

If you have not deployed a static web site to S3 before you can follow this guide.

Troubleshooting: Enabling CORS

Honestly, enabling CORS was the most challenging part for me. If you do not enable CORS, then your client application is not able to fetch data from your lambda function (API Gateway endpoint) via AJAX calls. After some research and trial/error; I found that my lambda functions should return some specific headers inside the http response as below:

 response = {
            statusCode: 200,
            headers: {
                "Access-Control-Allow-Origin": "*",
                "Access-Control-Allow-Headers": "Content-Type",
                "Access-Control-Allow-Methods": "OPTIONS,GET"
            },
            body: JSON.stringify(result)
        };

I should note that the allowed origin config above is over permissive. Ideally it should only allow the domains that are permitted to communicate with the backend.

Project Cost

Hosting: We keep scripts and html in S3. The cost is so small as to be negligible.

Backend: Each pageview results in a single AWS Lambda call. So the cost will be ignorable small until I reach millions of pageviews per day.

Database: As Lambda Store charges per request, I am not worried while my website has low traffic. Once my website starts to gain millions of pageviews, I can move my database to reserved pricing to optimize cost. But for now, a free database should be enough.

Conclusion and what is next?

I should admit that this was my first project that is implemented with pure serverless technologies end to end. I completed the project in 4 days and working 1-2 hours each day. I was not familiar much with React so I spend some time learning it. In overall, I am quite happy with my experience, honestly it was even smoother than I expected. This is thanks to the serverless ecosystem and tools. The only thing that I can complain is the build time of the lambda functions. To test each change on my function I had to run sam build, this was slowing me down a little bit.

Anyway, finally, I have a simple web application that reads and writes data via lambda functions.
http://serverless-hackernews.s3-website-us-east-1.amazonaws.com/

What can be next?

  • I can add authentication using Auth0 or Cognito.
  • I can write an intelligent filter to reject spam and inappropriate content.
  • I can keep stats for the visited links.
  • I can improve the scoring and sorting the news items.
  • I can write a function to evict and clean the broken links.

Thanks for reading, waiting your comments and feedback.

Posted on Jun 3 by:

svenanderson profile

Sven Anderson

@svenanderson

Making the world serverless

Lambda Store

Lambda Store is the first the `serverless Redis` service. In this blog, Lambda Store engineering team shares their experiences on Cloud, AWS, Kubernetes, Redis and of course Lambda Store.

Discussion

markdown guide