Write a simple Web Framework in Nim language from scratch

What is “from scratch”?

Sometimes we are told “Don’t reinvent wheel”. It may be right when we want to write reliable and stable software. We should rely on mature framework instead of write our own framework. However sometimes life is too boring if we don’t challenge ourselves. If we want to learn how web frame works, we should write it by ourselves. “I see and I remember. I do and I understand.”

Ok, you maybe want to write your own web framework “from scratch”. Well, now you have two problems :-). What is “from scratch” ? Does it mean that we need to reinvent “semiconductor”. Of course not. According to what you want to learn, you can “stand on the shoulders of giants”. If you want to learn low level module, you can write it beginning from TCP or HTTP Protocol. And you can write a HTTP server and make a web framework based on it.

Regarding this article, we will rely on others’ HTTP server, because I just want to know how a web framework works.

What is web framework?

Web framework is also called web application framework. It is built on the HTTP server which is built on HTTP protocol. Generally, HTTP server is only used to accept request from client and respond to client. Web framework supplies more useful and powerful utilities such as data validation or user authentication.

For example, when you shop online, you click on purchase button. Your browser will send a request to web server. Web server will parse request into data structure which web framework can understand. Web framework will verify your identity, add this product or user info into database and send a request to deduct money from your credit card and so on.

What we need mostly?

Nim programming language

Standard libraries: asyncdispatch and asynchttpserver.

Why Nim?

Nim is a statically typed compiled systems programming language. It has intuitive and clean syntax. Nim is efficient, expressive and elegant. Choose Nim and enjoy your life!

I love it because of three reasons:

• Indent and elegant syntax
• Static Type
• High performance

Simple usage of asynchttpserver

Nim has a built-in asynchronous HTTP server namely asynchttpserver. You don’t need to install it, just type import asynchttpserver to use it.

HTTP server helps us transfer contents(like HTML, Json and Text) to our client(like browser or curl). It will parse request into seq or tables in Nim language which our web framework can understand.

Let’s look at the code.

Http200 is a status code which tells client that everything is ok. “Hello World” is the text we responds to the client with. We also need to respond to the client with HTTP headers.

Now run the code below and enter localhost:8080 in your browser, you will get Hello World on the screen.

# nim c -r thisfile.nim
import asynchttpserver, asyncdispatch

var server = newAsyncHttpServer()
proc cb(req: Request) {.async.} =
  await req.respond(Http200, "Hello World", newHttpHeaders())

waitFor server.serve(Port(8080), cb)

Let’s look at request headers and response headers to get more intuitions.

request headers

GET / HTTP/1.1       # start line
Host: 127.0.0.1:8080 # request headers
......

response headers

HTTP/1.1 200 OK      # start line
Content-Length: 11   # response headers

Now let’s extend out framework!

Serve static files

Sometimes we want to serve static files like CSS, HTML, Images and so on. It is wise to use Nginx. But for small applications, we just want to use ourselves’ framework.

In Nim, you only need to prepare strings and send strings to client. If you want to send HTML files, you read it and send it by socket. If you want to send Json text, first convert/dump it to strings and then send it by socket.

Now follow three steps:

Firstly, judge whether the file exists. Secondly judge whether we have access to files. Finally judge whether we can open files.

Let’s look at Pseudo code. resp followed by contents we give to respond proc in asynchttpserver.

note: await must be used in functions

import asyncdispatch, asynchttpserver, os


proc serveStaticFile*(req: Request, dir, filename: string) {.async.} =
  # exist -> have access -> can open
  let path = dir / filename

  # whether exists file
  if not existsFile(path):
    await req.respond(Http404, "File doesn't exist.", newHttpHeaders())

  # whether has access to file
  var filePermission = getFilePermissions(path)
  if fpOthersRead notin filePermission:
    await req.respond(Http403, "You have no access to the file.", newHttpHeaders())

  # whether can open file
  try:
    let content = readFile(path)
    await req.respond(Http200, content, newHttpHeaders())
  except IOError:
    await req.respond(Http404, "404 Not Found.", newHttpHeaders())

Basic Route

Route is used to map URL to corresponding proc which responds to client with contents.

For example:

This example demonstrates that web framework execute some actions such as fetch info from database, login, return HTML and so on according to corresponding HTTP method and URL.

• get 127.0.0.1:8080/hello -> send email
• get 127.0.0.1:8080/home -> fetch info from Database

import httpcore


proc findHandler*(httpMethod: HttpMethod, path: string) =
  case httpMethod
  of HttpGet:
    case path
    of "/hello":
      echo "get 127.0.0.1:8080/hello"
      # sendEmail()
    of "/home":
      echo "get 127.0.0.1:8080/home"
      # fetchInfoFromDatabase()
    else:
      discard
  of HttpPost:
    case path
    of "/hello":
      echo "post 127.0.0.1:8080/hello"
      # login()
    of "/home":
      echo "post 127.0.0.1:8080/home"
      # returnHome()
    else:
      discard
  else:
    discard

Use Hash Map for static route

If client requests www.example.com/login, our application will look up hash table to find corresponding Handler. Handler is the proc which handles request from client and generates response.

It is fast to look up URL in hash table.

type
  Router* = Table[URL, procHandler]

Dynamic route

Sometimes we need dynamic URL to meet our demands. For example, we use only one procHandler to handle login operations for different customers. So every customer will have different login URLs like www.example.com/login/1, www.example.com/login/2 and so on.

We need URL match like www.example.com/login/{id} to catch different ids.

First we split URL with / , turn www.example.com/login/1 into @[“www.example.com”, “login”, “1”] . Then we iterate hash table to decide whether URL match patterns.

Let’s look at Pseudo code.

let routeList = route.split("/")
# iterate all URL and procHandler pairs
let pathList = iterate(HandlerTable)
for idx in 0 ..< pathList.len:
  # if match continue 
  # www.example.com => www.example.com
  # login => login
  if pathList[idx] == routeList[idx]:
    continue

  # match {id} => 2
  if routeList[idx].startsWith("{"):
    let key = routeList[idx]
    if key.len <= 2:
      raise newException(RouteError, "{} shouldn't be empty!")
    let
      params = key[1 ..< ^1]

Regex Route

We can use seq to store regex route.

/post(?P<num>[\d]+) => /post521 or /post1314 and gets matched parameters such as num = 521 or num = 1314.

type
  ReRouter* = seq[(URL, ProcHandler)]

We just iterate URL and procHandler in Reroute to match request URL.

Let’s look at Pseudo code.

# find regex route
for (URL, ProcHandler) in reRouter:
  if path.httpMethod != URL.httpMethod:
    continue
  var m: RegexMatch

  # save matched params like id = 2
  if URL.route.match(path.route, m):
    for name in m.groupNames():
      pathParams[name] = m.groupFirstCapture(name, URL.route)

Cookie

HTTP protocol is stateless. But we use plain Cookie to save user information.

Cookie is HTTP headers. It can carry user insensitive information. When you want to carry sensitive information, make sure encrypt it using Encryption Algorithm such as Sha-256, Sha-512 and so on. Only use Cookie with HTTPS to prevent it from man-in-the-middle attacks.

This is what cookie looks like. It is in the form of name-value pair. Different pairs are separated by semicolon.

Cookie: username=flywind; age=21

Parse CookieJar

We use StringTable to store different name-value pairs.

type
  CookieJar* = object
    data: StringTableRef

proc parse*(cookieJar: var CookieJar, text: string) {.inline.} =
  var 
    pos = 0
    name, value: string
  while true:
    pos += skipWhile(text, {' ', '\t'}, pos)
    # name = username
    pos += parseUntil(text, name, '=', pos)
    if pos >= text.len:
      break
    inc(pos) # skip '='
    # value = flywind
    pos += parseUntil(text, value, ';', pos)
    # username = flywind
    cookieJar[name] = move(value)
    if pos >= text.len:
      break
    inc(pos) # skip ';'

Also we want to set cookie.

import options, times, strtabs, parseutils, strutils


type
  SameSite* {.pure.} = enum
    None, Lax, Strict

  Cookie* = object
    name*, value*: string # root => admin
    expires*: string
    maxAge*: Option[int]
    domain*: string
    path*: string
    secure*: bool
    httpOnly*: bool
    sameSite*: SameSite

Set cookie

proc setCookie*(cookie: Cookie): string =
  result.add cookie.name & "=" & cookie.value
  if cookie.domain.strip.len != 0:
    result.add("; Domain=" & cookie.domain)
  if cookie.path.strip.len != 0:
    result.add("; Path=" & cookie.path)
  if cookie.maxAge.isSome:
    result.add("; Max-Age=" & $cookie.maxAge.get())
  if cookie.expires.strip.len != 0:
    result.add("; Expires=" & cookie.expires)
  if cookie.secure:
    result.add("; Secure")
  if cookie.httpOnly:
    result.add("; HttpOnly")
  if cookie.sameSite != None:
    result.add("; SameSite=" & $cookie.sameSite)

Middleware

Middleware is used to execute some operations before procHandler and after procHandler. For example, we want to verify user identity, we can use middleware to do this work.

Let’s look at Pseudo code.

proc verify(user: User, next: Handler) =
  # check whether user is valid
  if not user.isValid:
    resp 404, "You have no access to this URL."
    return

  # call next middleware or procHandler
  next()

  # If user is valid, print "Welcome" message
  resp "Welcome, " & $user 

There are two ways to implement middleware, one is based on function call, the other is based on hook.

Let’s look at hook way

We set before and after function type for every Middleware object.

type
  Middleware = object
    before: proc()
    after: proc()

In the lifetime of application, assuming that we have two middlewares. So the order of execution is:

m1.before -> m2.before -> hello -> m2.after -> m1.after

Let’s look at Pseudo code.

proc hello() =
  resp "Hello"

var app = Application()
var m1 = Middleware(...)
var m2 = Middleware(...)
app.addRoute(procHandler = hello, middlwares = [m1, m2])

Let’s look at Function call way

We use seq to store middlewares. await next is used to call next middleware or procHandler.

We have size variable to get current middleware. In the beginning, size is 0 and we get first middleware. Now we begin to execute the first middleware. When we encounter await next. size will increase by one, and we will call the second middleware. Finally we call the last middleware, and execute the rest program.

m1 -> m1.next -> m2 -> m2.next -> hello -> m2 -> m1

Let’s look at Pseudo code.

type
  Middlewares* = seq[procHandler]

proc httpRedirectMiddleWare*() =
  case request.scheme
  of "http":
    setScheme(request, "https")
  of "ws":
    setScheme(request, "wss")
  else:
    return

  # Will call next middleware or procHandler
  await next()

  response.code = Http307

Exception Handler

We can map HTTP status codes to user-defined exception Handler.

Sometimes, we want to associate HTTP status code to unified pages for example custom 404 pages.

type
  ErrorHandlerTable* = Table[HttpCode, ErrorHandler]

proc default404Handler*() =
  response.body = errorPage("404 Not Found!", PrologueVersion)

app.errorHandlerTable.add(Http404, default404Handler)

Let’s look at Pseudo code.

if response.code in app.errorHandlerTable:
  await (app.errorHandlerTable[response.code])()

More part

Regarding web framework, there are more parts than what I mentioned, you can checkout

• Web framework Prologue

• Network infrastructure Netkit