There’s been many times when I’ve just wanted to add a simple JSON endpoint to an app to, expose a service, or process webhook events, without the overhead of a full framework. Let’s see how easy it is to build a production ready endpoint with Plug, using Erlang’s Cowboy HTTP server.
Plug Is:
- A specification for composable modules between web applications
- Connection adapters for different web servers in the Erlang VM
If you’re coming from Ruby/Rails, think Rack, from Node, think Express, et al. Of course the concepts of these libraries are similar on the surface, they are unique in their own rights.
Cowboy Is:
A small, fast and modern HTTP server for Erlang/OTP
Additionally, it’s a fault tolerant “server for the modern web” supporting HTTP/2, providing a suite of handlers for Websockets and interfaces for long-lived connections. Without going into too much more detail, it’s safe to say, it’s an acceptable choice for production. Consult the docs for more info.
Poison Is:
A JSON library for Elixir focusing on wicked-fast speed without sacrificing simplicity, completeness, or correctness.
In other words, it’s a super fast, reliable JSON parsing library.
Building the Endpoint
With the short definitions out of the way, let’s build an endpoint to process incoming webhook events. Now, we want this to be “production ready”, what does that mean for our use case?
- Fault tolerant: Always available. Can never go down (at least not easily :)
- Easily configurable: Can be deployed to any environment
- Well tested: Give us confidence in what we’re shipping
We do have a very simple use case for this, it’s a good idea to understand your own requirements before selecting tools and investing time into a solution.
1. Create a new, supervised, Elixir app:
$ mix new webhook_processor --sup
$ cd webhook_processor
--sup
will create an app suitable for use as an OTP application. OTP and supervision will give us our #1 requirement from above. Our server will be supervised and restarted automatically in the event of a crash, while the server may crash the Erlang VM should not (at least not easily :).
2. Add Plug, Cowboy, and Poison as dependencies
# ./mix.exs
defmodule WebhookProcessor.MixProject do
use Mix.Project
def project do
[
app: :webhook_processor,
version: "0.1.0",
elixir: "~> 1.7",
start_permanent: Mix.env() == :prod,
deps: deps()
]
end
# Run "mix help compile.app" to learn about applications.
def application do
[
# Add :plug_cowboy to extra_applications
extra_applications: [:logger, :plug_cowboy],
mod: {WebhookProcessor.Application, []}
]
end
# Run "mix help deps" to learn about dependencies.
defp deps do
[
{:plug_cowboy, "~> 2.0"}, # This will pull in Plug AND Cowboy
{:poison, "~> 3.1"} # Latest version as of this writing
]
end
end
Couple notes here, we added plug_cowboy
(in deps
) as a single dependency for Plug AND Cowboy. We need to add :plug_cowboy
to the extra_applications
list (in application
) as well.
3. Mix deps.get
$ mix deps.get
4. Implement application.ex
# ./lib/webhook_processor/application.ex
defmodule WebhookProcessor.Application do
@moduledoc "OTP Application specification for WebhookProcessor"
use Application
def start(_type, _args) do
# List all child processes to be supervised
children = [
# Use Plug.Cowboy.child_spec/3 to register our endpoint as a plug
Plug.Cowboy.child_spec(
scheme: :http,
plug: WebhookProcessor.Endpoint,
options: [port: 4001]
)
]
# See https://hexdocs.pm/elixir/Supervisor.html
# for other strategies and supported options
opts = [strategy: :one_for_one, name: WebhookProcessor.Supervisor]
Supervisor.start_link(children, opts)
end
end
5. Implement WebhookProcessor.Endpoint
# ./lib/webhook_processor/endpoint.ex
defmodule WebhookProcessor.Endpoint do
@moduledoc """
A Plug responsible for logging request info, parsing request body's as JSON,
matching routes, and dispatching responses.
"""
use Plug.Router
# This module is a Plug, that also implements it's own plug pipeline, below:
# Using Plug.Logger for logging request information
plug(Plug.Logger)
# responsible for matching routes
plug(:match)
# Using Poison for JSON decoding
# Note, order of plugs is important, by placing this _after_ the 'match' plug,
# we will only parse the request AFTER there is a route match.
plug(Plug.Parsers, parsers: [:json], json_decoder: Poison)
# responsible for dispatching responses
plug(:dispatch)
# A simple route to test that the server is up
# Note, all routes must return a connection as per the Plug spec.
get "/ping" do
send_resp(conn, 200, "pong!")
end
# Handle incoming events, if the payload is the right shape, process the
# events, otherwise return an error.
post "/events" do
{status, body} =
case conn.body_params do
%{"events" => events} -> {200, process_events(events)}
_ -> {422, missing_events()}
end
send_resp(conn, status, body)
end
defp process_events(events) when is_list(events) do
# Do some processing on a list of events
Poison.encode!(%{response: "Received Events!"})
end
defp process_events(_) do
# If we can't process anything, let them know :)
Poison.encode!(%{response: "Please Send Some Events!"})
end
defp missing_events do
Poison.encode!(%{error: "Expected Payload: { 'events': [...] }"})
end
# A catchall route, 'match' will match no matter the request method,
# so a response is always returned, even if there is no route to match.
match _ do
send_resp(conn, 404, "oops... Nothing here :(")
end
end
This looks like a lot, but most of this file is just some helpful comments. The gist is, we are using the macros, get and post, from Plug.Router
to generate our routes. This module is a plug itself, and it defines its own plug pipeline. Note, match
and dispatch
are required in order for us to handle requests and dispatch responses. Pipeline is a key concept here, as the order of plugs determines the order of operations. Notice that match is before we define our parser, this means we will not parse anything unless there is a route match. IF the order was reversed, we’d be trying to parse requests regardless of routes matching. Refer to the docs on Plug.Router
for more info.
6. Make the Endpoint Configurable
# ./lib/webhook_processor/application.ex
defmodule WebhookProcessor.Application do
@moduledoc "OTP Application specification for WebhookProcessor"
use Application
def start(_type, _args) do
# List all child processes to be supervised
children = [
# Use Plug.Cowboy.child_spec/3 to register our endpoint as a plug
Plug.Cowboy.child_spec(
scheme: :http,
plug: WebhookProcessor.Endpoint,
# Set the port per environment, see ./config/MIX_ENV.exs
options: [port: Application.get_env(:webhook_processor, :port)]
)
]
# See https://hexdocs.pm/elixir/Supervisor.html
# for other strategies and supported options
opts = [strategy: :one_for_one, name: WebhookProcessor.Supervisor]
Supervisor.start_link(children, opts)
end
end
We’ve swapped out the hard coded port value of the Cowboy options for an environment variable, this will allow us to run the webhook processor in any environment we need to. Finally, create a config file for each MIX_ENV
you need:
#./config/config.exs
# This file is responsible for configuring your application
# and its dependencies with the aid of the Mix.Config module.
use Mix.Config
import_config "#{Mix.env()}.exs"
-------------------
# ./config/dev.exs
use Mix.Config
config :webhook_processor, port: 4001
-------------------
# ./config/test.exs
use Mix.Config
config :webhook_processor, port: 4002
-------------------
# ./config/prod.exs
use Mix.Config
config :webhook_processor, port: 80
7. Test
# ./test/webhook_processor/endpoint_test.exs
defmodule WebhookProcessor.EndpointTest do
use ExUnit.Case, async: true
use Plug.Test
@opts WebhookProcessor.Endpoint.init([])
test "it returns pong" do
# Create a test connection
conn = conn(:get, "/ping")
# Invoke the plug
conn = WebhookProcessor.Endpoint.call(conn, @opts)
# Assert the response and status
assert conn.state == :sent
assert conn.status == 200
assert conn.resp_body == "pong!"
end
test "it returns 200 with a valid payload" do
# Create a test connection
conn = conn(:post, "/events", %{events: [%{}]})
# Invoke the plug
conn = WebhookProcessor.Endpoint.call(conn, @opts)
# Assert the response
assert conn.status == 200
end
test "it returns 422 with an invalid payload" do
# Create a test connection
conn = conn(:post, "/events", %{})
# Invoke the plug
conn = WebhookProcessor.Endpoint.call(conn, @opts)
# Assert the response
assert conn.status == 422
end
test "it returns 404 when no route matches" do
# Create a test connection
conn = conn(:get, "/fail")
# Invoke the plug
conn = WebhookProcessor.Endpoint.call(conn, @opts)
# Assert the response
assert conn.status == 404
end
end
These tests are pretty simple, but they confirm our server is working as expected. One could argue that the only thing we should care about from these tests are the response codes, not the side effects that happen when events are processed. Always test UP TO the boundaries of your module, never beyond, unless you are writing an integration style test.
Conclusion
With very little effort we’ve built a small but mighty endpoint. Thanks to Cowboy, you should be able to serve more connections from one server than you’ll likely ever need, so let’s add low cost to the list of benefits as well.
What about deployment? Let's walk through building releases and deploying to AWS:
- Building Releases with Docker & Mix
- Terraforming an AWS EC2 Instance
- Deploying Releases with Ansible
As always, the code is available on GitHub: https://github.com/jonlunsford/webhook_processor
Top comments (14)
Great walkthrough of using Plug, Cowboy and Poison altogether! I've actually recently wrote a deep-dive on Plug.Cowboy to understand a bit more how Cowboy handled incoming HTTP requests, which I found super interesting: charta.dev/tours/plug_cowboy
thank you for throwing in the link here ... was really looking for something of sorts
Awesome stuff!
Very interesting stuff! I use Phoenix and I did know that its router is a set of macros for the Endpoint. This minimal implementation is very cool and useful to know :-)
On a side note: have you tried Jason instead of Poison? I've started using it, pretty the same API of Poison, also, if I recall correctly, it's the default encoder for Ecto 3.x.
Curious to see the deployment part.
Thank you so much for this. It's exactly what I needed.
Thanks :) Just ran my first Elixir endpoint
To try it, just use
iex -S mix
and go to your favorite browser :)mix run
doesn't work but I'll figure it out !mix run --no-halt
is what you need. Would recommend postman though rather than a browser otherwise, you get a load of warnings in the logger about favicons.Thanks for the article, looking forward to the followup on deployment!
Thanks for reading! Yeah, I will be walking through a few methods of deployment:
Thank you for this! Would really like to see a quick tutorial on deploying as well!
hey, nice and useful. would be curious about how you deploy also!
Very interesting article! I was wondering if you know how to add support for CORS. That would add the cherry the cake!
Fun!
🙌