DEV Community

Cover image for JSON validation for Cowboy REST API - Mimic Celebrate Joi in Elixir
Lionel Marco
Lionel Marco

Posted on • Updated on

JSON validation for Cowboy REST API - Mimic Celebrate Joi in Elixir

When we work in a REST API one of the first things to do is validate incomming data.

Suddenly I must to rewrite a REST API from Javascript to Elixir and find that some of the existing library does not feed my basic needs of simplicity and less code possible.
All of us who have a Node-Express background, have used Celebrate and Joi in order to validate the presence of key, their data type and their stored value.

In this article I will show you how to implement in Elixir a schema validator, trying to mimic a (minimal) functionality of Celebrate Joi.

Table of Contents

1) Plugs and Midlewares

There is a common architecture between Node-Express and Elixir-Cowboy, Express have middlewares and Cowboy have Plugs.
In both cases these are used for routing, body parsing, user authentication verification, cross-origin resource sharing, data logging, data validation, metrics request, etc..

How the documentation say :"An Express application is essentially a series of middleware function calls." Using middlewars

For example: A classic Node-Express midlewares chain

//file: app_router.js
// import boiler plate ...
app.use(cors(cors_configuration)); //CORS
app.use(express.json());//Json body parsing

app.use(isAuth);//user jwt authentication verification

app.use(function (err, req, res, next) { //  error handling
  if (err.name === "UnauthorizedError") {
    res.status(401).send("invalid token...");
  } else {
    next(err);
  }
});

app.use(attachUserId);// add extra information about the current user

app.use((req, res, next) => { // some debug 
  console.log("Method",req.method);  
  console.log("Url", req.originalUrl);
  console.log("Body:",req.body);  
  return next();
});


Enter fullscreen mode Exit fullscreen mode

The same happens in Elixir-Cowboy, where plugs are chained one after other, like this:

# file: app_router.ex
# use/import boiler plate ...

  plug Corsica,  cors_configuration # CORS  
  plug :match  
    plug Plug.Telemetry, telemetry_configuration #  metrics request  
    # body parsing
    plug Plug.Parsers,    parsers: [:json],    pass: ["application/json"],    json_decoder: Jason

    # Json validation
    plug Plug.RouterValidatorschema, on_error: &__MODULE__.on_error_fn/2

    # user jwt authentication verification
    plug Plug.Authenticator, on_error: &__MODULE__.on_error_fn/2
    # add extra information about the current user
    plug Plug.AttachUserId, on_error: &__MODULE__.on_error_fn/2
    # some debug 
    plug Plug.PrintUrlMethod

  plug :dispatch


Enter fullscreen mode Exit fullscreen mode

Celebrate is a middleware, the equivalent in Elixir is a Plug, let start writing a plug:

The next link is the started point to understand and learn about how to write a plug, I recommend read it and then come here: The Plug Specification

There are two ways to define it: function plugs and module plugs, I will choose the module plugs way.

2) Route-Schema Validation

Has programers we want to write the minimal possible lines of code. Bassically reduce all to a schema definition and then apply in our route.

Schema:

photo_book_schema = [
                      qty: {:number, 1, 200},
                      hardcover: {:boolean},
                      colors: {:options, ["bw", "color", "sepia"]},
                      client_name: {:string, 6, 20},
                      client_phone: {:string, 8, 25, ~r/^[\d\-()\s]+$/}
                    ]
Enter fullscreen mode Exit fullscreen mode

Route:

plug(:match)
plug(Plug.RouterValidatorschema, on_error: &__MODULE__.on_error_fn/2)
plug(:dispatch)

post "/create/", private: %{validate: photo_book_schema} do
  status = Orders.create(conn.body_params)
  {:ok, str} = Jason.encode(%{status: status})
  send_resp(conn |> put_resp_content_type("application/json"), str)
end

def on_error_fn(conn, errors, status \\ 422) do
  IO.inspect(errors, label: "errors")
  {:ok, str} = Jason.encode(%{status: "fail", errors: errors})
  send_resp(conn |> put_resp_content_type("application/json"), str) |> halt()
end

Enter fullscreen mode Exit fullscreen mode

Anxious for the code ?
Before to go deeper in the code, we learn a bite about Function Clause Matching.

2) Function Clause Matching

One of the powerfull tools of Elixir is pattern matching. It is applied in many places, thanks to that, we write less lines of code compared with another languages.

In this case is applied in function clauses, Elixir let us to write multiple definitions for the same function, every definitions is called a clause.

See the next example:

defmodule Discount do
  def calculate(type, price) do
  case type do
      :normal -> price * 1
      :high   -> price * 0.8
      :low    -> price * 0.9      
  end  
  end  
end
Enter fullscreen mode Exit fullscreen mode

Instead we can write:

defmodule Discount do
  def calculate(:normal, price), do: price
  def calculate(:high, price), do: price * 0.8
  def calculate(:low, price), do: price  * 0.9  
end
Enter fullscreen mode Exit fullscreen mode

Where the "case" statement is moved outside of the function and the virtual machine take the decision of which must to apply.All definition refer to the same function "calculate" with arity 2.

2.1) Tips.

Some tips when use function clause matching.

Group:

Always group clauses of the same function together,

Don't mix with others:


defmodule FooBar do
  def foo(:a), do: 'a'
  def foo(:b), do: 'b'
  def bar(arg), do: IO.puts arg
  def foo(:c), do: 'c'
  def foo(arg), do: IO.puts arg
end

Enter fullscreen mode Exit fullscreen mode

Will receive a compiler warning:

warning: clauses with the same name and arity (number of arguments) should be grouped together, "def foo/1" was previously defined.
Enter fullscreen mode Exit fullscreen mode

Correct way:


defmodule FooBar do
  def foo(:a), do: 'a'
  def foo(:b), do: 'b'
  def foo(:c), do: 'c'
  def foo(arg), do: IO.puts arg
  def bar(arg), do: IO.puts arg
end

Enter fullscreen mode Exit fullscreen mode

Order:

The declaration order really matter:
Don't do:


defmodule FooBar do
  def foo(arg), do: arg
  def foo(:b), do: 'b'  
  def foo(:c), do: 'c'
end
Enter fullscreen mode Exit fullscreen mode

Will receive a compiler warning:

warning: this clause for foo/1 cannot match because a previous clause always matches
Enter fullscreen mode Exit fullscreen mode

Correct way, sort for selective weight:

defmodule FooBar do  
  def foo(:b), do: 'b'  
  def foo(:c), do: 'c'
  def foo(arg), do: arg
end

Enter fullscreen mode Exit fullscreen mode

Match all:

Alway write a final match all clause.

Sometimes we receive a inexpectable argument value and then a error is triggered:
(FunctionClauseError) no function clause matching

Don't do:


defmodule FooBar do
  def foo(:b), do: 'b'  
  def foo(:c), do: 'c'
end
Enter fullscreen mode Exit fullscreen mode

Correct way:


defmodule FooBar do
  def foo(:b), do: 'b'  
  def foo(:c), do: 'c'
  def foo(_), do: "Something happen"
end
Enter fullscreen mode Exit fullscreen mode

4) Schema Validation

The Plug.Router provide a way to passing data between routes and plugs

The Plug.conn struct

A analogy to better undertand the Plug.conn and the plug chain is think in a train locomotive with wagons, when every plug is a station where we load or unload data and the wagons are the fields of the struct.

The Plug.conn struct have a field named private, it will be seted with a key and a value.


  post "/create/" , private: %{validate: photo_book_schema} do  
     .....
  end
Enter fullscreen mode Exit fullscreen mode

When the plug find the validate key inside the private field, the validation process start, calling the function validate_schema()


defmodule Plug.RouterValidatorSchema do

  def init(opts), do: opts

    def call(conn, opts) do
    case conn.private[:validate] do
      nil -> conn
      schema -> validate_schema(conn,schema, opts[:on_error])
    end
  end
Enter fullscreen mode Exit fullscreen mode

The validate_schema() walk over the schema list and validate the keys inside conn.body_params

If all validations are successful returns an empty map.
Otherwise when validation fail, call the given on_error callback with a list of the collected errors.
The errors list have the shape [ %{"key1"=> "error"}, %{"key2"=> "error"}]


  defp validate_schema(conn, schema, on_error) do
  errors = Enum.reduce(schema, [], validate_key(conn.body_params))

  if Enum.empty?(errors) do
    conn
  else
    on_error.(conn, %{errors: errors})
  end
end

defp validate_key(data) do
  fn {key, test}, errors ->
    skey = Atom.to_string(key)
    cond do
      not Map.has_key?(data, skey) -> [%{skey => "missing key"} | errors]      
      is_nil(data[skey]) -> [%{skey => "value is nil"} | errors] # or just return errors if value can accept nil
      true ->
        case is_t(test, data[skey]) do
          :ok -> errors
          {:invalid, reason} -> [%{skey => reason} | errors]
        end
    end
  end
end

Enter fullscreen mode Exit fullscreen mode

The validate_key(data) return a parameterized function that be in charge of apply every test to the key associated data.
The is_t(test,value) will perform the test and here is where we take advantage of function clause matching, the function test for boolean, number, string, string patterns, options, and is very easy add a new test for any desired data type.


defp is_t({:boolean}, value) do
  if (not is_boolean(value)) do
    {:invalid, "not is boolean"}
  else
   :ok
  end
 end

 defp is_t({:number}, value) do
  if (not is_number(value)) do
    {:invalid, "not is number"}
  else
   :ok
  end
end

defp is_t({:number, vmin, vmax}, value) do
  cond do
    not is_number(value)   ->  {:invalid, "not is number"}
    value <= vmin ->  {:invalid, "min value"}
    value > vmax ->  {:invalid, "max value"}
    true ->:ok
  end
end


 defp is_t({:string}, value) do
    if is_bitstring(value) do
      :ok
    else
      {:invalid, "not is string"}
    end
 end

 defp is_t({:string, lmin, lmax}, value) do
     cond do
       not is_bitstring(value)   ->  {:invalid, "not is string"}
       String.length(value) <= lmin ->  {:invalid, "min length"}
       String.length(value) > lmax ->  {:invalid, "max length"}
       true ->:ok
     end
 end

 defp is_t({:string, lmin, lmax, reg}, value) do
  cond do
    not is_bitstring(value)   ->  {:invalid, "not is string"}
    String.length(value) <= lmin ->  {:invalid, "min length"}
    String.length(value) > lmax ->  {:invalid, "max length"}
    not Regex.match?(reg,value) -> {:invalid, "regex not match"}
    true ->:ok
  end
end

defp is_t({:options, alloweds}, value) do
  cond do
    value not in alloweds -> {:invalid, "not allowed"}
    true ->:ok
  end
end

# ---------------------------
# Final match :
# ---------------------------
defp is_t(_value,_test) do
  {:invalid, "test match fail"}
end


Enter fullscreen mode Exit fullscreen mode

Top comments (0)