Template Language for JSON data

Motivation

We are an integration platform and support 200+ integrations. We implemented these integrations using native Javascript code to transform incoming events to destination payload, so in summary, it is JSON data manipulation. Maintaining all these integrations is challenging, so we explored jsonata to write less code to transform JSON data. While this library is excellent, we still need to meet our performance needs. For example, JSONata parses the given template, creates an Abstract Syntax Tree (AST), and interprets the AST for the given input. Since we need to traverse the AST every time, it is slow, so we wanted to build a template engine that generates Javascript code so there will be less overhead during the runtime.

Overview

A library to process JSON data using a custom syntax based on javascript and jspath. We thank the jspath authors for their excellent work, as our library is an extension of the original library. We also want to thank IBM team for their work on jsonata, as we have taken several ideas from the library. You can also consider our library as an alternative to jsonata.

This library generates a javascript function code from the template and then uses the function to evaluate the JSON data. It outputs the javascript code in the following stages:

1. Lexing (Tokenization)
2. Parsing (AST Creation)
3. Translation (Code generation)

  flowchart TD;
      A[Code] --> B[Convert code to tokens];
      B --> C[Parse tokens to create Expressions];
      C --> D[Combine expressions to create statements];
      D --> E[Combine statements to create AST];
      E --> F[Translate AST to JS code]

Engine class abstracts the above steps and provides a convenient way to use the json templates to evaluate the inputs.

Getting started

npm install rudder-json-template-engine

const engine = new JsonTemplateEngine(`'Hello ' + .name`);
engine.evaluate({name: 'World'});

Features

Template is a set of statements and result the last statement is the output of the template.

Variables

const a = 1
let b = a + 2
a + b

Input and Bindings

Input refers to the JSON document we would like to process using a template. Bindings refer to additional data or functions we would provide to process the data efficiently.

Example:

• Template: "Hello " + (.name ?? $.defaultName)
• Evaluation: engine.evaluate({name: 'World'}, {defaultName: 'World'});
• {name: 'World'} is input.
  • ^.name refers to "name" property of the input. We can also use .name to refer the same. ^ always refers to the root of the input and . refers to current context. Refer the example for more clarity.
• {defaultName: 'World'} is bindings.
  • $.defaultName refers to "defaultName" property of the bindings. Refer the example for more clarity.

Arrays

let arr = [1, 2, 3, 4]
let a = arr[1, 2] // [2, 3]
let b = arr[0:2] // [1, 2]
let c = arr[-2:] // [3, 4]

Refer the example for more clarity.

Objects

let key = "some key"
// { "a": 1, "b": 2, "c": 3, "some key": 4 }
let obj = {a: 1, b: 2, c: 3, [key]: 4 } 
let a = obj["a"] // 1
let b = obj.a // 1
let c = obj{["a", "b"]} // { "a": 1, "b": 2}
let d = obj{~["a", "b"]} // { "c": 3, "some key": 4}

Refer the example for more clarity.

Functions

Normal functions

let fn = function(arg1, arg2){
  arg1 + arg2
}

The result of the last statement of function will be returned as result of the function. We can also use rest params (...args).

Lambda/Short functions

let fn = array.map(lambda 2 * ?0);

This function gets converted to:

let fn = array.map(function(args) {
  2 * args[0]
})

Lambda functions are short to express the intention and it is convenient sometimes.

Async functions

let fn = async function(arg1, arg2){
  const result = await doSomethingAsync(arg1, arg2)
  doSomethingSync(result)
}

Note: When we want to use async functions then we need to create template engine using JsonTemplateEngine.create. If you create a template this way then it will be created as an async function so we can await anywhere in the template.

let result = await doSomething(.a, .b)

Paths

Paths are used to access properties in input, bindings and variables.

Simple Paths

Simple paths support limited path features and get translated as direct property access statements in the generate javascript code.
a.b.c gets translated to a?.b?.c so they are very fast compared to Rich paths. Simple paths are ideal when we know the object structure.

Supported features:

• Simple Selectors
• Single Index Filters Refer the example for more clarity. #### Rich Paths Rich paths gets converted complex code to support different variations in the data.

If we use rich path for expression: a.b.c then it automatically following variations.

• [{"a": { "b": [{"c": 2}]}}]
• {"a": { "b": [{"c": 2}]}}
• {"a": [{ "b": [{"c": 2}]}]}
• Automatically handles selection from nested objects and arrays.

Simple selectors

let x = a.b.c;
let y = a."some key".c

Refer the example for more clarity.

Wildcard selectors

a.*.c // selects c from any direct property of a

Refer the example for more clarity.

Descendent selectors

// selects c from any child property of a
// a.b.c, a.b1.b2.c or a.b1.b2.b3.c
let x = a..c; 
let y = a.."some key";

Refer the example for more clarity.

Single Index or Property Filters

let x = a[0].c; 
let y = a[-1].c;  // selects last element from array
let z = a["some key"].c

Refer the example for more clarity.

Multi Indexes or Properties Filters

let x = a[0, 2, 5].c; 
let y = a["some key1", "some key2"].c;

Refer the example for more clarity.

Range filters

let x = a[2:5].c; 
let y = a[:-2].c; 
let z = a[2:].c; 

Object Property Filters

let x = obj{["a", "b"]};  // selects a and b
let y = obj{~["a", "b"]}; // selects all properties except a and b

Refer the example for more clarity.

Conditional or Object Filters

let x = obj{.a > 1};

Refer the example for more clarity.

Block expressions

let x = obj.({
  a: .a + 1,
  b: .b + 2
});
let x = obj.([.a+1, .b+2]);

Refer the example for more clarity.

Context Variables

.orders@order#idx.products.({
    name: .name,
    price: .price,
    orderNum: idx,
    orderId: order.id
})

Use context variables: @order and #idx, we can combine properties of orders and products together. Refer the example for more clarity.

Path Options

We can mention defaultPathType while creating engine instance.

// For using simple path as default path type
// a.b.c will be treated as simple path
JsonTemplateEngine.create(`a.b.c`, {defaultPathType: PathType.SIMPLE});
// For using rich path as default path type
// a.b.c will be treated as rich path
JsonTemplateEngine.create(`a.b.c`, {defaultPathType: PathType.RICH});

We can override the default path option using tags.

// Use ~s to treat a.b.c as simple path
~s a.b.c 
// Use ~r to treat a.b.c as rich path
~r a.b.c

Note: Rich paths are slower compare to the simple paths.

Compile time expressions

Compile time expressions are evaluated during compilation phase using compileTimeBindings option.

// {{$.a.b.c}} gets translated to 1 and
// final translated code will be "let a = 1;"
JsonTemplateEngine.create(`let a = {{$.a.b.c}};`, {
  compileTimeBindings: {
    a: {
      b: {
        c: 1
      }
    }
  }
});

We can use compile time expressions to generate a template and then recompile it as expression. Refer the example.

Conclusion

We have created a simple language to process JSON data efficiently and made it open-source. It was fun building a language, and I hope you have learned something new today.