Hey !

Third time that i come to wrote an article on this platform (yay ✨) ! This time i will introduce the project i has been working on since almost three years 😍 (with my own words/heart .. sorry if this is not clear enough 😆).

⚠️ The project is not finished (we are working hard for the first MVP release).

Introduction

Genesis

I started my professional career by working on monitoring products. My job was to implement custom scripts and probes (in lua, perl5, JAVA and C) to meet customer needs that could not always be implemented by the product editor.

My job has always been more or less to be able to meet any needs and this is what I like about being a developer with a monitoring expertise (this force you to always discover new horizons and fight with new technical and human challenge 💪).

For example, I recently worked with SNMP to monitore some network routers. It was a fun experience to discover this new protocol.

I have always approached monitoring with a vision of a developer / integrator (and i instilled this vision to SlimIO).

The idea

The idea of developing SlimIO came to me from the different problems of my clients but also from the difficulty of implementing new functionalities (or changing the behavior of existing functionalities) in products. At least this is the reason that pushed me to craft SlimIO (but I'm not the only part of the final project DNA.).

Most products have been created with the need of harvesting a bunch of given raw metrics with a Configuration Item like infrastructure or network metrology products.

But the real world is not about answering linear need and from one client to another these requirements are very different (and clients often end up with many products and third-party solutions/custom codes to achieve the goal they want.).

This is why our goal was to design a high level monitoring product that anybody can configure and extend with ease (while staying neutral as possible on many complicated technical subjects). A good example of this is the question of pulling the data from a concentrator or pushing it from the agent... Our product has been designed to be able to achieve both pull and push.

It's always difficult and funny to explain it but SlimIO does not focus at all on collecting or storing the metric 🙊 (Even if we consider these subjects as very important... Things like Modularity, neutrality and accessibility are in our eyes more important subjects 😎).

I thought and built SlimIO to be a foundation / paradigms that will give you the power and the tools to craft the monitoring that your team/company REALLY NEED. Sure we are working on our own "core" components but your always free to go with your own ways of doing it (by crafting new core addons yourself for example).

Technologies we use

The product is mainly written in JavaScript (Node.js) and C++.

Why doing it with Node.js ? People often think that we are crazy for choosing that path for an "On-premise" monitoring product. There is a part of ideology in this choice because we believe that the JavaScript ecosystem match at 100% the idea of an accessible and extensible product.

Some of the advantages of chosing JavaScript:

• The event-loop pattern works well with metrology concepts.
• Match the desire to construct a SaaS solution in the future.
• Allow us to tests the whole product more easily (already more than five hundreds tests across the whole projects).
• A big ecosystem available for those who want to craft new addons (the community or even our team ¯_(ツ)_/¯).
• Get introduced to how to code an Addon with JavaScript easily.
• Allow us to sandbox and hot-load addons.

However metrics are recolted with low-level binding written in C/C++ (and maybe even some in Rust in the future).

I have the feeling and the desire to build which would one day potentially be a symbiosis between Rust and JavaScript (I have the idea of rewriting some of the core components in pure Rust and even allow in the future pure Rust addons).

Our governance list some of the current product weakness. In terms of performance I think we will stay very similar to product like netdata.

A lot more to read on the governance repo https://github.com/SlimIO/Governance

SlimIO (concepts)

Agent

A SlimIO agent is an addition between a core and many containers (that we call addons).

One agent can be a complete monitoring solution with no need of centralization (perfectly designed to be Standalone). The role of an agent is defined by each addon that compose it. For example a concentrator is just a SlimIO Agent with a concentrator addon 😅.

The concept of modularity is deeply rooted in the product DNA. Everything we do, you do, would be the result of an Addon.

Our team work on addons that we call built-in because they bring features related to metrics and alarms management (and allow you to develop addons without having to worry about things not related to your own business logic.).

This mean that you can craft an agent as you were playing lego. If a part is missing or does not suit you, just replace it. More docs and links on the Agent github.

Our team is open to any debate concerning the neutrality of the core components we created! We think in the interest of the greatest number, but we know that more specific needs will require new core addon in the future.

Current "built-in" addons are:

• Gate: An abstraction of the core but as an addon (avoid implementing exceptions or behaviors to the core itself).
• Socket: Addon crafted to achieve fast TCP communication.
• Events: Addon that will store the data (metrics, alarms, events...).
• Aggregator: Aggregate stored metrics.
• Alerting: Everything related to the management of alarms (storm, alerts on time threshold etc).

👀 An HTTP/2 addon will be added in the future to consume the data as API (Like Prometheus).

Addons

An addon is a container that will bring you the tools to interact with the product (other addons) and embed your own business logic/algorithms to bring new metrics, alarms and events to the product.

Take, for example the SlimIO cpu addon (follow the link for the full code on github).

import os from "os";
import Units from "@slimio/units";
import metrics from "@slimio/metrics";
import Addon from "@slimio/addon";

const CPU = new Addon("cpu", { version: "1.1.0", description: "CPU Addon" });

const { Entity, MetricIdentityCard, sendRawQoS } = metrics(CPU);

// Declare Entities and MIC
const CPU_E = new Entity("cpu", {
    description: "Central Processing Unit"
});
new MetricIdentityCard("TOTAL", { unit: Units.Pourcent, entity: CPU_E });

const cpus = os.cpus();
for (let id = 0; id < cpus.length; id++) {
    const entity = new Entity(`CPU.${id}`, { parent: CPU_E })
        .set("speed", cpus[id].speed)
        .set("model", cpus[id].model);

    const options = { unit: Units.MilliSecond, entity };
    new MetricIdentityCard("USER", options);
    new MetricIdentityCard("NICE", options);
    new MetricIdentityCard("SYS", options);
    new MetricIdentityCard("IDLE", options);
    new MetricIdentityCard("IRQ", options);
}

function cpuInterval() {
    // we do the work here (check the github)
}
CPU.registerInterval(cpuInterval, 5000);

export default CPU;

The following example register new cpu entities (A SlimIO entity is a Configuration Item) and metric identity card (it is a complete description of the metric that we will collect.).

In the CPU addon we are not registering any new custom callbacks... Callbacks are communication functions belonging to addons. They can trigger actions or recover data from an addon. They allow communication between addons in a one-to-one relationship.

Each addon has a set of callbacks (native or declared by the developer itself).

Callbacks name must respect the snake_case typographic convention. (Note: the registerCallback automatically switch camelCase to snake_case).

These "routes" or "targets" are printed for each addon when you start the agent with silent mode to off.

[core] 10 Mar 2020, 23:17:15 - Setup routing target: events.summary_stats
[core] 10 Mar 2020, 23:17:15 - Setup routing target: events.declare_entity
[core] 10 Mar 2020, 23:17:15 - Setup routing target: events.declare_entity_descriptor
[core] 10 Mar 2020, 23:17:15 - Setup routing target: events.get_descriptors
[core] 10 Mar 2020, 23:17:15 - Setup routing target: events.search_entities
[core] 10 Mar 2020, 23:17:15 - Setup routing target: events.get_entity_by_id
[core] 10 Mar 2020, 23:17:15 - Setup routing target: events.remove_entity
[core] 10 Mar 2020, 23:17:15 - Setup routing target: events.declare_mic
[core] 10 Mar 2020, 23:17:15 - Setup routing target: events.publish_metric
[core] 10 Mar 2020, 23:17:15 - Setup routing target: events.get_mic_stats
[core] 10 Mar 2020, 23:17:15 - Setup routing target: events.pull_mic
[core] 10 Mar 2020, 23:17:15 - Setup routing target: events.delete_mic_rows
[core] 10 Mar 2020, 23:17:15 - Setup routing target: events.get_mic
[core] 10 Mar 2020, 23:17:15 - Setup routing target: events.create_alarm
[core] 10 Mar 2020, 23:17:15 - Setup routing target: events.get_alarms
[core] 10 Mar 2020, 23:17:15 - Setup routing target: events.get_alarms_occurence
[core] 10 Mar 2020, 23:17:15 - Setup routing target: events.remove_alarm
[core] 10 Mar 2020, 23:17:15 - Setup routing target: alerting.start
[core] 10 Mar 2020, 23:17:15 - Setup routing target: alerting.stop
[core] 10 Mar 2020, 23:17:15 - Setup routing target: alerting.sleep
[core] 10 Mar 2020, 23:17:15 - Setup routing target: alerting.status
[core] 10 Mar 2020, 23:17:15 - Setup routing target: alerting.event
[core] 10 Mar 2020, 23:17:15 - Setup routing target: alerting.health_check
[core] 10 Mar 2020, 23:17:15 - Setup routing target: alerting.register_storm_rule
[core] 10 Mar 2020, 23:17:15 - Setup routing target: alerting.assert_entity

The following example implement a sayHello callback. (The header argument is the callback metadata with informations like who is asking etc..).

import Addon from "@slimio/addon";

const myAddon = new Addon("my-addon");

async function sayHello(header, name = header.from) {
    return `hello ${name}`;
}
myAddon.registerCallback(sayHello);

export default myAddon;

This callback can be triggered by other addons like this

const result = await currentAddon.sendOne("my-addon.say_hello", "fraxken");
console.log(result); // stdout: hello fraxken

The product support lazy communication with Node.js streams and Observable-like implementation.

Packages @slimio/metrics and @slimio/alert were created with the aim of simplifying development and avoiding any "custom" implementation.. More examples and links on the Addon github page.

We have few "prototype" guides that you may read if you want to dig deeper:

• First addon
• Manage alarms and metrics

We are not yet at the stade of an MVP (so don't expect much from these guides).

CLI

The product has his own CLI to install, setup and configure a SlimIO Agent (we even have beta commands to build and distribute agent and addons). The CLI has been designed to be a tool that help developers and integrators in their daily work with the product (the tool has not been designed to be included/embedded on a distant host).

$ npm i @slimio/cli -g
$ slimio --help

Then to setup a complete agent just run the following commands (at the location you want).

$ slimio init --set metrology --add ihm
$ cd agent
$ npm start

And this is it ^^ ! For more commands and more details please check our getting started guide.

Note: in the current published CLI, additional addons like ihm are not sync in the agent configuration file. If this is your case just run:

$ slimio config sync
$ slimio config enable ihm

IHM

The ihm addon bring to the product a little UI (no memory footprint, pure light and fast vanillajs UI). When installed and activated (in the agent config) the UI is accessible on localhost:1338

# to add the ihm addon (is not installed yet).
$ slimio add ihm

Still early stage right now (will require month of work). The UI will be customizable too (custom widgets for the dashboard, custom menu and containers). We are even thinking of adding fully customizable addon interfaces to allow a rich experience.

That's it for this article

This article do not cover 1/10 of the SlimIO project but that was not my goal here. This is an introduction that will leave room for dozens of articles for each parts of the project!

Orange tiles are SlimIO repositories!

Next articles will talk about all the tools we created for/around SlimIO (All are open-source and MIT licensed by the way). These tools allow me to maintain more than 120 git repositories (~70 npm packages) ALONE.

Do not hesitate to check the github (governance) for more informations. https://github.com/SlimIO

Thanks you for reading me and see you for the next article :)

Best Regards,
Thomas