The Advent of Monitoring, Day 3: Easy Monitoring for Self-Hosted Projects with Checkly

This is the third part of our 12-day Advent of Monitoring series. In this series, Checkly's engineers will share practical monitoring tips from their own experience.

This article was written by Daniel Paulus, Checkly's Director of Engineering.

When it comes to running self-hosted services or side projects, monitoring is key. But, who has the time to set up a complex monitoring system?

We want to deliver cool software and not be busy with configuring Prometheus servers or Grafana Dashboards. That’s where synthetic monitoring fits in perfectly – it's straightforward to set up and reliably keeps an eye on things.

Setting Up Multi-Step API Checks

I love to use Checkly for this because of the great Developer Experience it gives me. It's particularly handy with its latest feature: multi-step API checks (still in beta). Multi step checks allow you to write code and use HTTP requests to implement an arbitrary number of requests, organized in steps, and custom logic to monitor any HTTP based service. This is ideal for more advanced monitoring needs without getting too complicated.

The task at hand was to set up basic monitoring for my self-hosted ClickHouse database. For those who want to jump straight in, all the necessary code is available on GitHub. Just clone it, set up ch_pass (clickhouse password), ch_user (clickhouse user name) and ch_url (’https://IP:port’) account variables in your checkly account and run npx checkly deploy

If you prefer a step-by-step guide, here’s how I did it:

We will use Clickhouse’ HTTP interface to run queries against it to check the database health.

First, create a Checkly project. It's as easy as running npm create checkly and following the instructions. Then, you'll need four files in your checks directory:

1. clickhouse.check.ts:

import { MultiStepCheck, Frequency } from "checkly/constructs";
import path from "node:path";

//this defines a check. The actual code is in the spec files
new MultiStepCheck("clickhouse-check-1", {
  name: "Clickhouse Version Check",
  runtimeId: "2023.09",
  frequency: Frequency.EVERY_10M,
  locations: ["us-east-1", "eu-west-1"],
  code: {
    entrypoint: path.join(__dirname, "clickhouse.spec.ts"),
  },
});

new MultiStepCheck("clickhouse-check-2", {
  name: "Clickhouse Free Diskspace",
  runtimeId: "2023.09",
  frequency: Frequency.EVERY_12H,
  locations: ["us-east-1", "eu-west-1"],
  code: {
    entrypoint: path.join(__dirname, "clickhouse-disk.spec.ts"),
  },
});

1. clickhouse.spec.ts contains a very basic set of checks. First it pings the Clickhouse health endpoint to check if the server is running at all. If that succeeds, we run the most basic query, a SELECT version() and make sure the version string is returned as expected. Now we know that Clickhouse is responsive and ready to go.

import { test, expect } from "@playwright/test";
import { baseUrl, queryClickhouse } from "./utils";

const clickHouseVersion = "23.9.1.1854";

async function pingClickhouse(request) {
  const response = await request.get(`${baseUrl}/ping`);
  expect(response.ok()).toBeTruthy();
  const msg = await response.text();
  expect(msg.trim()).toBe("Ok.");
}

test("check clickhouse", async ({ request }) => {
  await test.step("ping clickhouse", async () => {
    await pingClickhouse(request);
  });

  await test.step("check clickhouse version", async () => {
    const response = await queryClickhouse("SELECT version()", request);
    expect(response[0]).toBe(clickHouseVersion);
  });
});

1. clickhouse-disk.spec.ts is a very simple check to make sure Clickhouse has at least 5GB of free disk space left. If not, the check will fail and we will get an alert from Checkly. Clickhouse offers a lot of interesting metrics through it’s query interface so you can monitor many other things. Take a look: https://clickhouse.com/blog/clickhouse-debugging-issues-with-system-tables

import { test, expect } from "@playwright/test";
import { queryClickhouse } from "./utils";

const diskSpaceQuery = `
SELECT 
    name, 
    free_space / 1024 / 1024 / 1024 AS free_space_gb
FROM system.disks;

`;

test("check clickhouse", async ({ request }) => {
  await test.step("check clickhouse diskspace", async () => {
    const response = await queryClickhouse(diskSpaceQuery, request);
    const row = response[0].split("\t");
    console.log(`Disk ${row[0]} has ${row[1]}GB free`);
    //make sure we got more than 5GB left
    expect(parseFloat(row[1])).toBeGreaterThan(5);
  });
});

1. utils.js contains a very simple Clickhouse response parser.. I know there is a lot of room for improvement here. But you know, I want to build apps not monitoring ;-) As you can see, you need to set up the environment variables for user name, password and the Clickhouse http url in your Checkly env variables for this to work.

import { expect } from "@playwright/test";
export const baseUrl = process.env.ch_url;

export async function queryClickhouse(query, request) {
  const buff = Buffer.from(query, "utf-8");
  const response = await request.post(`${baseUrl}`, {
    headers: {
      "X-ClickHouse-User": process.env.ch_user,
      "X-ClickHouse-Key": process.env.ch_pass,
    },
    data: buff,
  });

  const buf = await response.body();
  expect(response.ok()).toBeTruthy();
  const res = new String(buf).toString();
  const resArray = res.split("\n");
  resArray.pop();
  return resArray;
}

Once you have your files ready, run npx checkly login to login to your account and then give it a try by running npx checkly test. If that works and you Clickhouse cluster responds, it is time to unite testing with monitoring and deploy your checks to Checkly for continuous active monitoring. Deploy your checks using npx checkly deploy.

And we’re done! This setup will continuously monitor your ClickHouse database, ensuring it’s up and running all the time. How cool is it that you can put the monitoring code next to your application code and deploy changes that easily?

In case your service does encounter an issue, it's important to get notified quickly. Checkly offers various alerting methods, including SMS, phone calls, Slack, and Pagerduty. You can set these up by checking out Checkly's Alert Channels documentation.

With Checkly, setting up monitoring for your self-hosted projects is a breeze, giving you both peace of mind and more time to focus on other aspects of your projects.

This is the third part of our 12-day Advent of Monitoring series. In this series, Checkly's engineers will share practical monitoring tips from their own experience.

This article was written by Daniel Paulus, Checkly's Director of Engineering.