Yann IRBAH

Posted on Feb 17, 2022 • Originally published at l0ck3.hashnode.dev on Feb 17, 2022

NotionOps - Part 2: Write Qovery data into Notion

Here is the next step in our NotionOps journey. Recap of what we have done so far:

We created a basic Node.js project.
We integrated with the Qovery API to retrieve the list of the projects in our organization.
We integrated with the Notion API to retrieve a page.

Goal

Now things get exciting. Reading from API is cool, but the real fun begins when we make tools interact. We will automatically create pages and databases in Notion for this second part, based on our running applications on Qovery. We'll also refresh the list every X seconds.

Create the top-level Notion database

In PART 1, we created a simple NotionOps page for the sake of simplicity. However, using a database as our parent will make things easier.

Delete your NotionOps page
Create a new NotionOps page as a database. We'll pick the Gallery view since it will be a nice way to display our projects.
Share the page with your integration.
Retrieve the ID of the page from the sharing link.
Update the NOTION_PAGE_ID value in your .env file.

You now have a database with a gallery view as a parent page. Let's edit the properties of the database:

Click on the Properties link at the top of your gallery.
Delete the existing properties except for Name.
Create two new text properties: Description and Qovery ID.
Uncheck Qovery ID since we don't need to see it on our card.

Add projects to our Notion database

Now let's get some data into our freshly created database. We already have a function to read Qovery projects from the API. Let’s first enrich our notion.js file with a new function for adding entries to our projects database:

exports.createProject = async (project) => {
  const res = await notion.pages.create({
    parent: {
      type: "database_id",
      database_id: notionPageID,
    },
    cover: {
      type: "external",
      external: {
        url: `https://source.unsplash.com/random/900x700/?abstract,${project.id}`
      }
    },    
    properties: {
      Name: {
        title: [{
          text: {
            content: project.name
          }
        }]
      },
      "Qovery ID": {
        rich_text: [{
          text: {
            content: project.id
          }
        }]
      },      
      "Description": {
        rich_text: [{
          text: {
            content: project.description
          }
        }]
      },      
    }
  });  

  return res;
}

The payload structure looks a bit complex, but you can refer to the API documentation for all the details.

The only thing you might wonder about is the cover part. Using the URL https://source.unsplash.com/random/900x700/?abstract,${project.id}, we are loading random images from Unsplash to use as a cover for our cards. It is optional, but we want our gallery to look fancy, don't we?

Update your index.js to call this new function:

require("dotenv").config();

const qoveryClient = require("./src/apis/qovery");
const notionClient = require("./src/apis/notion");

const createNotionProjects = async (projectsList) => {
  await Promise.all(projectsList.map((project) => (
    notionClient.createProject(project)
  )))
}

qoveryClient.listProjects()
  .then(async (projects) => await createNotionProjects(projects))
  .catch((e) => console.log("ERROR", e.message));

Rerun your program:

yarn start

And voila! You should see your Qovery projects in your Notion database.

Run the program one more time. Oops! Our projects are duplicated. Let's fix that! First, remove the exports on the createProject function. We'll export another one called createOrUpdateProject later.

Now we need to look into our database to find if we already have an entry with a given Qovery ID:

const findProjectByQoveryID = async (qoveryID) => {
  const res = await notion.databases.query({
    database_id: notionPageID,
    filter: {
      property: 'Qovery ID',
      text: {
        equals: qoveryID,
      },
    }
  });

  return res.results[0];  
}

We use the filter property, which works the same as in the Notion UI, matching the actual Qovery ID with the same property in the database. Since a database query always returns an Array, we only need to return the first element.

Then, create a new updateProject function:

const updateProject = async (notionID, project) => {
  const res = await notion.pages.update({
    page_id: notionID,
    properties: {
      Name: {
        title: [{
          text: {
            content: project.name
          }
        }]
      },   
      "Description": {
        rich_text: [{
          text: {
            content: project.description
          }
        }]
      },
    },
  });

  return res;
}

And our new exported function createOrUpdateProject:

exports.createOrUpdateProject = async (project) => {
  let res;
  const notionProject = await findProjectByQoveryID(project.id);

  if (notionProject) {
    res = await updateProject(notionProject.id, project);
  } else {
    res = await createProject(project);
  }

  return res;
}

Last step, we update our index.js to call this new function:

require("dotenv").config();

const qoveryClient = require("./src/apis/qovery");
const notionClient = require("./src/apis/notion");

const refreshNotionProjects = async (projectsList) => {
  await Promise.all(projectsList.map((project) => (
    notionClient.createOrUpdateProject(project)
  )))
}

qoveryClient.listProjects()
  .then(async (projects) => await refreshNotionProjects(projects))
  .catch((e) => console.log("ERROR", e.message));

Before rerunning the program, let's delete all the entries from our Notion database. Try running the program several times, no duplicates! Good. Go to the Qovery console and change the name and description for one of your projects. Rerun the program, and like magic, your changes are reflected on Notion.

We are not handling the case where a project is deleted in Qovery; otherwise, this article would be way too long, so I leave space for your creativity.

List Applications

Our project gallery looks fantastic but is not very useful at the moment. What we want is to see our applications and their statuses. Qovery has a three levels hierarchy:

Project: which would typically correspond to one of your applications, including all services and their dependencies, like databases.
Environment: which allows you to deploy separate instances of your applications with different settings. For example, staging and production.
Application: A service running on your Kubernetes cluster.

For our Notion dashboard, we prefer to have a list of all the applications running in a project at first glance. So getting all the applications will be a two-steps process:

Retrieve all the environments for a project.
List all the applications for each environment.

First, we need to list the environments in a project. This one is pretty straightforward. In qovery.js add:

const listEnvironments = async (projectID) => {
  const res = await qoveryClient.get(`/project/${projectID}/environment`);

  return res.data.results.map(({ id, name }) => ({
    id,
    name,
  }));
};

Then to list all the applications in an environment. Very similar to the previous one:

const listApplications = async (environmentID) => {
  const res = await qoveryClient.get(
    `/environment/${environmentID}/application`
  );

  return res.data.results.map(({ id, name }) => ({
    id,
    name
  }));
};

Finally we create an exported function to return all the applications in a project:

exports.listAllApplications = async (projectID) => {
  const envs = await listEnvironments(projectID);
  const apps = [];

  for (env of envs) {
    const res = await listApplications(env.id);

    apps.push(
      res.map((app) => ({
        id: app.id,
        name: app.name,
        environment: {
          id: env.id,
          name: env.name
        },
        status: "unknown",
      }))
    );
  }

  return apps.flat();
};

Here we return all the applications for a project while conveniently shaping the application object. Note that for now, we hardcore the status. We'll get the actual application status later.

Add applications to Notion

Now that we can list our applications for a Qovery project let's add them to our Notion pages. The ideal would be to have a database with all applications on our projects pages. Unfortunately, there is no way to create an inline database on a page through the API at the time of writing. We can still make a child database that will appear as a link on our project page. (You'll be able to turn it inline through the UI).

To add this database to our project pages, update the createProject function in notion.js:

const createProject = async (project) => {
  const res = await notion.pages.create({
    parent: {
      type: "database_id",
      database_id: notionPageID,
    },
    cover: {
      type: "external",
      external: {
        url: `https://source.unsplash.com/random/900x700/?abstract,${project.id}`,
      },
    },
    properties: {
      Name: {
        title: [
          {
            text: {
              content: project.name,
            },
          },
        ],
      },
      "Qovery ID": {
        rich_text: [
          {
            text: {
              content: project.id,
            },
          },
        ],
      },
      Description: {
        rich_text: [
          {
            text: {
              content: project.description,
            },
          },
        ],
      },
    },
  });

  // ADDED PART
  await notion.databases.create({
    parent: {
      type: "page_id",
      page_id: res.id,
    },
    title: [{ type: "text", text: { content: "Applications List" } }],
    properties: {
      Name: {
        title: {},
      },
      Environment: {
        rich_text: {},
      },
      Status: {
        select: {
          options: [
            {
              name: "Not Deployed",
              color: "gray",
            },
            {
              name: "Running",
              color: "green",
            },
            {
              name: "Error",
              color: "red",
            },
            {
              name: "Deploying",
              color: "yellow",
            },
          ],
        },
      },
      "Qovery ID": {
        rich_text: {},
      },
    },
  });
  // END

  return res;
};

We're just creating a new database as a child of our project page. For the statuses, we use an arbitrary list convenient for us. We'll map the actual Qovery statuses and this list when we implement this part.

Delete your projects in Notion and relaunch your program.

Now let's fill that list, shall we?

We’ll start by creating a new createApplication function in notion.js:

const createApplication = async (projectID, application) => {
  const appsList = await getAppsList(projectID);

  const res = await notion.pages.create({
    parent: {
      type: "database_id",
      database_id: appsList.id,
    },
    properties: {
      Name: {
        title: [
          {
            text: {
              content: application.name,
            },
          },
        ],
      },
      Environment: {
        rich_text: [
          {
            text: {
              content: application.environment.name,
            },
          },
        ],
      },
      Status: {
        select: {
          name: application.status,
        },
      },
      "Qovery ID": {
        rich_text: [
          {
            text: {
              content: application.id,
            },
          },
        ],
      },
    },
  });

  return res.id;
};

This one is very similar to the createProject one, except for the properties. Except we need to get the Applications List database ID since we only have the projectID, which is just the parent page.

The getAppsList function looks like this:

const getAppsList = async (projectID) => {
  const res = await notion.blocks.children.list({
    block_id: projectID,
  });

  return res.results[0];
};

We retrieve all the children of our project page, and since the database is the only child, we return the first result.

Since we will want to update our applications as well instead of duplicating them, let’s create an updateApplication function:

const updateApplication = async (applicationID, application) => {
  const res = await notion.pages.update({
    page_id: applicationID,
    properties: {
      Name: {
        title: [
          {
            text: {
              content: application.name,
            },
          },
        ],
      },
      Environment: {
        rich_text: [
          {
            text: {
              content: application.environment.name,
            },
          },
        ],
      },
      Status: {
        select: {
          name: application.status,
        },
      },
    },
  });

  return res;
};

And a createOrUpdateApplication one:

exports.createOrUpdateApplication = async (projectID, application) => {
  let res;
  const notionApplication = await findApplicationByQoveryID(
    projectID,
    application.id
  );

  if (notionApplication) {
    res = await updateApplication(notionApplication.id, application);
  } else {
    res = await createApplication(projectID, application);
  }

  return res;
};

We need to implement findApplicationByQoveryID as well:

const findApplicationByQoveryID = async (projectID, qoveryID) => {
  const appsList = await getAppsList(projectID);

  res = await notion.databases.query({
    database_id: appsList.id,
    filter: {
      property: "Qovery ID",
      text: {
        equals: qoveryID,
      },
    },
  });

  return res.results[0];
};

For the same reason as with createApplication, we need to retrieve our Applications List database ID, hence another call to getAppsList.

Ok. Let's wire this up and check if it works. We’ll rewrite our index.js to match our latest changes:

require("dotenv").config();

const qoveryClient = require("./src/apis/qovery");
const notionClient = require("./src/apis/notion");

const refreshNotion = async () => {
  const projects = await qoveryClient.listProjects();

  for (project of projects) {
    const notionProject = await notionClient.createOrUpdateProject(project);
    const apps = await qoveryClient.listAllApplications(project.id);

    for (app of apps) {
      await notionClient.createOrUpdateApplication(notionProject.id, app)
    }
  }
}

refreshNotion()
  .then(async => console.log("SUCCESS", "Notion has been refreshed"))
  .catch((e) => console.log("ERROR", e.message));

It should refresh all the projects and applications every time we launch our program. Give it a try. You should see your applications appear in your Application List databases.

Get the actual status of the applications

Right now, all the applications appear to have the status Not Deployed because we hardcoded it. Let's get the proper status from Qovery. According to the Qovery API documentation, here are the possible statuses:

"INITIALIZED" "BUILDING_QUEUED" "BUILDING" "BUILD_ERROR" "BUILT" "DEPLOYMENT_QUEUED" "DEPLOYING" "DEPLOYMENT_ERROR" "DEPLOYED" "STOP_QUEUED" "STOPPING" "STOP_ERROR" "STOPPED" "DELETE_QUEUED" "DELETING" "DELETE_ERROR" "DELETED" "RUNNING" "RUNNING_ERROR" "CANCEL_QUEUED" "CANCELLING" "CANCEL_ERROR" "CANCELLED"

We'll map them to the four statuses we defined for our Notion dashboard.

In qovery.js add a getApplicationStatus function:

const getApplicationStatus = async (applicationID) => {
  const res = await qoveryClient.get(`application/${applicationID}/status`);
  const qoveryState = res.data.state;
  let state;

  switch (qoveryState) {
    case "INITIALIZED":
    case "STOP_QUEUED":
    case "STOPPING":
    case "STOPPED":    
    case "DELETE_QUEUED":     
    case "DELETING":
    case "DELETED":    
    case "CANCEL_QUEUED":     
    case "CANCELLING":
    case "CANCELLED":
      state = "Not Deployed";
      break;            
    case "BUILDING_QUEUED":
    case "BUILDING":
    case "DEPLOYMENT_QUEUED":
    case "DEPLOYING":
    case "BUILT":
      state = "Deploying";
      break;      
    case "BUILD_ERROR":
      state = "Error";
      break;      
    case "DEPLOYMENT_ERROR":
    case "STOP_ERROR":
    case "RUNNING_ERROR":  
    case "DELETE_ERROR":
    case "CANCEL_ERROR":
      state = "Error";
      break;            
    case "DEPLOYED":
    case "RUNNING":
      state = "Running";
      break;            
  }

  return state;
}

Since we over-simplified the status, some mappings might not be accurate. You can tweak this list to something that suits you better.

Now we update the listAllApplications functions to use the actual status:

exports.listAllApplications = async (projectID) => {
  const envs = await listEnvironments(projectID);
  const apps = [];

  for (env of envs) {
    const res = await listApplications(env.id);

    apps.push(
      await Promise.all(res.map(async (app) => {
        const appStatus = await getApplicationStatus(app.id)

        return {
          id: app.id,
          name: app.name,
          environment: {
            id: env.id,
            name: env.name
          },
          status: appStatus,
        };
      })
    ));
  }

  return apps.flat();
};

And we should be done.

Rerun your program, and you should see your applications statuses change to reflect their actual state on Qovery. You can try to launch a redeploy on the Qovery console, then run your program, and you will see the statuses change again.

Conclusion

Congratulations, and thanks for having read this far. We achieved a lot this time. We now have made Notion reflect the actual state of our Qovery account. In the next part, we'll make it possible to launch the redeploy of Qovery applications from our Notion dashboard.

DEV Community

NotionOps - Part 2: Write Qovery data into Notion

Goal

Create the top-level Notion database

Add projects to our Notion database

List Applications

Add applications to Notion

Get the actual status of the applications

Conclusion

Top comments (0)

Read next

Why Next.js is Better than React for Building an E-commerce Website

Why Clean Architecture Struggles in Golang and What Works Better

RESTART Series: Day 1 – Laying the Foundations

Cross-Project Dependencies Handling with DBT in AWS MWAA