Designing MyUnisoft Next-Gen Accounting APIs

Hello 👋

I'm excited to present my latest technical article detailing the design and development process behind the next iteration of the MyUnisoft Set of accounting external APIs.

Internally known as MAD for MyUnisoft Accounting Data 😄, this project showcases our team's innovative approach and dedication to enhancing API and Partners experience.

💡 Context about the business requirements

You might be questioning the need for developing new APIs. We embarked on this effort to address a spectrum of issues and fulfill unmet needs, including:

• 🔄 Introducing a modern and efficient format for exporting and importing accounting folders, along with their parameters (particularly useful for migrations between tenants).
• 📊 Incorporating accounting data that was previously absent from historical partners APIs, such as Analytics.
• 🚀 Constructing high-performance, user-friendly APIs for our partners, adhering to modern API best practices.

When I refer to high performance, I'm not solely emphasizing faster response times. It also includes improving the security and strength of our systems by using cache and queues, as well as ensuring better observability.

👻 TRA

Historically, the CEGID TRA format has been predominant in the French market, but it faces a number of issues:

• Tailored for CEGID's specific needs, many columns within the format prove irrelevant for MyUnisoft's purposes.
• Featuring a column-oriented structure with fixed positions, it becomes resistant to evolution and challenging to implement and maintain 😡.
• Parsing and generating data in this format incur significant costs and complexities 😞.

However, it can be difficult for software editors and accounting firms to adapt to unfamiliar formats quickly. That's why we've decided to be compatible with TRA.

The basic idea is to enable faster integration and smoother migration to our format. If this format has worked so far, we can learn a lot from it 😊.

💪 Accounting imputation for Factur-X

A significant disappointment lies in the realization that simply exchanging Factur-X via APIs will prove largely inadequate for creating a satisfactory experience for accounting editors and their customers (CPA), as the format does not permit embedding data for accounting imputation 😞.

To address this issue, we are planning to implement Factur-X natively within MAD. This format will enable the fields to be omitted, provided that the attached file is a valid Factur-X. Developers will only need to fill in the required fields for imputation purposes.

🚫 There will be no necessity for creating non-standard extensions of the XML format.

✒️ Technical specification

From the beginning, my aim was to develop accounting APIs that incorporate the following features:

• Versioned interface contracts to establish lifecycles and facilitate iterative changes more smoothly.
• Utilization of identical interfaces for both import and export functions, thereby simplifying API interactions.
• 🧠 Maximizing intelligence and independence to minimize dependency on other APIs. For instance, this includes eliminating the necessity for IDs or internal codes during imports.
• Striving for neutrality when crafting the format, ensuring compatibility and flexibility across various systems and use cases.

Lastly, it's imperative to highlight the significance of neutrality in our approach. Too often, technical teams can fall into the trap of designing formats and APIs solely with internal use cases in mind, often under the pressure of immediate business requirements.

However, it's essential to recognize that our target audience primarily comprises developers who experience accounting through the lens of their own unique business contexts. By prioritizing neutrality in our design principles, we aim to simplify the lives of our consumers and empower them with flexible solutions that seamlessly adapt to diverse workflows.

Through MAD, we're committed to fostering an ecosystem that prioritizes developer-centric design, ensuring that our APIs serve as versatile tools capable of meeting a wide range of needs 🌐.

🔍 Gathering information on existing assets

The initial step involved conducting an inventory of existing APIs and comprehensively understanding the data consumed by our partners. We didn't have to do a lot of work, given that we have a nice public documentation available 😎!

We reviewed the APIs of several of our competitors, complemented by our experience integrating with various accounting editors, which guided us in making the right design decisions.

Additionally, we meticulously analyzed our TRA backend to extract the required columns for a preliminary version. This enabled us to identify the essential elements for a version 1, including:

• accounting folder
• payments
• banks
• journals
• exercices
• analytics (axes and their sections)
• entries & movements

🔍 Our plan is to expand this list to incorporate more elements over time.

📚 Building interfaces

The second step involved crafting prototypes for our JSON object interfaces, utilizing TypeScript and JSON Schema.

Here is the finalized version for Journal, which underwent several weeks of iteration, concurrent with the creation of initial drafts of SQL.

interface SimplifiedAccount {
  producerId?: string;
  number: string;
  name: string;
}

type TypeJournal = "Achat" | "Vente" | "Banque";

interface Journal {
  producerId: string;
  name: string;
  customerReferenceCode: string;
  type: TypeJournal;
  counterpartAccount: SimplifiedAccount | null;
  additionalProducerProperties: {
    type: TypeJournalInternal;
    locked: boolean;
  };
}

💡 Properties that are too specific to MyUnisoft are stored in additionalProducerProperties

Simultaneously iterating on the SQL has proven essential, allowing us to eliminate keys that significantly impact performance without providing commensurate value.

Here is an another example with Bank

interface Bank {
  producerId: string;
  IBAN: string;
  BIC: string;
  account: SimplifiedAccount | null;
  journal: {
    producerId: string;
    name: string;
    type: "BANQUE";
    customerReferenceCode: string;
  };
  additionalProducerProperties: {
    isDefault: boolean;
  };
}

A significant effort has been invested in creating objects that are both clean and user-friendly.

🌀 (Bonus) Known your abstraction

We frequently fall into the trap of assuming that our abstractions are universally understood across a entire ecosystem. In accounting, for instance, it's typical to categorize transactions into an abstraction known as an "Entry." Each entry comprises a set of balanced movements (or transactions).

However, many software solutions, including Dataviz, often prefer consuming raw movements without additional abstractions. This preference explains why such solutions historically lean towards formats like FEC, which are simpler compared to TRA-heavy formats.

From my experience working with partners over the years, I've learned that many of them employ various abstractions for VAT, accounts, or analytics. Interestingly, the choice of abstraction can sometimes be influenced by the nature of their software.

The way we structure our data can play a vital role in ensuring a positive experience for specific consumers 💡.

💬 Technical implementation

For this project we decided to create an in-house package that would bring together all the necessary parts. This decision primarily aims to accommodate diverse use cases without the need for repetitive API (or SDK) integration.

The second step was to design a testable architecture for the package. For some parts I drew inspiration from open source libs such as undici and their mocking API.

At one point I was thinking that it might be a problem to implement HATEOAS... However, I discovered that with our architecture, dynamically injecting new links to specific resources using transformers is surprisingly straightforward 😅.

📦 Package usage

Here's a simple example

import * as MAD from "@myunisoft/mad";

const exporter = new MAD.Export({
  version: "1.x",
  accountingFolderId: "10",
  postgresDataSource
});

const { data: exercices } = await exporter.exercices();

Underneath we fetch elements from the database and transform them into the correct format, leveraging the SemVer version injected into the constructor.

Furthermore, we've integrated an observability layer to track the execution times of each component. However, this aspect likely requires further refinement 😕.

async exercices(): Promise<ExportResponse<postgre.Exercice[]>> {
  const startTime = performance.now();

  const rawExercices = await this.provider.exercices();
  const sqlTime = performance.now() - startTime;

  const transformed = this.transformer.exercices(rawExercices);

  return {
    data: transformed.data,
    executionTime: {
      sql: sqlTime,
      transformer: transformed.executionTime,
      total: performance.now() - startTime
    }
  };
}

The provider can simply be mocked up if you need to run unit tests (by forcing a global provider or forcing it using the class constructor).

const kGlobalDispatcher = mock.fn();

before(() => {
  MAD.setGlobalProvider(kGlobalDispatcher);
});

beforeEach(() => {
  kGlobalDispatcher.mock.resetCalls();
});

after(() => {
  MAD.resetGlobalProvider();
});

(Bonus) TRA compatibility

Effectively writing or reading a TRA isn't inherently complex, although it can be quite tedious due to the time-consuming process of properly implementing all its sections.

The challenge lies in being as fast as possible ⚡.

After one or two months of work, our JavaScript implementation can now read and write a complete TRA of several million lines in just a few hundred milliseconds 📈💻.

To achieve this, we've minimized the number of operations and iterations for each section. Additionally, each section is equipped with a comprehensive definition containing all column information, such as position, alignment, and length.

const config: TRAConfig = {
  section: "rib",
  JSONToTRA,
  TRAToJSON,
  columns: [
    {
      name: "fixe",
      mandatory: true,
      type: "a",
      length: 3,
      align: "left",
      position: 0
    },
    {
      name: "identifiant",
      mandatory: true,
      type: "a",
      length: 3,
      align: "left",
      position: 3
    }
    // ... other fields
  ]
}

Each section is equipped with functions designed for efficient conversion between TRA and JSON formats.

function JSONToTRA(
  lineObj: IRib,
  opts: TRAParsingOptions
): string {
  return [
    convertJSONValue(config.columns[0], lineObj.fixe, opts),
    convertJSONValue(config.columns[1], lineObj.identifiant, opts),
    convertJSONValue(config.columns[2], lineObj.auxiliaire, opts),
    // other fields..
    os.EOL
  ].join("");
}

We utilized tools like flamegraphs to pinpoint the longest-running elements and effectively eliminate them. Additionally, we optimized our process by implementing caching mechanisms to avoid redundant operations, such as repeatedly parsing the same dates.

Exploring a Rust implementation could be intriguing to further reduce execution time 🚀. Perhaps in the future, if performance becomes a critical concern 👀.

🌏 API

In parallel with the work done on the MAD package, we drew up a plan for creating the APIs 📝.

However, we encountered a challenge: some of the APIs we'll be exposing are potentially risky ⚠️, as they may permit exporting entire folders along with all associated parameters.

To address this, we're designing asynchronous APIs, incorporating status tracking and caching mechanisms integrated with our event architecture. This setup enables us to trigger webhooks as needed, enhancing security and reliability.

The sequence diagram above illustrates the interactions between the various components involved in exporting an entire accounting folder.

For this export, we have two distinct routes:

• mad/all?format=json&version=1.0.0 to initialize the export.
• mad/all/status?exportId=export_sch-5_acf-1 to fetch the status.

The second route will return a DONE status when processing is complete (Real-time webhook notification can be subscribed to as an option).

{
    "id": "export_sch-1_acf-1",
    "status": "DONE",
    "url": "<url>"
}

📜 Documentation

We've seamlessly extended our documentation, which is readily accessible on Github. Recently, we've enhanced the user experience by incorporating an online Vitepress site, hosted with Github Pages.

It required several weeks of effort, and the pivotal elements include:

• 📝 A comprehensive description outlining our decisions.
• 📋 A detailed specification encompassing all interfaces, schemas, etc.
• 📚 Additional insights, explanations, and guidance tailored for beginners in accounting.
• 🗺️ Instructions on navigating MyUnisoft for developers unfamiliar with our product.

My primary objective is to ensure a positive developer experience (DX) while maintaining thorough and accessible documentation. You're welcome to explore it yourself here, and we eagerly await your feedback 😊.

📆 What's next?

We're only at the very beginning of the project. We will continue to add new content to our export format and work on new options for our various APIs.

In parallel, we're working on the import (I'll probably have a chance to write a new article when we're well advanced on this).

❤️ Credits

This project is primarily a collaborative effort, made possible by the contributions of key team members:

• Benoit GARIAZZO (Product Owner)
• Alexandre MALAJ (Specification, MAD package & SQL).
• Cédric LIONNET (API)
• Pierre DEMAILLY (TRA/JSON compatibility & review)
• Frédéric GUIOU (Help with naming and currently leading import)
• Me 😁 (Specification, Observability, Documentation)
• Léon and Aymeric for their invaluable assistance with existing accounting APIs & formats.

And all other team members who provided valuable feedback and support during the implementation process 🙌.

🙏 That concludes this article. Thank you for reading!