Ilya Krukowski for Lokalise

Posted on Aug 19, 2022 • Edited on Aug 25, 2022 • Originally published at lokalise.com

Lokalise custom processors: Node and Fastify

In this article, we will discuss a new Lokalise feature called Custom processor. The Custom processor app enables you to intercept translation keys uploaded and downloaded on Lokalise, then analyze or transform those as needed.

Here we’ll show you how to create a Custom processor using Node and Fastify, set it up on Lokalise, and test it.

You can find even more code samples in our DevHub.

What is a Custom processor?

The Custom processor app enables you to utilize your own server or a third-party service to fetch translation keys imported to and exported from Lokalise, analyze, format, transform, or even remove those as needed.

To better understand the idea, let's see how a regular upload is performed on Lokalise:

You choose one or more translation files to upload.
Lokalise detects the translation file languages automatically, but you can adjust those as needed.
You adjust any additional uploading options and click Upload.
Lokalise parses the uploaded data, and extracts keys and translation values.
Extracted data is displayed in the translation editor.

However, if you have a Custom processor set up, after performing step 4 Lokalise will automatically send the parsed data to your processor. There you can perform any additional actions, such as analyzing the uploaded data, transforming your translations, adding special formatting, removing unnecessary special characters or certain banned words, restructuring translation keys, and so on. The only requirement is that the data returned by your processor must preserve the initial structure.

Now let's briefly cover the download process:

You choose file formats to download.
You adjust any additional options as needed and click Build and download.
Lokalise collects the chosen keys and translations.
Lokalise generates translation files based on the download options and zips those into an archive.

So, a Custom processor will intercept the data collected in step 3 and once again it can perform any additional modifications before these data are zipped into an archive.

As you can see, the idea is quite simple, however with this feature you can automate repetitive manual work.

Why would you need a Custom processor?

There are various use cases:

Translate the uploaded content using your own or third-party service.
Perform text analysis for your localization workflow needs.
Apply special formatting to the imported or exported translations.
Clean up the imported translations by removing unnecessary special characters.
Remove or replace banned or undesired words.
Perform placeholder replacement.
Restructure or rename the exported translation keys.

So, as you can see, there are numerous potential use cases and you can think of very complex scenarios! Therefore, let's see how to code a custom processor.

Creating a Custom processor

In this section, you'll learn how to create a script performing pre-processing and post-processing of your translation data.

Pre-processing happens when you are uploading translation files. We are going to use a third-party API called funtranslations.com to translate regular English texts into pirate speech.
Post-processing is performed when you are downloading your translation files back. We'll create a script to remove a banned phrase from all our translations.

Preparing the app

So, let’s get started by creating a new project directory (for example, custom_processor) with a package.json file inside:

{
  "name": "custom-processors",
  "main": "server.js",
  "scripts": {
    "start": "node server.js",
    "dev": "fastify start -w -l info -P app.js",
  },
  "dependencies": {
    "config": "^3.3.7",
    "fastify": "^3.29.0",
    "fastify-autoload": "^3.13.0",
    "fastify-cli": "^3.1.0",
    "fastify-plugin": "^3.0.1"
  },
  "devDependencies": {
    "nodemon": "^2.0.16"
  }
}

Run:

npm install

Then, create a server.js file with the following contents:

require('dotenv').config()
const Fastify = require('fastify')
const app = Fastify({
  logger: true,
  pluginTimeout: 30000,
})
app.register(require('./app.js'))
app.listen(process.env.PORT || 3000, '0.0.0.0', (err) => {
  if (err) {
    app.log.error(err)
    process.exit(1)
  }
})

Finally, add an app.js file:

const path = require('path')
const AutoLoad = require('fastify-autoload')
module.exports = async function (fastify, opts) {
  fastify.register(AutoLoad, {
    dir: path.join(__dirname, 'plugins'),
    options: Object.assign({}, opts),
  })
  fastify.register(AutoLoad, {
    dir: path.join(__dirname, 'routes'),
    options: Object.assign({}, opts),
  })
}

Nice!

Performing pre-processing

Now let’s see how to perform pre-processing. I’d like to find all English translations and turn them into pirate speech using funtranslations.com. To achieve that, create a new file called routes/preprocess.js. Inside it we’ll iterate over translation keys:

module.exports = async function (fastify, opts) {
  fastify.register(require('fastify-http-client'))
  fastify.post('/preprocess', async function (request, reply) {
    if (!request.body) {
      return { "error" : "Invalid request: Missing body"};
    }
    const payload = request.body;
    for (const [keyId, keyValue] of Object.entries(payload.collection.keys)) {
    }
    return payload;
  })
}

Then we have to check the language ID; it should be equal to 640 which is English. If the language ID is the proper one, we’ll send an API request to translate our texts:

module.exports = async function (fastify, opts) {
  fastify.register(require('fastify-http-client'))
  fastify.post('/preprocess', async function (request, reply) {
    if (!request.body) {
      return { "error" : "Invalid request: Missing body"};
    }
    const payload = request.body;
    for (const [keyId, keyValue] of Object.entries(payload.collection.keys)) {
      for (const [lang, v] of Object.entries(keyValue.translations)) {
        if(v.languageId === 640) {
          const res = await fastify.curl('https://api.funtranslations.com/translate/pirate.json&text=' + v.translation).then((result) => {
            const translated = JSON.parse(result.data.toString())
            payload.collection.keys[keyId].translations[lang].translation = translated.contents.translated
          }).catch((err) => {
            payload.collection.keys[keyId].translations[lang].translation = v.translation
          })
        }
      }
    }
    return payload;
  })
}

Basically, this is it.

Performing post-processing

Post-processing is performed in a very similar way. Let’s create a new route which is going to remove a banned phrase, “Flying Dutchman”, from all the translations (after all, pirates are usually afraid of this mystical ship). Populate the routes/postprocess.js file with the following code:

module.exports = async function (fastify) {
  fastify.post('/postprocess', async function (request, reply) {
    if (!request.body) {
      return { error: 'Invalid request: Missing body' }
    }
    const payload = request.body
    for (const [keyId, keyValue] of Object.entries(payload.collection.keys)) {
      for (const [lang, v] of Object.entries(keyValue.translations)) {
        payload.collection.keys[keyId].translations[lang].translation = v.translation.replace(
          /flying\sdutchman/gi,
          '',
        )
      }
    }
    await reply.send(payload)
  })
}

Here we are simply removing the banned phrase from all translations.

Great job!

Setting up a Custom processor

Once you have created a Custom processor, it’s time to enable it! Therefore, perform the following steps:

Proceed to lokalise.com, log in to the system, and open your translation project.
Proceed to Apps, find Custom processor in the list, and click on it.

Then click Install and configure the app by entering a preprocess and a postprocess URL.
If needed, choose a file format to run this processor for.
When you are ready, click Save changes.

That’s it, great job!

Testing it out

Now that everything is ready, you can test your new processor. To do that, proceed to the Upload page, choose an English translation file, and click Upload.

Return to the project editor and observe the results:

To test the postprocessing feature, simply add the "Flying Dutchman" phrase to any translation, and proceed to the Download page.

Click Build and download.

You should not see the banned phrase in the resulting translations.

{
  "password": "Say th' secret phrase and enter. It's  over there",
  "welcome": "Welcome, me bucko!"
}

Congratulations!

Conclusion

And this concludes our tutorial. As you can see, the Custom processor is a very powerful feature that can be used to build custom workflows, so be sure to check it out. You can find even more code samples in our DevHub. However, if you have any additional questions, please don’t hesitate to drop us a line.

Thank you for sticking with me today, and happy coding!