String.prototype.replace asynchronously?

Original post: https://www.ycmjason.com/blog/2018/04/28.html

this article assumes basic knowledge of RegExp.

Background

I was working with vuepress last week and I realise I want to be able to break my very long markdown into partials. So I raised this issue. And the legend, Evan You, suggested to use <!-- include ./sectionA.md -->. Then I picked up his advise and started digging into the code of vuepress.

String.prototype.replace

Before I explain how I solved the problem, I would like to make sure we are all on the same page. My solution is based on String.prototype.replace function which I will very briefly explain how this function works. This function takes in two arguments:

1. What to replace (RegExp | String)
2. What to replace with (String | Function)

String.prototype.replace(String, String)

const str = 'I am very happy, happy, happy.';
str.replace('happy', 'sad'); // I am very sad, happy, happy.

The above example shows how we could replace a word in a string. Notice that only the first occurrence of happy is replaced by sad. This behaviour is similar to when you pass in a RegExp without global flag.

String.prototype.replace(String, Function)

const str = 'I am very happy, happy, happy.';
str.replace('happy', word => 'not ' + word);
// ^ I am very not happy, happy, happy.

You could retrieve the matched word by passing in a replacer function. The value returned from the replacer function would be used to replace the word.

This use case is rare and probably not very useful as you already know the targeting word. You could simply do str.replace('happy', 'not happy') to have the same effect.

String.prototype.replace(RegExp, String)

const str = 'I am very happyyyyy, happy, happy.';
str.replace(/happ(y+)/, 'sleep$1'); // I am very sleepyyyyy, happy, happy.
str.replace(/happ(y+)/g, 'sleep$1'); // I am very sleepyyyyy, sleepy, sleepy.

Should be fairly straight forward. Two things to note:

1. /happ(y+)/ matches "happy" and all the "y"s that come after it.
2. $1 will be replaced by whatever is matched in the groups () of the RegExp. You can have more than one groups and simply use $2, $3, $4 as their placeholders.

String.prototype.replace(RegExp, Function)

const str = 'I am very happyyyyy, happy, happyy.';

str.replace(/happ(y+)/, (match, ys) => {
    // match: 'happyyyyy'; ys: 'yyyyy'
    return 'sleep' + ys;
}); // I am very sleepyyyyy, happy, happyy.

str.replace(/happ(y+)/g, (match, ys) => {
    // This function is called 3 times:
    //     1. match: 'happyyyyy'; ys: 'yyyyy'
    //     2. match: 'happy'; ys: 'y'
    //     3. match: 'happyy'; ys: 'yy'
    return 'sleep' + ys;
}); // I am very sleepyyyyy, sleepy, sleepyy.

The comments should be quite self-explanatory.

The synchronous way

Back to the problem we have, to replace <!-- include ./sectionA.md --> with the content of ./sectionA.md.

Any decent regex-er could come up with a regex to match that placeholder, and we came up with something like:

const placeholderRe = /<!--\s*include\s+([^\s]+)\s*-->/g

Note: \s matches any space/tab etc. See here for more information.

This RegExp will match the placeholder as a whole and group the filename after the include.

So I basically use the String.prototype.replace to do the job:

const { readFileSync, existsSync } = require('fs');

const replaceIncludePlaceholdersWithFileContents = str => {
    const placeholderRe = /<!--\s*include\s+([^\s]+)\s*-->/g;
    return str.replace(placeholderRe, (placeholder, filename) => {
        if (!existsSync(filename)) return placeholder;
        return readFileSync(filename, 'utf8');
    });
};

This works, we just need to handle one more case, i.e. when the partial being included also contain <!-- include file.md -->. Obviously this become a recursive problem. The way to deal with this is simply doing the Leap of faith.

Simply by applying replaceIncludePlaceholdersWithFileContents recursively on the content of each file included by the current file would do the job!

So we have something like:

const { readFileSync, existsSync } = require('fs');

const replaceIncludePlaceholdersWithFileContents = str => {
    const placeholderRe = /<!--\s*include\s+([^\s]+)\s*-->/g;
    return str.replace(placeholderRe, (placeholder, filename) => {
        if (!existsSync(filename)) return placeholder;
        return replaceIncludePlaceholdersWithFileContents(
            readFileSync(filename, 'utf8')
        );
    });
};

This time our base case is when the included file do not contain the placeholder, then the function should terminate as the replacer function would not be called.

The asynchronous way

So I submitted the pull request, and some feedback has been given to me suggesting the use of fs.readFile, the async version of fs.readFileSync.

Immediately I realise, if I have a function called asyncStringReplace(str, search, replacer) which does what String.prototype.replace does but allow replacer to return a Promise, then I could just change my code to the following and it would work.

const { readFile, existsSync } = require('fs-extra');

const replaceIncludePlaceholdersWithFileContents = async str => {
    const placeholderRe = /<!--\s*include\s+([^\s]+)\s*-->/g;
    return await asyncStringReplace(str, placeholderRe, async (placeholder, filename) => {
        if (!existsSync(filename)) return placeholder;
        return await replaceIncludePlaceholdersWithFileContents(
            await readFile(filename, 'utf8')
        );
    });
};

Spent so much time on thinking about the replacement of the placeholder, I would love to retain the already existing logic as much as possible.

So now what I need to write is just the asyncStringReplace method.

asyncStringReplace

The asyncStringReplace method should take in three arguments:

1. str - the original string
2. regex - the RegExp that represents the substring of str to be replaced
3. aReplacer - an asynchronous function that takes in each match, should return Promise.

I basically copied from mdn the "while-loop" that loops through the matches using RegExp.prototype.exec. By using RegExp.prototype.exec we could track the RegExp.lastIndex and match.index of each match, which I couldn't think of a way to achieve this with String.prototype.match.

const asyncStringReplace = async (str, regex, aReplacer) => {
    const substrs = [];
    let match;
    let i = 0;
    while ((match = regex.exec(str)) !== null) {
        // put non matching string
        substrs.push(str.slice(i, match.index));
        // call the async replacer function with the matched array spreaded
        substrs.push(aReplacer(...match));
        i = regex.lastIndex;
    }
    // put the rest of str
    substrs.push(str.slice(i));
    // wait for aReplacer calls to finish and join them back into string
    return (await Promise.all(substrs)).join('');
};

My approach basically split the given str with the given regex into substrings and put them into substrs.

substrs therefore contains:

[
    /* first loop in while */
    NON_MATCHING_STRING,
    aReplacer(MATCHING_STRING),

    /* second loop in while */  
    NON_MATCHING_STRING,
    aReplacer(MATCHING_STRING),

    /* ... */,

    /* n-th loop in while */  
    NON_MATCHING_STRING,
    aReplacer(MATCHING_STRING),

    /* substrs.push(restStr) */
    REST_NON_MATCHING_STRING
]

E.g.
If we call the following

asyncStringReplace('i am happyy, happy === happyyy very!', /happ(y+)/g, someAsyncReplacer);

The corresponding substrs would be:

[
    /* first loop in while */
    'i am ',
    someAsyncReplacer('happyy', 'yy'),

    /* second loop in while */
    ', ',
    someAsyncReplacer('happy', 'y'),

    /* third loop in while */
    ' === ',
    someAsyncReplacer('happyyy', 'yyy'),

    /* substrs.push(restStr) */
    ' very!'
]

Notice since aReplacer is an asynchronous function, aReplacer(MATCHING_STRING) would therefore be a Promise. Promise.all could be used here to construct a Promise which resolves when all promises are resolved in this list.

The last line

    return (await Promise.all(substrs)).join('')

await Promise.all(substrs) would yield to an array of string and .join('') would join all of them back together.

An example of how this could be applied:

const { readFile, existsSync } = require('fs-extra');

const replaceIncludePlaceholdersWithFileContents = async str => {
    const placeholderRe = /<!--\s*include\s+([^\s]+)\s*-->/g;
    return await asyncStringReplace(str, placeholderRe, async (placeholder, filename) => {
        if (!existsSync(filename)) return placeholder;
        return await replaceIncludePlaceholdersWithFileContents(
            await readFile(filename, 'utf8')
        );
    });
};