DEV Community

KiminLee
KiminLee

Posted on

Release 04 part3: Changing the lexer algorithm

Read the previous blog please (this blog is written assuming readers already read the previous blog)

Before start

Previosuly, I used es6 arrow function, but after I went through the program, I realized that the program did not use arrow function at all, so I changed the new arrow function to the es5 function.

const regexTerms = () => {
function regexTerms() {
...

const resolveRegex = () => {
function resolveRegex() {
...
Enter fullscreen mode Exit fullscreen mode

Also, most of the code was using let, even if the variable was actually constant. So I changed them either.

  let rawRegex = regexTerms.join("|");
  let mainRegex = new RegExp(rawRegex, "gi");
  let mainRegex = regexTerms();
  let resolverRegexes = resolveRegex();

Enter fullscreen mode Exit fullscreen mode
  const rawRegex = regexTerms.join("|");
  const mainRegex = new RegExp(rawRegex, "gi");
  const mainRegex = regexTerms();
  const resolverRegexes = resolveRegex(lexerItem);
Enter fullscreen mode Exit fullscreen mode

Problem

So, the main problem was there was too many loop to generate "lexer object" to be compiled after all.

Simply the algorithm looks like this,

// three levels loop
lexerResults.map(lexerResult=>{
   for(let i = 0; i < resolverRegexes.length; i++){
      ...
      Object.keys(tokens.tokens).forEach((token)=>{
       ...
      })
   }
})
Enter fullscreen mode Exit fullscreen mode

So, inside the map loop, there is for loop, then inside the for there is also foreach loop. If we can solve this problem out. (For example, reduce loop), the efficiency will be enhanced.

Solution

So, essentially resolveLexerItemMetadata was doing compare first, then make the object later (which was made in another function beforehand). But, what if the process is combined into one single function, I mean, compare and make the object at the same time.

// function: creating the object.
function resolveRegex() {
  let resolverRegexes = [];
  Object.keys(tokens.tokens).forEach((token) => {
    resolverRegexes.push({
      match: new RegExp(`(${tokens.tokens[token].match})`, "gi"),
      data: {
        identifier: token,
      },
    });

// function: comparing with the given token and object
unction resolveLexerItemMetadata(lexerItem) {
  let data = {};
  const resolverRegexes = resolveRegex(lexerItem);
  for (let i = 0; i < resolverRegexes.length; i++) {
    if (lexerItem.match(resolverRegexes[i].match) != null) {
      data["identifier"] = resolverRegexes[i].data.identifier;
      data["token"] = lexerItem;

      break;
    }
  }
  return data;
}
Enter fullscreen mode Exit fullscreen mode

Solution 2

/* Instead of creating an array of objects for Token to be compared with given lexerItm,
*  Find the token with the lexerItm, then return a single object
*  Previous version, let n = the num of tokens, the best case is Ω(n) runtime.
*  This version, let n = the num of tokens, the best case is Ω(1) runtime.
*/
function resolveRegex(lexerItem) {
  const foundToken = Object.keys(tokens.tokens).find((token) => {
    const matchReg = new RegExp(`(${tokens.tokens[token].match})`, "gi");
    return lexerItem.match(matchReg) != null ? true : false;
  });
Enter fullscreen mode Exit fullscreen mode

Instead of creating a resolvedToken object (intermediate object), simply find the token and if found, creating a final version of object.

function parseLine(line) {
  const lexerResults = line.match(mainRegex); // The Regex result is quickly translated into an array
  let lexeredLine = [];
  lexerResults.map((lexerResult) => {
    lexeredLine.push(resolveRegex(lexerResult));
  });
  return lexeredLine;

Enter fullscreen mode Exit fullscreen mode

By doing so, we can remove the resolveLexerItemMetadata function and put the process altogether.
This will results in reducing one extra loop (foreach loop).

After testing the runtime, I could see the runtime is much faster: approximately 20%.

Visit the PR

Conclusion

At first, I tried to contribute to the bigger project (even if a small change). However, It is hard to find a good one for me even though I watched lecture and tried to practice.
The simple bug is already taken by others, and lots of code I cannot understand.
The reason, I think, is I might be too afraid of failing to create PR. So personally, I score myself 80/100

Even though I could not achieve my goal(contributing to the real project like react.js, next.js ...), I learned a lot of Open Source Development community and environment. How to look for contribution, license, unit testing, git command and communicate with open source development community.

These assets are not something to disappear. This semester is to be end, but my programming experience just begin. I will keep practice the thing I have learned in this course and try to keep connecting with the community. And later I will be able to contribute to the real world project.

Top comments (0)