Min

Posted on Mar 15, 2022

Working alone is so exhausting so I created my own assistant

#javascript #node #codenewbie #showdev

Working alone is great... but... I am so tired of dealing with all of these.. these shitty uninspiring, repetitive, tedious tasks...

I didn't realize how many emails I had to reply to and how many invoices I had to make on my own. How many tasks do I have to manage and schedule?!

I sometimes freaked out alone because I was too busy, and sometimes I freaked out because suddenly there's nothing to do..

Console.log( my_life )

I need something stabilizes me all the time no matter what happens to me. Maybe I need a manager. Manage me! (Funny to say that because I left work to not be managed by others.LOL)

For a sec, I think about hiring somebody, but I don't have money for that.(of course)

So here is my attempt to create my personal assistant with discord and notion API and, what I learned from this project.

If there is anything incorrect, please let me know. I'd love to learn from you!

1. I learned NLP

The first thing in my head to create a chatbot is how a chatbot analyze the intention of my text. People say the same thing in various forms of expression

How are you?
How's it going?
What's up?

I can write the code like if( text.includes('how are you') ) for every expression, but how inefficient would it be? This is the time when NLP is useful.

Natural language processing (NLP) is the ability of a computer program to understand human language as it is spoken and written -- referred to as natural language. It is a component of artificial intelligence (AI). NLP has existed for more than 50 years and has roots in the field of linguistics. from techtarget.com

While I still don't fully understand how NLP works, my understanding is so far like this.

A developer submits a set of sentences future users might send
There are three types of variables A user can assign for each text.
(1) intension: The most important part of the sentence. Only one can be assigned for one sentence. ex) I need a new iPhone! -> Request_add_wishlist
(2) Entities: This is a smaller assembly piece, basically a critical keyword to define the intention. ex) I need shopping! -> need, shopping.
(3) Traits : ( I'll ignore this part for now )
An NLP program analyzes what kind of entity composition is for specific intension.
Next time a user send complete new text, a trained NLP will spit out what a user means.

Among multiple different services to provide NLP, I chose to use wit.ai service, which is an open-source service developed by Facebook, Meta.

The useful thing about wit.ai is chatting text I sent by discord also showed up on the wit.ai training session, so even if a bot failed to understand what I meant, I can always go to wit and correct it.

The downside is documentation is not so clear, so... anyway... it's alright. I will take it...

2. I learned Discord.js

In the end, I chose to create a discord bot over a Slack bot even tho I am not really a discord user.

Some servers I joined to get help usually has too many channels on the sidebar, and too many discussion is going on. It just intimates too much. Like you entered in a club and see full of people dancing a Michael Jackson thriller perfectly, but I don't know that damn dance. "Quick, get out from that club!"

Anyway, during this project, I could learn about discord.js and at the same time a little bit more about discord itself...

npm install discord.js

The exciting thing about discord js is it gives so many options to interact with other users. Can create buttons, slash commands, interact with emoji etc.. way more than just sending a message.

The most useful message type is Embed Message.

( I set my coordinator as Tahani from the TV series 'The good place' because I love her character on the show so much. so jolly!)

Embed message returns the beautifully formatted information. You can directly add the field like this.

_embed.addFields({name : "Count", value : Count })

or you can create the dictionary and loop for each keys

Object.keys(style).forEach( k=>{
    _newEmbed.addFields({name : k , value :style[k].toString() })
})

The useful reference for discord bot -> Discord JS Guide

3. I learned Notion API

I love notionHQ. Actually, I am obsessed with it. I've been using it since it was not the mainstream. Recently I heard it acquired automate.io. Alight, NotionHQ, Let's go! Bigger and bigger! But there were a few moments.. when I was considering leaving Notion... it's because Notion doesn't have a recurring task option!

Like I wrote in the intro, repeating and repeating and repeating tasks is already so boring, and I have to create the checkbox of it every time? Oh shit, I don't want that.

I left the Notion and came back again over and over, like a troubled teenager arguing with a parent. A year ago, Notion released notion API; "This is the final, guys. I am going to build an automation system for you and going to be with you happily ever after 4everr."

npm install @notionhq/client

Notion API is great! but unfortunately, there are a few limits for (v1.0.3) right now.

Editing a block is not available yet. Basically, if you want to tick the checkbox block, you should delete the current checkbox item and append a duplicated block again. It's not impossible but kinda annoying. Duplicating a single block is not such a bad case, but how about if you want to duplicate a whole page? Sadly you still need to delete a whole page and duplicate it. If the page has a children element, the problem gets more annoying.
Get children to block only return the top-level items on the hierarchy.

   var children =  await NOTION.blocks.children.list({ block_id: _block.id });

so for example, if there's a notion block like this

The code output will be only [block_A, block_B]. So even if you duplicate the parent with its children, [block_Aa, block_Ab, block_Ba,block_Bb] won't be duplicated.

If you want all the children under parent, there's no other way than keep asking children if they has grand-grand children…

   if( block_A.hasChildren ){
    children.push( await NOTION.blocks.children.list({ block_id: block_A.id }) )
   }

Some block type is not supported. Just think non-native notion blocks such as embed tweet or embed Figma are not supported, which is totally fine, but. butttt. code blocks are also not supported, so I felt kinda sad. :(

All i want is block to save the code so I can run it by ‘eval()’ which can be just any text block. Just not pretty as I wished, that’s all. Haha

var scripts = blocks.filter( block => Object.keys(block)[0] == "callout"  )
await eval(scripts[0])

4. I learned about Cron

One of the must-have features for the coordinator bot was sending scheduled messages, so I can do the task on time.

To do so, I got to know Cron.

npm i cron

A CRON expression is a string comprising five or six fields separated by white space[16] that represents a set of times, normally as a schedule to execute some routine.
from wikipedia

***??... Another regular expression all over again.. horror.. But actually, the breakdown of cron time is pretty simple, min + hour + date + month + week.

Now I should generate cron time by ‘Date()’. This time, instead of writing in js, I wrote as notion built-in formula so even if I update the schedule by notion, the clon time value also can be updated at the same time.

min : 
empty(prop("Unit")) ? if(minute(prop("Date")) + minute(prop("Date")) == 0, format(minute(prop("Edited"))), format(minute(prop("Date")))) : if(prop("Unit") == "minute", "*/" + format(if(empty(prop("Recurring")), 1, prop("Recurring"))), if(prop("Unit") != "hour" or prop("Unit") != "minute", if(empty(prop("Date")), if(empty(prop("Date")), format(minute(prop("Edited"))), format(minute(prop("Date")))), format(minute(prop("Date")))), "*"))

Such a ugly code, for now but so far it seems working as wish. (full code is on my repo)

While I can create a new scheduled event by Notion, I wished to create the new reminder by chat as well. Luckily Wit.ai has built-in entities called "duration" and "datetime", such a lifesaver!

5. I learned Puppeteer

I started to feel more greedy and needy. Hey, bot, you can give weather and different time zone time, my today's tasks and project due date now.. how about.. recommendation for today's meal?? M.M.M

First, I found the biggest recipe API called Spoonacular API, but .. the food pictures look kinda.. not really inspiring..

So instead of food API, I decided to fetch the information directly from my favourite cooking site. I heard the Puppeteer is good for web scraping, I always wanted to try, but this is my first time trying Puppeteer!

npm i puppeteer

Puppeteer is such an amazing package, but there were a few confusing parts though.

When I tried to get element by document.querySelector() .this returned undefined or null object. I was panicked, and I thought Puppeteer failed to find the element, but it's there.. when I specifically ask textContent or src, href?

await page.evaluate(()=>{
    return document.querySelectorAll(".selector").map( sel => sel.textContent)
});

As I understand is what Puppeteer gives me is not an HTML element but something else.. ( is there anybody explain a bit easy for me? i am helpless lol)

Anyway, now I get a recommendation for a recipe. yay

6. I learned Heroku

Because my bot will be deactivated when I don't run bot.js, I needed Heroku to serve it on the cloud.

While I am hosting my sites on Netlify, I am still not familiar with the hosting service. I misunderstood Heroku and Netlify as the same kind of service, but this time, I learned they are not. Like this whole post is such lack of professional knowledge, this is my short understanding about Netlify and Heroku so far

Netlify is more for the front-end, and Heroku is more for the back-end
Netlify is always active, but Heroku fall asleep...zzz
Netlify is site hosting online, but Heroku is run the script on Linux
Netlify start charging by build time, Heroku charging with something called Dyno hours

Dyno is a container that runs the command in Linux. The busier the app will be, the more dyno is required to be purchased. Because I am a free user, I have a single dyno.

Free user has 1000 free dyno hours. It means that even if my app runs consistently without resting for the whole month, it wouldn't get over 730 hours. Free is always good. It makes me feel cosy.😊

Concluding

My Git Repo

While I want to improve more and add more features, my Tahani bot works quite wonderfully right now. Great to have a companion! (sad to say that. lol)

I tried a lot of new things this time, I learned so much; however, I also feel like I didn't spend enough time to learn about each of them more precisely. Maybe that's what I'm going to do with next dev.to

Besides having a customised virtual assistant for my daily routine, the knowledge I earned was so valuable. There are so much potential to improve my interactive character project
Thank you for reading this messy post. I am not a trained developer (I a visual artist), but I am very excited to share thoughts and get to know more about creating something cool.

Please comment on anything if you want to correct my post and educate me. It would be awesome to learn from all of you!

Top comments (56)

lionel-rowe • Mar 15 '22 • Edited

As I understand is what Puppeteer gives me is not an HTML element but something else.. ( is there anybody explain a bit easy for me? i am helpless lol)

The function passed as a callback runs in the context of the headless browser, with all the relevant web APIs available. So within that function, document.querySelector gives a real live DOM element that you can manipulate however you like:

const isItARealDiv = await page.evaluate(() => {
    return document.querySelector("div") instanceof HTMLDivElement
})

isItARealDiv // true

The problem is that when passing the return value back to the parent script, everything is serialized and then deserialized again — something similar to JSON.parse(JSON.serialize(result)). DOM elements can't be properly serialized, getting converted to undefined, so you need to return only the serializable data that you need (text content, specific attributes, inner/outer HTML, etc).

Min • Mar 15 '22

Thank you Lionel-rowe! ( Still confused lol)

Cody Swartz • Mar 17 '22 • Edited

I think Puppeteer and Selenium both suffer from the same problem -- Selenium has been around longer, likely before websites were more complicated with single page magic and dynamic content. The last thing I remembered is that if you query for something before it exists/mounts/renders you'll get nothing back, so you need to wait/poll for it to be available. I thought Puppeteer has helpers around this, or this is why people start to reach for additional libraries on top of these tools to help with this problem. It's been a while since I've touched Puppeteer or Selenium, but I do remember the pains of working with them in single page applications.

Edit: puppeteer.github.io/puppeteer/docs...

lionel-rowe • Mar 17 '22

@ctsstc yeah, Puppeteer gives you various APIs, such as page.waitForSelector, to deal with that, but it can be finnicky knowing exactly what you need to wait for and avoiding race conditons.

lionel-rowe • Mar 15 '22

The parent puppeteer script runs in the Node.JS runtime, whereas the callback to page.evaluate runs in the Chromium browser runtime, headlessly by default ("headless" basically just means that it runs in the background, so you can't visibly see it running). Passing complex data between runtimes is often not possible, because the different runtimes don't know how to interpret it, and DOM elements (DOM is the way the browser interprets HTML) are internally very complex. So to simplify the message passing, Puppeteer uses a serialized format that both runtimes can easily understand. The drawback is that any data that can't be converted to this serialized format is lost.

You can think of "serialized" as meaning something like flat, like a string of letters or binary digits. JSON is a typical serialization format and is useful because it allows the "flattening" of "deep" structures. For example, the JavaScript object { a: { b: 1 } } nests b within a, yet it can be serialized to the JSON string {"a":{"b":1}}. Why is this flat? Well, it's simply the character {, followed by ", followed by a, etc., so it can be read left-to-right; even though the object it represents is a tree structure.

Puppeteer does much of this JSON serialization "under the hood", so you often don't need to worry about it; but JSON can't serialize DOM nodes, because they contain circular structures, e.g. *{ a: { b: *{ a: ... } } } (where * represents a reference to the exact same object). So you need to return only things that JSON can represent — strings, numbers, booleans, null, arrays, and objects containing other JSON-able stuff.

const elementData = await page.evaluate(() => {
    const el = document.querySelector('h1')

    return {
        textContent: el.textContent, // string — OK
        childElementCount: el.childElementCount, // number — OK
        className: el.className, // string — OK
        outerHTML: el.outerHTML, // string — OK
    }
})

console.log(elementData)
// {
//     textContent: 'Posted on Mar 15'
//     childElementCount: 1,
//     className: 'fs-xs color-base-60'
//     outerHTML: '<p class="fs-xs color-base-60">Posted on <time datetime="2022-03-15T02:18:47Z" class="date-no-year" title="Tuesday, March 15, 2022, 2:18:47 AM">Mar 15</time></p>',
// }

Min • Mar 18 '22

Omg @lionelrowe
Big thank you to explain this with full of kind detail!
This is really easy to understand! You are the best!👍👍👍👍👍

JoelBonetR 🥇 • Mar 15 '22 • Edited

Love it 😂 i did a discord weekend project in the past and just for fun and this seems more like a month project.
Did you published it? Can i get a link to include it to my discord server? 🤩

Min • Mar 15 '22

It's not published because I didn't set the notion authentification per user.
it would be so embarrasing if people manage their tasks on my notion. lol

JoelBonetR 🥇 • Mar 16 '22

Hahaha and what about the fun? 😅

Min • Mar 16 '22

I will consider to publish it :)

Akin C. • Mar 16 '22

Hello Min,

thank you for your article.
I've never looked into implementing chatbots, so this is interesting for me.
I like your creative but simple idea of having a digital companion. I don't think that's sad, it's wise, because like you said
"I am so tired of dealing with all of these.. these shitty uninspiring, repetitive, tedious tasks...".

Min • Mar 18 '22

hehe thank you to saying that! Akin!
very glad to hear i inspire somebody with my idea 🥰

🚩 Atul Prajapati 🇮🇳 • Mar 16 '22

Wow my you are the Tony stark

Min • Mar 16 '22

Actually I considered to name it as Jarvis 😂

🚩 Atul Prajapati 🇮🇳 • Mar 17 '22

that's how we inspire

Timmortal • Mar 25 '22

To make a "Jarvis" that follows you from conception to birth and before you merge back in to the universe, is my dream

tq-bit • Mar 16 '22

I love the idea. If you're into shares, you might as well give the Yahoon finance API a shot. It gives you 100 free requests per day, that's more than enough for a single person. For instance, I've built a telegram bot earlier that tells me when is a good time to buy & sell

yahoofinanceapi.com/