DEV Community

Cover image for Working alone is so exhausting so I created my own assistant
Min
Min

Posted on

Working alone is so exhausting so I created my own assistant

Working alone is great... but... I am so tired of dealing with all of these.. these shitty uninspiring, repetitive, tedious tasks...

I didn't realize how many emails I had to reply to and how many invoices I had to make on my own. How many tasks do I have to manage and schedule?!

I sometimes freaked out alone because I was too busy, and sometimes I freaked out because suddenly there's nothing to do..

Console.log( my_life )
Enter fullscreen mode Exit fullscreen mode

I need something stabilizes me all the time no matter what happens to me. Maybe I need a manager. Manage me! (Funny to say that because I left work to not be managed by others.LOL)

For a sec, I think about hiring somebody, but I don't have money for that.(of course)

So here is my attempt to create my personal assistant with discord and notion API and, what I learned from this project.

If there is anything incorrect, please let me know. I'd love to learn from you!


1. I learned NLP

The first thing in my head to create a chatbot is how a chatbot analyze the intention of my text. People say the same thing in various forms of expression

  • How are you?
  • How's it going?
  • What's up?

I can write the code like if( text.includes('how are you') ) for every expression, but how inefficient would it be? This is the time when NLP is useful.

Natural language processing (NLP) is the ability of a computer program to understand human language as it is spoken and written -- referred to as natural language. It is a component of artificial intelligence (AI). NLP has existed for more than 50 years and has roots in the field of linguistics. from techtarget.com

While I still don't fully understand how NLP works, my understanding is so far like this.

  1. A developer submits a set of sentences future users might send

  2. There are three types of variables A user can assign for each text.
    (1) intension: The most important part of the sentence. Only one can be assigned for one sentence. ex) I need a new iPhone! -> Request_add_wishlist
    (2) Entities: This is a smaller assembly piece, basically a critical keyword to define the intention. ex) I need shopping! -> need, shopping.
    (3) Traits : ( I'll ignore this part for now )

  3. An NLP program analyzes what kind of entity composition is for specific intension.

  4. Next time a user send complete new text, a trained NLP will spit out what a user means.

Among multiple different services to provide NLP, I chose to use wit.ai service, which is an open-source service developed by Facebook, Meta.

The useful thing about wit.ai is chatting text I sent by discord also showed up on the wit.ai training session, so even if a bot failed to understand what I meant, I can always go to wit and correct it.

The downside is documentation is not so clear, so... anyway... it's alright. I will take it...


2. I learned Discord.js

In the end, I chose to create a discord bot over a Slack bot even tho I am not really a discord user.

Some servers I joined to get help usually has too many channels on the sidebar, and too many discussion is going on. It just intimates too much. Like you entered in a club and see full of people dancing a Michael Jackson thriller perfectly, but I don't know that damn dance. "Quick, get out from that club!"

Anyway, during this project, I could learn about discord.js and at the same time a little bit more about discord itself...

npm install discord.js
Enter fullscreen mode Exit fullscreen mode

The exciting thing about discord js is it gives so many options to interact with other users. Can create buttons, slash commands, interact with emoji etc.. way more than just sending a message.

The most useful message type is Embed Message.

( I set my coordinator as Tahani from the TV series 'The good place' because I love her character on the show so much. so jolly!)

Embed message returns the beautifully formatted information. You can directly add the field like this.

_embed.addFields({name : "Count", value : Count })
Enter fullscreen mode Exit fullscreen mode

or you can create the dictionary and loop for each keys

Object.keys(style).forEach( k=>{
    _newEmbed.addFields({name : k , value :style[k].toString() })
})
Enter fullscreen mode Exit fullscreen mode

The useful reference for discord bot -> Discord JS Guide


3. I learned Notion API

I love notionHQ. Actually, I am obsessed with it. I've been using it since it was not the mainstream. Recently I heard it acquired automate.io. Alight, NotionHQ, Let's go! Bigger and bigger! But there were a few moments.. when I was considering leaving Notion... it's because Notion doesn't have a recurring task option!

Like I wrote in the intro, repeating and repeating and repeating tasks is already so boring, and I have to create the checkbox of it every time? Oh shit, I don't want that.

I left the Notion and came back again over and over, like a troubled teenager arguing with a parent. A year ago, Notion released notion API; "This is the final, guys. I am going to build an automation system for you and going to be with you happily ever after 4everr."

npm install @notionhq/client

Notion API is great! but unfortunately, there are a few limits for (v1.0.3) right now.

  1. Editing a block is not available yet. Basically, if you want to tick the checkbox block, you should delete the current checkbox item and append a duplicated block again. It's not impossible but kinda annoying. Duplicating a single block is not such a bad case, but how about if you want to duplicate a whole page? Sadly you still need to delete a whole page and duplicate it. If the page has a children element, the problem gets more annoying.

  2. Get children to block only return the top-level items on the hierarchy.

   var children =  await NOTION.blocks.children.list({ block_id: _block.id });
Enter fullscreen mode Exit fullscreen mode

so for example, if there's a notion block like this

The code output will be only [block_A, block_B]. So even if you duplicate the parent with its children, [block_Aa, block_Ab, block_Ba,block_Bb] won't be duplicated.

If you want all the children under parent, there's no other way than keep asking children if they has grand-grand children…


   if( block_A.hasChildren ){
    children.push( await NOTION.blocks.children.list({ block_id: block_A.id }) )
   }
Enter fullscreen mode Exit fullscreen mode

  1. Some block type is not supported. Just think non-native notion blocks such as embed tweet or embed Figma are not supported, which is totally fine, but. butttt. code blocks are also not supported, so I felt kinda sad. :(

All i want is block to save the code so I can run it by ‘eval()’ which can be just any text block. Just not pretty as I wished, that’s all. Haha

var scripts = blocks.filter( block => Object.keys(block)[0] == "callout"  )
await eval(scripts[0])
Enter fullscreen mode Exit fullscreen mode

Every lunch time, recipe inspiration function runs.


4. I learned about Cron

One of the must-have features for the coordinator bot was sending scheduled messages, so I can do the task on time.

To do so, I got to know Cron.

npm i cron
Enter fullscreen mode Exit fullscreen mode

A CRON expression is a string comprising five or six fields separated by white space[16] that represents a set of times, normally as a schedule to execute some routine.
from wikipedia

***??... Another regular expression all over again.. horror.. But actually, the breakdown of cron time is pretty simple, min + hour + date + month + week.

Now I should generate cron time by ‘Date()’. This time, instead of writing in js, I wrote as notion built-in formula so even if I update the schedule by notion, the clon time value also can be updated at the same time.

min : 
empty(prop("Unit")) ? if(minute(prop("Date")) + minute(prop("Date")) == 0, format(minute(prop("Edited"))), format(minute(prop("Date")))) : if(prop("Unit") == "minute", "*/" + format(if(empty(prop("Recurring")), 1, prop("Recurring"))), if(prop("Unit") != "hour" or prop("Unit") != "minute", if(empty(prop("Date")), if(empty(prop("Date")), format(minute(prop("Edited"))), format(minute(prop("Date")))), format(minute(prop("Date")))), "*"))
Enter fullscreen mode Exit fullscreen mode

Such a ugly code, for now but so far it seems working as wish. (full code is on my repo)

While I can create a new scheduled event by Notion, I wished to create the new reminder by chat as well. Luckily Wit.ai has built-in entities called "duration" and "datetime", such a lifesaver!

Text to clean format


5. I learned Puppeteer

I started to feel more greedy and needy. Hey, bot, you can give weather and different time zone time, my today's tasks and project due date now.. how about.. recommendation for today's meal?? M.M.M

First, I found the biggest recipe API called Spoonacular API, but .. the food pictures look kinda.. not really inspiring..

So instead of food API, I decided to fetch the information directly from my favourite cooking site. I heard the Puppeteer is good for web scraping, I always wanted to try, but this is my first time trying Puppeteer!

npm i puppeteer
Enter fullscreen mode Exit fullscreen mode

Puppeteer is such an amazing package, but there were a few confusing parts though.

When I tried to get element by document.querySelector() .this returned undefined or null object. I was panicked, and I thought Puppeteer failed to find the element, but it's there.. when I specifically ask textContent or src, href?

await page.evaluate(()=>{
    return document.querySelectorAll(".selector").map( sel => sel.textContent)
});
Enter fullscreen mode Exit fullscreen mode

As I understand is what Puppeteer gives me is not an HTML element but something else.. ( is there anybody explain a bit easy for me? i am helpless lol)

Anyway, now I get a recommendation for a recipe. yay

6. I learned Heroku

Because my bot will be deactivated when I don't run bot.js, I needed Heroku to serve it on the cloud.

While I am hosting my sites on Netlify, I am still not familiar with the hosting service. I misunderstood Heroku and Netlify as the same kind of service, but this time, I learned they are not. Like this whole post is such lack of professional knowledge, this is my short understanding about Netlify and Heroku so far

  • Netlify is more for the front-end, and Heroku is more for the back-end
  • Netlify is always active, but Heroku fall asleep...zzz
  • Netlify is site hosting online, but Heroku is run the script on Linux
  • Netlify start charging by build time, Heroku charging with something called Dyno hours

Dyno is a container that runs the command in Linux. The busier the app will be, the more dyno is required to be purchased. Because I am a free user, I have a single dyno.

Free user has 1000 free dyno hours. It means that even if my app runs consistently without resting for the whole month, it wouldn't get over 730 hours. Free is always good. It makes me feel cosy.😊


Concluding

Image description

My Git Repo

While I want to improve more and add more features, my Tahani bot works quite wonderfully right now. Great to have a companion! (sad to say that. lol)

I tried a lot of new things this time, I learned so much; however, I also feel like I didn't spend enough time to learn about each of them more precisely. Maybe that's what I'm going to do with next dev.to

Besides having a customised virtual assistant for my daily routine, the knowledge I earned was so valuable. There are so much potential to improve my interactive character project
Thank you for reading this messy post. I am not a trained developer (I a visual artist), but I am very excited to share thoughts and get to know more about creating something cool.

Please comment on anything if you want to correct my post and educate me. It would be awesome to learn from all of you!

Top comments (56)

Collapse
 
lionelrowe profile image
lionel-rowe • Edited

As I understand is what Puppeteer gives me is not an HTML element but something else.. ( is there anybody explain a bit easy for me? i am helpless lol)

The function passed as a callback runs in the context of the headless browser, with all the relevant web APIs available. So within that function, document.querySelector gives a real live DOM element that you can manipulate however you like:

const isItARealDiv = await page.evaluate(() => {
    return document.querySelector("div") instanceof HTMLDivElement
})

isItARealDiv // true
Enter fullscreen mode Exit fullscreen mode

The problem is that when passing the return value back to the parent script, everything is serialized and then deserialized again — something similar to JSON.parse(JSON.serialize(result)). DOM elements can't be properly serialized, getting converted to undefined, so you need to return only the serializable data that you need (text content, specific attributes, inner/outer HTML, etc).

Collapse
 
happping_min profile image
Min

Thank you Lionel-rowe! ( Still confused lol)

Collapse
 
ctsstc profile image
Cody Swartz • Edited

I think Puppeteer and Selenium both suffer from the same problem -- Selenium has been around longer, likely before websites were more complicated with single page magic and dynamic content. The last thing I remembered is that if you query for something before it exists/mounts/renders you'll get nothing back, so you need to wait/poll for it to be available. I thought Puppeteer has helpers around this, or this is why people start to reach for additional libraries on top of these tools to help with this problem. It's been a while since I've touched Puppeteer or Selenium, but I do remember the pains of working with them in single page applications.

Edit: puppeteer.github.io/puppeteer/docs...

Thread Thread
 
lionelrowe profile image
lionel-rowe

@ctsstc yeah, Puppeteer gives you various APIs, such as page.waitForSelector, to deal with that, but it can be finnicky knowing exactly what you need to wait for and avoiding race conditons.

Collapse
 
lionelrowe profile image
lionel-rowe

The parent puppeteer script runs in the Node.JS runtime, whereas the callback to page.evaluate runs in the Chromium browser runtime, headlessly by default ("headless" basically just means that it runs in the background, so you can't visibly see it running). Passing complex data between runtimes is often not possible, because the different runtimes don't know how to interpret it, and DOM elements (DOM is the way the browser interprets HTML) are internally very complex. So to simplify the message passing, Puppeteer uses a serialized format that both runtimes can easily understand. The drawback is that any data that can't be converted to this serialized format is lost.

You can think of "serialized" as meaning something like flat, like a string of letters or binary digits. JSON is a typical serialization format and is useful because it allows the "flattening" of "deep" structures. For example, the JavaScript object { a: { b: 1 } } nests b within a, yet it can be serialized to the JSON string {"a":{"b":1}}. Why is this flat? Well, it's simply the character {, followed by ", followed by a, etc., so it can be read left-to-right; even though the object it represents is a tree structure.

Puppeteer does much of this JSON serialization "under the hood", so you often don't need to worry about it; but JSON can't serialize DOM nodes, because they contain circular structures, e.g. *{ a: { b: *{ a: ... } } } (where * represents a reference to the exact same object). So you need to return only things that JSON can represent — strings, numbers, booleans, null, arrays, and objects containing other JSON-able stuff.

const elementData = await page.evaluate(() => {
    const el = document.querySelector('h1')

    return {
        textContent: el.textContent, // string — OK
        childElementCount: el.childElementCount, // number — OK
        className: el.className, // string — OK
        outerHTML: el.outerHTML, // string — OK
    }
})

console.log(elementData)
// {
//     textContent: 'Posted on Mar 15'
//     childElementCount: 1,
//     className: 'fs-xs color-base-60'
//     outerHTML: '<p class="fs-xs color-base-60">Posted on <time datetime="2022-03-15T02:18:47Z" class="date-no-year" title="Tuesday, March 15, 2022, 2:18:47 AM">Mar 15</time></p>',
// }
Enter fullscreen mode Exit fullscreen mode
Thread Thread
 
happping_min profile image
Min

Omg @lionelrowe
Big thank you to explain this with full of kind detail!
This is really easy to understand! You are the best!👍👍👍👍👍

Collapse
 
joncojonathan profile image
Jonathan Haddock

Thanks for sharing this, I've contemplated making an assistant for years. Sure, I could use Google, but something more custom like you've made is more useful I feel. Maybe when I have some free time (!!!) I'll give something similar a go.

Collapse
 
happping_min profile image
Min

haha I tried siri and google assitant. They sucks because there no way to customize.
I am glad to hear you found this is useful. show your progress when you make any progress. I am sure you can make it better one!

Collapse
 
joelbonetr profile image
JoelBonetR 🥇 • Edited

Love it 😂 i did a discord weekend project in the past and just for fun and this seems more like a month project.
Did you published it? Can i get a link to include it to my discord server? 🤩

Collapse
 
happping_min profile image
Min

It's not published because I didn't set the notion authentification per user.
it would be so embarrasing if people manage their tasks on my notion. lol

Collapse
 
joelbonetr profile image
JoelBonetR 🥇

Hahaha and what about the fun? 😅

Thread Thread
 
happping_min profile image
Min

I will consider to publish it :)

Collapse
 
atulcodex profile image
🚩 Atul Prajapati 🇮🇳

Wow my you are the Tony stark

Collapse
 
happping_min profile image
Min

Actually I considered to name it as Jarvis 😂

Collapse
 
atulcodex profile image
🚩 Atul Prajapati 🇮🇳

that's how we inspire

Collapse
 
timmortal profile image
Timmortal

To make a "Jarvis" that follows you from conception to birth and before you merge back in to the universe, is my dream

Collapse
 
incrementis profile image
Akin C.

Hello Min,

thank you for your article.
I've never looked into implementing chatbots, so this is interesting for me.
I like your creative but simple idea of ​​having a digital companion. I don't think that's sad, it's wise, because like you said
"I am so tired of dealing with all of these.. these shitty uninspiring, repetitive, tedious tasks...".

Collapse
 
happping_min profile image
Min

hehe thank you to saying that! Akin!
very glad to hear i inspire somebody with my idea 🥰

Collapse
 
tqbit profile image
tq-bit

I love the idea. If you're into shares, you might as well give the Yahoon finance API a shot. It gives you 100 free requests per day, that's more than enough for a single person. For instance, I've built a telegram bot earlier that tells me when is a good time to buy & sell

yahoofinanceapi.com/

Collapse
 
happping_min profile image
Min

Ah-ha! that’s nice idea. Maybe i can add that!

Collapse
 
jacksonkasi profile image
Jackson Kasi

wow, that's cool...

Collapse
 
happping_min profile image
Min

Thank you for saying that🥰

Collapse
 
tomaszs2 profile image
Tom Smykowski

Congrats on your project. It is extremely epic. Now please monetize it :-)

Collapse
 
happping_min profile image
Min

Thanks for saying it's epic haha

Collapse
 
tomaszs2 profile image
Tom Smykowski

Hobby money Good money :)

Collapse
 
marissab profile image
Marissa B

Good job on learning so much new stuff! That was a lot of steps and your story from start to finish was clear. I might have to make one of these.

Collapse
 
happping_min profile image
Min

Really fun project, you should try and let me know your progress!

Collapse
 
ashwinv profile image
Ashwin V

Nice, I learned many terms from this post! Keep going!😊

Collapse
 
happping_min profile image
Min

Thank you! 🥰