Working alone is great... but... I am so tired of dealing with all of these.. these shitty uninspiring, repetitive, tedious tasks...
I didn't real...
For further actions, you may consider blocking this person and/or reporting abuse
The function passed as a callback runs in the context of the headless browser, with all the relevant web APIs available. So within that function,
document.querySelector
gives a real live DOM element that you can manipulate however you like:The problem is that when passing the return value back to the parent script, everything is serialized and then deserialized again — something similar to
JSON.parse(JSON.serialize(result))
. DOM elements can't be properly serialized, getting converted toundefined
, so you need to return only the serializable data that you need (text content, specific attributes, inner/outer HTML, etc).Thank you Lionel-rowe! ( Still confused lol)
I think Puppeteer and Selenium both suffer from the same problem -- Selenium has been around longer, likely before websites were more complicated with single page magic and dynamic content. The last thing I remembered is that if you query for something before it exists/mounts/renders you'll get nothing back, so you need to wait/poll for it to be available. I thought Puppeteer has helpers around this, or this is why people start to reach for additional libraries on top of these tools to help with this problem. It's been a while since I've touched Puppeteer or Selenium, but I do remember the pains of working with them in single page applications.
Edit: puppeteer.github.io/puppeteer/docs...
@ctsstc yeah, Puppeteer gives you various APIs, such as
page.waitForSelector
, to deal with that, but it can be finnicky knowing exactly what you need to wait for and avoiding race conditons.The parent puppeteer script runs in the Node.JS runtime, whereas the callback to
page.evaluate
runs in the Chromium browser runtime, headlessly by default ("headless" basically just means that it runs in the background, so you can't visibly see it running). Passing complex data between runtimes is often not possible, because the different runtimes don't know how to interpret it, and DOM elements (DOM is the way the browser interprets HTML) are internally very complex. So to simplify the message passing, Puppeteer uses a serialized format that both runtimes can easily understand. The drawback is that any data that can't be converted to this serialized format is lost.You can think of "serialized" as meaning something like flat, like a string of letters or binary digits. JSON is a typical serialization format and is useful because it allows the "flattening" of "deep" structures. For example, the JavaScript object
{ a: { b: 1 } }
nestsb
withina
, yet it can be serialized to the JSON string{"a":{"b":1}}
. Why is this flat? Well, it's simply the character{
, followed by"
, followed bya
, etc., so it can be read left-to-right; even though the object it represents is a tree structure.Puppeteer does much of this JSON serialization "under the hood", so you often don't need to worry about it; but JSON can't serialize DOM nodes, because they contain circular structures, e.g.
*{ a: { b: *{ a: ... } } }
(where*
represents a reference to the exact same object). So you need to return only things that JSON can represent — strings, numbers, booleans, null, arrays, and objects containing other JSON-able stuff.Omg @lionelrowe
Big thank you to explain this with full of kind detail!
This is really easy to understand! You are the best!👍👍👍👍👍
Thanks for sharing this, I've contemplated making an assistant for years. Sure, I could use Google, but something more custom like you've made is more useful I feel. Maybe when I have some free time (!!!) I'll give something similar a go.
haha I tried siri and google assitant. They sucks because there no way to customize.
I am glad to hear you found this is useful. show your progress when you make any progress. I am sure you can make it better one!
Wow my you are the Tony stark
Actually I considered to name it as Jarvis 😂
that's how we inspire
To make a "Jarvis" that follows you from conception to birth and before you merge back in to the universe, is my dream
Hello Min,
thank you for your article.
I've never looked into implementing chatbots, so this is interesting for me.
I like your creative but simple idea of having a digital companion. I don't think that's sad, it's wise, because like you said
"I am so tired of dealing with all of these.. these shitty uninspiring, repetitive, tedious tasks...".
hehe thank you to saying that! Akin!
very glad to hear i inspire somebody with my idea 🥰
Love it 😂 i did a discord weekend project in the past and just for fun and this seems more like a month project.
Did you published it? Can i get a link to include it to my discord server? 🤩
It's not published because I didn't set the notion authentification per user.
it would be so embarrasing if people manage their tasks on my notion. lol
Hahaha and what about the fun? 😅
I will consider to publish it :)
I love the idea. If you're into shares, you might as well give the Yahoon finance API a shot. It gives you 100 free requests per day, that's more than enough for a single person. For instance, I've built a telegram bot earlier that tells me when is a good time to buy & sell
yahoofinanceapi.com/
Ah-ha! that’s nice idea. Maybe i can add that!
Congrats on your project. It is extremely epic. Now please monetize it :-)
Thanks for saying it's epic haha
Hobby money Good money :)
Good job on learning so much new stuff! That was a lot of steps and your story from start to finish was clear. I might have to make one of these.
Really fun project, you should try and let me know your progress!
Nice, I learned many terms from this post! Keep going!😊
Thank you! 🥰
Damn, this is an amazing project! Great work. Amazing discord bot!
Thank you for saying that🥰
Wow, I really enjoyed reading through your article! Great work on the project, this is pretty impressive! Thanks for sharing @happping_min !
Thank you very much Andras!😊
Super impressive feat! It’s one thing to build something new; it’s a whole other thing to learn several libraries/systems in the process. Great work, never stop learning! 😇
hehe thank you very much Dan!
Hell I love this idea, now I need to implement it in Python and slack
Thank you very much! I want to see your version! please share the progress! 😊
This is a great project! And I've been meaning to make a discord bot and you've really broken the process down quite well.
Haha thanks! Great to hear my article helps others!
wow, that's cool...
Thank you for saying that🥰
Thank you for this wonderful post. I planned on learning Discord.js to build a personal bot eventually, and I'm so glad I discovered this!
Please show what you make 😀
Impressive, to say the least!
Thank you!
This was an inspiring and well written article. Thank you!
Thank you very much Bill!
This project is incredible!
Thank you for saying that!🤗
Thanks Leonid! Yeah I like what I am doing haha
This is great stuff! I love the way you approached an under-constrained problem and found a value-added approach that contributed real help to your workflow.
One thing I found when going through your steps was, of course, that different developers might want different things from an automated assistant. How did you decide what you wanted an "assistant" to do for you--that is, what were your objectives? Are there specific tedious things you seek to automate for your own workflow versus collaborative tools that support team-based efforts?
Two thumbs up!
What node js library did you use for NPL?
this!
npmjs.com/package/node-wit
1- great work, you built it using different languages than we did.
2-i see photo of that indian hot chick. dream about it to be your assitant. :)
3- we did it differently, ajax, js, prolog, aiml used in our bot. she has intelligence
check it out in our website. lots of videos there.
4- send me a link to go and talk with her to see how she is doing.
email: support at robot-hosting.com
have a happy time
Moses
What a cool concept!
Wow. Impressive. I was going to ask if you tried Character.AI for this but 'duh' it isn't meant for this type of thing. But, wow, this is cool!
Thank you as well Lionel-rowe.
Thank You