You are wrong about WWW::Curl when it comes to forking {remember forking is really bad if the codebase is large!}- it has it's own way of doing multiple requests which is very good - as it is much easier to write a dynamically queued request chain. I use this for parallelised web crawling - each request gets handled while other requests are being fetched, and can add more URLs to the search string.
This is not so easy in the methods you are using.... If you are using Anyevent it works in a similar way (but again not as efficiently) I think than WWW::Curl::Multi.
WWW::Curl::Multi is a bit harder to use - but with a few lines of wrapper script modules this is resolved nicely...
The above should all happen within the same process, so I’m not sure where you’re coming from regarding forking.
WWW::Curl::Multi, as far as I can tell, uses curl’s internal select-based event loop exclusively, which won’t be as efficient as platform-native polling methods like epoll or kqueue. It appears to mimic the C API pretty closely, much like Net::Curl::Multi.
Do you mind clarifying what you see as “not so easy” regarding adding more URLs to the request queue?
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
You are wrong about WWW::Curl when it comes to forking {remember forking is really bad if the codebase is large!}- it has it's own way of doing multiple requests which is very good - as it is much easier to write a dynamically queued request chain. I use this for parallelised web crawling - each request gets handled while other requests are being fetched, and can add more URLs to the search string.
This is not so easy in the methods you are using.... If you are using Anyevent it works in a similar way (but again not as efficiently) I think than WWW::Curl::Multi.
WWW::Curl::Multi is a bit harder to use - but with a few lines of wrapper script modules this is resolved nicely...
Hi!
The above should all happen within the same process, so I’m not sure where you’re coming from regarding forking.
WWW::Curl::Multi, as far as I can tell, uses curl’s internal select-based event loop exclusively, which won’t be as efficient as platform-native polling methods like epoll or kqueue. It appears to mimic the C API pretty closely, much like Net::Curl::Multi.
Do you mind clarifying what you see as “not so easy” regarding adding more URLs to the request queue?