Ive been following your posts closely in regards to instascrape, a great little library! Just wondered if you have hit any issues yet with instagram updating their layout / adding in any new blocking mechanisms to your scraper since working on it so far? I'm kind of thinking along the lines of how long is the shelf life for a scraper before some new breaking change comes along (ideally they want to push users into their API)
Funny that you mention it, today was the first day after about two months of the lib's existence that I had to fix something because of a change on Instagram's end lol, I kept getting hit with 429 status codes on every request I made. I kind of figured something like this was going to happen eventually because I wasn't passing any header info with the requests; I quickly added support for passing default/custom header info though and now it's back up and running like a charm
One of the driving factors in design choice since day one has been to account for a changing Instagram API as well as the tightening of restrictions that Instagram has been trending towards. I'm hoping I'll be able to roll with the punches as they come and continue to float under their radar lol. I deliberately excluded selenium and any sort of interaction with Instagram content to avoid their wrath as much as possible so we'll see how it goes 😅
Thanks for following and asking! With the library in a comfortably stable place and no major internal design changes in the near future, I'm ready to go back and fix up some of the stuff I was kind of neglecting (i.e. missing headers, fine tuning with arguments, etc.)
Nice to see it wasn't anything catastrophic 👍 I'll be likely giving this a go to monitor some insta accounts for new posts and publishing them via my telegram bot into a chat. I've been considering running through lumintai.io as I do with twitter and YouTube which has served well to proxy from multiple locations so the traffic on a single IP doesn't stack up I'll get a tutorial up on Dev if it works out 👍
Awesome! Would love to see that tutorial, I'll keep an eye out. I wrote a script a couple months ago that rotates free anonymous proxies but kept getting hit with 403's, probably because the IP's are blacklisted since everyone else is using them lol. Haven't done too much more research into proxies since I haven't really needed it yet but I plan future versions of instascrape to have support for it; it's definitely a vital tool for any large scale scraping
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Ive been following your posts closely in regards to instascrape, a great little library! Just wondered if you have hit any issues yet with instagram updating their layout / adding in any new blocking mechanisms to your scraper since working on it so far? I'm kind of thinking along the lines of how long is the shelf life for a scraper before some new breaking change comes along (ideally they want to push users into their API)
Funny that you mention it, today was the first day after about two months of the lib's existence that I had to fix something because of a change on Instagram's end lol, I kept getting hit with 429 status codes on every request I made. I kind of figured something like this was going to happen eventually because I wasn't passing any header info with the requests; I quickly added support for passing default/custom header info though and now it's back up and running like a charm
One of the driving factors in design choice since day one has been to account for a changing Instagram API as well as the tightening of restrictions that Instagram has been trending towards. I'm hoping I'll be able to roll with the punches as they come and continue to float under their radar lol. I deliberately excluded
seleniumand any sort of interaction with Instagram content to avoid their wrath as much as possible so we'll see how it goes 😅Thanks for following and asking! With the library in a comfortably stable place and no major internal design changes in the near future, I'm ready to go back and fix up some of the stuff I was kind of neglecting (i.e. missing headers, fine tuning with arguments, etc.)
Nice to see it wasn't anything catastrophic 👍 I'll be likely giving this a go to monitor some insta accounts for new posts and publishing them via my telegram bot into a chat. I've been considering running through lumintai.io as I do with twitter and YouTube which has served well to proxy from multiple locations so the traffic on a single IP doesn't stack up I'll get a tutorial up on Dev if it works out 👍
Awesome! Would love to see that tutorial, I'll keep an eye out. I wrote a script a couple months ago that rotates free anonymous proxies but kept getting hit with 403's, probably because the IP's are blacklisted since everyone else is using them lol. Haven't done too much more research into proxies since I haven't really needed it yet but I plan future versions of
instascrapeto have support for it; it's definitely a vital tool for any large scale scraping