Thanks so much for these tutorials and sharing your work! Question: I ran this exact code for my own instagram data and it has been executing for ~30 minutes now. I have 552 instagram posts. I'm hesitant to kill it but I am unsure if it is stuck. Any ideas?
Unfortunately it's an incredibly slow approach, Instagram starts blocking if you scrape too much too fast so I try to play the long game and let it run in the background.
In the scrape_posts function, you'll see pause=10 which refers to a 10s pause between each post scrape. Considering you have 552 posts, that'll be (552*10)/60 = 92 minutes 😬
In the future, passing silent=False as an argument will print what number the scrape is currently on, I'm actually gonna edit that in right now for anyone else reading the article in the future!
haha thank you! So it did eventually finish without error but then I appeared to have a list of "Post" objects of which I could not tell how I was to get the data from. From reading the GitHub documentation I tried various methods but to no avail (this isn't a knock on you more a knock on my learning curve).
So now after a few hours of messing around I tried to run the "joe biden code" for my own account and even though I am setting login_first=False in the get_posts function, the chrome driver brings me to a login page. Im able to log into instagram but meanwhile my code says it has finished running without error but my posts and scraped_posts objects are now just empty lists.
oh I guess I should also mention that my end goal is to collect data similar to the data you analyzed in your donald trump post. I saw you published a notebook of the analysis code (thank you!) but didn't see a line-by-line on how you got that data.
scraped Post objects contain the scraped data as instance attributes! Try using the to_dict method on one of the Post's and it should return a dictionary with the data it scraped for that Post. The key/values of the returned dict will correspond one-to-one with the available instance attributes
I'll take a look at the login_first bug rn and see if I can replicate it, it might be on the library's end! Instagram has been making a lot of changes the last month or so and have been making it increasingly harder to scrape
ahhh okay, so when you set login_first=False, Instagram is still redirecting to the login page automatically but instascrape is trying to start scrolling immediately which results in an empty list since there are no posts rendered on the page
to access dynamically rendered content like posts you're pretty much always gonna have to be logged in so it's best to leave login_first as True unless you're chaining scrapes and your webdriver is already logged in manually
amazing thank you! So I was able to get my first 10 posts no problem by specifying amount=10 but then I tried to do all ~500 pictures and after 232 pictures I came across this error:
Thanks so much for these tutorials and sharing your work! Question: I ran this exact code for my own instagram data and it has been executing for ~30 minutes now. I have 552 instagram posts. I'm hesitant to kill it but I am unsure if it is stuck. Any ideas?
Unfortunately it's an incredibly slow approach, Instagram starts blocking if you scrape too much too fast so I try to play the long game and let it run in the background.
In the
scrape_posts
function, you'll seepause=10
which refers to a 10s pause between each post scrape. Considering you have 552 posts, that'll be (552*10)/60 = 92 minutes 😬In the future, passing
silent=False
as an argument will print what number the scrape is currently on, I'm actually gonna edit that in right now for anyone else reading the article in the future!Thanks for reaching out!
If it's any consolation though, that means it's working! You're just gonna have to wait an extra hour or so before you can get your data 😬
haha thank you! So it did eventually finish without error but then I appeared to have a list of "Post" objects of which I could not tell how I was to get the data from. From reading the GitHub documentation I tried various methods but to no avail (this isn't a knock on you more a knock on my learning curve).
So now after a few hours of messing around I tried to run the "joe biden code" for my own account and even though I am setting login_first=False in the get_posts function, the chrome driver brings me to a login page. Im able to log into instagram but meanwhile my code says it has finished running without error but my posts and scraped_posts objects are now just empty lists.
oh I guess I should also mention that my end goal is to collect data similar to the data you analyzed in your donald trump post. I saw you published a notebook of the analysis code (thank you!) but didn't see a line-by-line on how you got that data.
scraped
Post
objects contain the scraped data as instance attributes! Try using theto_dict
method on one of thePost
's and it should return a dictionary with the data it scraped for thatPost
. The key/values of the returneddict
will correspond one-to-one with the available instance attributesI'll take a look at the
login_first
bug rn and see if I can replicate it, it might be on the library's end! Instagram has been making a lot of changes the last month or so and have been making it increasingly harder to scrapeahhh okay, so when you set
login_first=False
, Instagram is still redirecting to the login page automatically butinstascrape
is trying to start scrolling immediately which results in an empty list since there are no posts rendered on the pageto access dynamically rendered content like posts you're pretty much always gonna have to be logged in so it's best to leave
login_first
asTrue
unless you're chaining scrapes and your webdriver is already logged in manuallyamazing thank you! So I was able to get my first 10 posts no problem by specifying amount=10 but then I tried to do all ~500 pictures and after 232 pictures I came across this error:
ConnectionError: ('Connection aborted.', OSError("(54, 'ECONNRESET')"))
Im guessing this means instagram blocked my request? Have you come across this issue?