Well, timeout error may occur in natural way considering that your app crawls websites. You need to catch such errors and handle them appropriately.
I wrote this post because this research helped me to solve problems in my day to day job. This post doesn't have answers for all questions. It can be used as a reference when applied to your specific situation.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
I'd go the easiest path at first - made requests recursively one by one. If performance is not ok then try to make requests concurrent where possible.
I'm making the requests one by one, but I still get the connection reset and timeout errors.
Well, timeout error may occur in natural way considering that your app crawls websites. You need to catch such errors and handle them appropriately.
I wrote this post because this research helped me to solve problems in my day to day job. This post doesn't have answers for all questions. It can be used as a reference when applied to your specific situation.