My first web scraper crashed after 500 requests and I panicked lol

I was building a little Python script to grab movie ratings from a site for a side project, and everything was going smooth until it hit request number 503ish. Suddenly my terminal filled with red errors and I thought I broke the whole internet or something haha. Turns out the site had a rate limit I totally ignored because I'm a total noob at this stuff. I spent like an hour googling HTTP 429 errors before I realized what was going on. Ended up adding a simple time.sleep(2) between requests and it fixed everything. Now I'm curious, what's the dumbest bug you guys have run into that took forever to debug? I feel like I need to know I'm not alone in this lol.

2 comments

2 Comments

cole_walker263d ago

Bro you had the same panic I did when it happened to me lol. I was scraping some sports stats and got hit with a 503 error, thought my IP was banned for life. What finally worked for me was adding a random delay between 1 and 5 seconds plus setting a user agent string. That combo stopped all the errors and the site didn't even notice me.

tessa_hill862d ago

Bro you had the same panic" - relatable af lol