Finally figured out why my Python script kept crashing after 3 weeks of head scratching

I was building a simple web scraper for a side project on local real estate listings in Portland. Turns out I was using a list comprehension wrong and it was eating up all my memory after about 100 entries. Learned that .append() in a loop is way safer for big datasets than trying to be clever with one-liners. Has anyone else had a basic mistake turn into a huge time sink?

2 comments

2 Comments

susan4242d ago

Oh man, that sounds like a rough three weeks. I've definitely been there with stuff like that, where you're sure it's some complex bug and it ends up being the simplest thing. I spent a whole weekend once because I forgot to close a file handle and my script just kept locking up, felt pretty silly after. Your experience with list comprehensions vs loops is a good reminder, I've seen folks go back and forth on that depending on the use case. Hope your scraper is running smooth now.

nora1842d ago

But is it REALLY always simpler to use loops? I've seen @susan424's point about list comprehensions and honestly I think people overstate how "complex" they are. Once you get used to reading them they're WAY faster to write and often easier to scan than a whole separate loop block. That file handle thing though, oh man, that's a classic. I've left a database connection open before and it just silently crashed everything hours later.