Loading…
6
Finally figured out why my Python script kept crashing after 3 weeks of head scratching
I was building a simple web scraper for a side project on local real estate listings in Portland. Turns out I was using a list comprehension wrong and it was eating up all my memory after about 100 entries. Learned that .append() in a loop is way safer for big datasets than trying to be clever with one-liners. Has anyone else had a basic mistake turn into a huge time sink?
2 comments
Log in to join the discussion
Log In2 Comments
susan4243mo ago
Oh man, that sounds like a rough three weeks. I've definitely been there with stuff like that, where you're sure it's some complex bug and it ends up being the simplest thing. I spent a whole weekend once because I forgot to close a file handle and my script just kept locking up, felt pretty silly after. Your experience with list comprehensions vs loops is a good reminder, I've seen folks go back and forth on that depending on the use case. Hope your scraper is running smooth now.
8
nora1843mo ago
But is it REALLY always simpler to use loops? I've seen @susan424's point about list comprehensions and honestly I think people overstate how "complex" they are. Once you get used to reading them they're WAY faster to write and often easier to scan than a whole separate loop block. That file handle thing though, oh man, that's a classic. I've left a database connection open before and it just silently crashed everything hours later.
1