Classic scraping breaks the moment a site redesigns a class name. So instead of hand-rolling CSS selectors per site, I built a scraping engine that renders the page headlessly and hands the content to an LLM for structured extraction — the model reads the page roughly the way a human would, which makes it far less brittle to markup churn.
The engineering problems that actually mattered weren't the extraction itself — they were the boring operational stuff: batching requests so one slow site doesn't stall the whole run, per-URL checkpoints so a crash doesn't mean starting over, a fail-fast deadline per site so a hung page doesn't eat the whole job, and listing-level filters so you're not paying LLM tokens to extract pages you don't need.
Running this against hundreds of real listings in one sitting exposed the next bottleneck fast: a synchronous request/response loop can't carry that kind of volume. The fix is background execution with progress you can poll — which turned out to be a more interesting problem than the scraping itself.