标签：#pagination

A scraper can pass every check you wrote and still be wrong about the one thing you actually care about: how much it collected. No exception. No 500. No broken row. Exit code 0, logs green, every field valid. And the set on disk is a quarter of what the site actually has. I have run scrapers in production enough times to stop trusting a green run on its own, and this is the failure that taught me to count. TL;DR A paginated source can serve fewer rows than it claims and never throw — page caps, hidden offset limits, infinite scroll that "ends" early. Your status check (200), schema check (valid row), and byte check (you got data) all pass. None of them counts records. The tell: declared total vs unique ids collected. Or, when there's no declared total, the page that quietly repeats an earlier page. Below is a 40-line probe you can run right now. On a source that caps at 1,500 of a declared 4,000, it returned VERDICT: INCOMPLETE (missing 2500 rows) . This is a completeness check, not a correctness check. Different layer, different bug. What actually goes wrong You write the loop everyone writes. Walk ?page=1 , ?page=2 , keep going until a page comes back empty. Stop. Save. Done. The source has other plans. It says it has 4,000 records — the count is right there in the envelope, or in a "Showing 4,000 results" line in the HTML. But it only ever hands out real data for the first 30 pages. Page 31 doesn't error. It doesn't return empty either. It returns page 1 again. Still HTTP 200. Still 50 valid rows. Your loop has no reason to stop, so it grinds on until its own page budget runs out, collects a pile of rows, and exits clean. You now have 5,000 rows in hand and feel great about it. Looks like plenty. The catch: only 1,500 are unique. The page cap fed you the same first page over and over, and those duplicates hid the shortfall behind a big-looking row count. That is the exact shape of "50 rows passed every check while 4,000 existed" — the scraper saw a lot of rows an