Stop pretending your scraper worked: honest JSON for AI agents

Most scraper demos lie by accident. They show the happy path: one URL, one clean page, one neat JSON object. Then the first real user tries a marketplace search page, a login wall, a JavaScript shell, a rate-limited product page, or a site that serves different HTML to every fetch path. The response still comes back as JSON, so everyone relaxes. That is the trap. A JSON response is not the same thing as a useful extraction. The failure mode agents hate AI agents do not just need scraped text. They need to know what happened. Bad extraction output looks like this: { "title" : "Example product" , "price" : "$29.99" , "availability" : "in stock" } That looks fine until you inspect the source and discover the page was a login prompt, a bot challenge, or a thin JavaScript shell. The extractor filled the schema because the schema was requested. Helpful. Like a smoke alarm that hums a little song while the kitchen burns. Better extraction output separates the data from the confidence and the failure class: { "status" : "failed" , "failure_type" : "login_required" , "confidence" : 0.94 , "extracted" : null , "evidence" : { "final_url_type" : "restricted_page" , "visible_content" : "login prompt" , "structured_data_found" : false }, "next_step" : "Use an authorised source, public item URL, feed, API, or sample HTML." } That is less flashy. It is also much more useful. The useful contract is not “scrape anything” “Scrape anything” is usually a warning label wearing lipstick. For agent workflows, the better contract is: Return structured data when the page provides enough evidence. Return a specific honest failure when it does not. Preserve enough metadata for the caller to decide what to do next. Never invent fields just because a prompt asked nicely. This matters for ecommerce, lead enrichment, price monitoring, competitor tracking, procurement, and internal research agents. If the agent cannot tell the difference between “product unavailable”, “page blocked”, “login require