How to test whether your web extraction API is lying to your agent

The dangerous part of web extraction is not the error. The dangerous part is a clean JSON response that looks correct and is not. If an AI agent uses that output, the mistake does not stay inside a scraper. It moves into a report, a lead list, a price alert, a CRM update, or an automated decision. So before you trust any web extraction API in an agent workflow, test whether it fails honestly. Here is the checklist I use. 1. Test a real page, not a demo page Demo pages are usually polite little museum exhibits. Use pages that behave like the actual internet: a JavaScript-heavy product page a search results page a page with missing fields a login-gated page a blocked or rate-limited page a thin page that has layout but little useful content If the tool only looks good on clean static pages, you have not learned much. 2. Check whether the API separates fetch success from extraction success A page can return HTTP 200 and still be useless. Common examples: the final page is a login screen the visible content is a bot challenge the useful data loads after hydration and never appeared in the fetch the page contains a generic country selector instead of the product the source has navigation text but no actual item data The extractor should not treat these as success just because the request completed. A better response says something like: { "status" : "failed" , "failure_type" : "login_required" , "extracted" : null , "next_step" : "Use a public URL, authorised access, or sample content." } That is boring. Boring is good here. Boring means the agent can stop safely. 3. Ask for fields that are not on the page This is the easiest lie detector. Ask the extractor for something impossible: Extract the product name, price, stock status, warranty length, CEO name, office address, and refund policy. If the page only contains product name and price, the API should say the other fields are missing or low confidence. It should not politely invent them because the schema asked nicely.