How We Built Perceive: Web Content Extraction for RAG Pipelines

A browser and a language model can look at the same URL and effectively see two different things. A browser sees a rendered interface: navigation, cookie banners, buttons, ads, sidebars, images, scripts, interactive components, and eventually the text a human came to read. A language model sees whatever representation we decide to give it. That distinction matters when the URL is going into a RAG pipeline. Open the developer tools on any major news or documentation site and look at the raw HTML. A typical article page runs between 300KB and 800KB of markup. The article text itself is usually between 2KB and 10KB. The ratio of markup to content is consistently between 10:1 and 40:1 depending on how heavily templated the site is. When you pass raw HTML to a language model, you are passing all of it, and most pipelines treat this as an acceptable default. Perceive is the endpoint we built to fix that. You give it a URL. It returns clean Markdown. This post is about what happens in between and why we made the engineering decisions we did. Why raw HTML is a poor RAG input The token waste is real but it is not the worst problem. Three failure modes compound each other. Token waste . A blog post with 800 words of real content can run to 6,000–12,000 tokens as raw HTML once you include navigation, scripts, inline styles, and layout markup. The same content in Markdown is often 900–1,200 tokens. That is not just a cost issue. It is context window space that cannot go to content. Embedding contamination . Embedding models are trained predominantly on natural language. When you embed a chunk containing <div class="sidebar-widget__title">Related Articles</div> alongside the article content, the vector is pulled toward the markup semantics rather than the content semantics . The embedding does not cleanly represent the article; it represents a mixture of the article and the site's component naming conventions. Retrieval degrades as a result: chunks that should be semantically si