Generating scraper logic at runtime instead of writing it per site

pluckmd exists so an agent can pull blog posts into markdown, index them into a wiki, and generate interactive HTML to learn from. This post is about the first step, the part with no per-site code, because the design is the interesting bit. If you want the practical side, how I actually use it day to day, I wrote that up separately: https://dev.to/taisei_ide/how-i-use-pluckmd-to-read-blogs-with-an-ai-agent-1jpe It downloads articles from a blog without any per-site code. No handler for Medium, no handler for Substack, nothing keyed on a domain. Here's how that works. The core idea: treat extraction as data, not code. AdapterSpec Instead of branching on which site you're on, pluckmd resolves an AdapterSpec . It's a plain object that says which selector finds article links, what the URL pattern looks like, and how pagination behaves. interface AdapterSpec { listing : ListingExtractionSpec ; // how to find article links article : ArticleExtractionSpec ; // how to pull the body pagination : PaginationSpec ; // none | scroll | button-click | next-url | auto evidence : string ; } Because it's data, the same shape can come from a heuristic, an LLM, an agent, or a person typing it by hand. They all produce the same thing, and they all go through the same checks. Resolving it, cheapest path first cache -> heuristics (local, free) -> LLM (only if needed) Cache first, rechecked against today's DOM so a stale entry can't sneak through. Then local heuristics. The LLM only gets called when the heuristics aren't sure. Every result that works gets written back, so the second run on a site is basically instant. How the heuristics find an article list This part has no idea what site it's looking at. It takes every link, normalizes the path, and collapses the parts that vary into wildcards. / blog / my - first - post -> / blog /* / blog / another - article -> / blog /* / about -> / about Group by that shape. Any group with the same pattern repeated three or more times is a candidate f