标签：#scraping

pluckmd exists so an agent can pull blog posts into markdown, index them into a wiki, and generate interactive HTML to learn from. This post is about the first step, the part with no per-site code, because the design is the interesting bit. If you want the practical side, how I actually use it day to day, I wrote that up separately: https://dev.to/taisei_ide/how-i-use-pluckmd-to-read-blogs-with-an-ai-agent-1jpe It downloads articles from a blog without any per-site code. No handler for Medium, no handler for Substack, nothing keyed on a domain. Here's how that works. The core idea: treat extraction as data, not code. AdapterSpec Instead of branching on which site you're on, pluckmd resolves an AdapterSpec . It's a plain object that says which selector finds article links, what the URL pattern looks like, and how pagination behaves. interface AdapterSpec { listing : ListingExtractionSpec ; // how to find article links article : ArticleExtractionSpec ; // how to pull the body pagination : PaginationSpec ; // none | scroll | button-click | next-url | auto evidence : string ; } Because it's data, the same shape can come from a heuristic, an LLM, an agent, or a person typing it by hand. They all produce the same thing, and they all go through the same checks. Resolving it, cheapest path first cache -> heuristics (local, free) -> LLM (only if needed) Cache first, rechecked against today's DOM so a stale entry can't sneak through. Then local heuristics. The LLM only gets called when the heuristics aren't sure. Every result that works gets written back, so the second run on a site is basically instant. How the heuristics find an article list This part has no idea what site it's looking at. It takes every link, normalizes the path, and collapses the parts that vary into wildcards. / blog / my - first - post -> / blog /* / blog / another - article -> / blog /* / about -> / about Group by that shape. Any group with the same pattern repeated three or more times is a candidate f

Most scraper demos lie by accident. They show the happy path: one URL, one clean page, one neat JSON object. Then the first real user tries a marketplace search page, a login wall, a JavaScript shell, a rate-limited product page, or a site that serves different HTML to every fetch path. The response still comes back as JSON, so everyone relaxes. That is the trap. A JSON response is not the same thing as a useful extraction. The failure mode agents hate AI agents do not just need scraped text. They need to know what happened. Bad extraction output looks like this: { "title" : "Example product" , "price" : "$29.99" , "availability" : "in stock" } That looks fine until you inspect the source and discover the page was a login prompt, a bot challenge, or a thin JavaScript shell. The extractor filled the schema because the schema was requested. Helpful. Like a smoke alarm that hums a little song while the kitchen burns. Better extraction output separates the data from the confidence and the failure class: { "status" : "failed" , "failure_type" : "login_required" , "confidence" : 0.94 , "extracted" : null , "evidence" : { "final_url_type" : "restricted_page" , "visible_content" : "login prompt" , "structured_data_found" : false }, "next_step" : "Use an authorised source, public item URL, feed, API, or sample HTML." } That is less flashy. It is also much more useful. The useful contract is not “scrape anything” “Scrape anything” is usually a warning label wearing lipstick. For agent workflows, the better contract is: Return structured data when the page provides enough evidence. Return a specific honest failure when it does not. Preserve enough metadata for the caller to decide what to do next. Never invent fields just because a prompt asked nicely. This matters for ecommerce, lead enrichment, price monitoring, competitor tracking, procurement, and internal research agents. If the agent cannot tell the difference between “product unavailable”, “page blocked”, “login require

Strava will charge a flat monthly fee from developers to access its API

The row looked perfect. rating: 7 . Valid JSON, right type, no nulls, no missing keys. My schema check waved it through. The page had returned HTTP 200. The selectors hadn't moved. Everything green. A rating of 7 on a 5-star site is impossible. The model invented it, formatted it correctly, and handed it to me with total confidence. That's the failure I want to talk about. Not the scraper that breaks loudly. The one that hands you a clean-looking row that is quietly, plausibly false — and sails past every check you have, because your checks are all looking at the shape of the data, and the lie is in the value . TL;DR HTTP 200, intact selectors, and valid JSON tell you the form is fine. They say nothing about whether the value is true. When an LLM extracts from messy free-text, structured-output mode guarantees you get valid JSON. It does not guarantee the content is real. The model fills uncertain fields rather than leaving them empty — because the schema demands a complete row. A ~60-line value-level sanity gate (ranges, dates, cross-field, reference, language) catches the obvious lies before they hit your database. Real code and real output below. The honest catch: this gate catches rule violations , not plausible lies inside the allowed range . A rating: 4 where the truth is 2 slides right through. I'll be specific about where the gate stops. Two different ways a scraper lies to you I wrote about source drift last week — the case where the page changes underneath you and a 30-line schema check catches the structure shifting. That's an input problem. The source mutated; your agreement with the page broke; you detect it by watching the shape. This is the other end of the pipe. The source is fine. The page is intact, the selectors are correct, the structure is exactly what you expected. The thing that lied to you is the model , on the extraction step, when you asked it to pull structured fields out of a paragraph of human prose. Those two failures feel similar and t

I built a small web scraping framework in Rust, mostly with an AI doing the typing. It's called ferrous — a Colly-style collector: register CSS selector callbacks, queue URLs, write JSONL. About 700 lines. The pitch I kept hearing, and half-believed, was that Rust and LLMs are a good match now: the borrow checker is a correctness oracle the model can lean on, so the class of bugs that plagues AI-written Python just won't compile. That's true. It's also where the story gets uncomfortable, because the build was green and the code was still wrong. How I worked I'm not a Rust native. ferrous was partly an excuse to get fluent — build something real instead of reading about lifetimes — and partly a test of how far an LLM could carry the typing while I drove. The loop was plain: describe the next change in English, let the model write the Rust, read what came back, run cargo , move on. It kept observations.md as a running design journal, one entry per change, each with a short rationale for the decision it made. That setup has a soft spot, and it's the whole point of this post. When you drive a language you don't fully know, the only reviewer you've got with real authority is the compiler. Everything past that — is this idiomatic, is it the right abstraction, does it actually do what the journal claims — depends on already knowing what correct looks like. Which is exactly the knowledge a learner doesn't have yet. Keep that in mind through the next part, because it's the difference between the one bug I could have caught and the six I couldn't. The bug that the toolchain told me wasn't there I ship two fetch backends. The default goes through the Zyte API; an optional one, gated behind a wreq feature, makes direct requests with browser TLS emulation. Each has an example. After a refactor that added URL resolution — ctx.resolve_and_visit(href) so callbacks stop hand-building absolute URLs — I had the model update the examples to use it. It did, and it wrote up the change in

Quick answer: The Google Ads Transparency Center is a public registry of every ad Google runs — but it ships no API and no bulk export . To get the data programmatically you scrape it. A Google Ads Transparency scraper sends the same RPC call the website uses and returns every ad creative for an advertiser as structured JSON. The Apify Actor below does it for $0.0012 per ad (~$1.20 per 1,000), with the TLS fingerprinting, proxy rotation, and pagination handled for you. Google's Ads Transparency Center is one of the most underused datasets in marketing. Launched in 2023 under the EU Digital Services Act and parallel US pressure, it indexes every ad campaign currently running on Search, YouTube, Display, Shopping, Maps, and Play — keyed by advertiser. Google's own counter lists 300,000+ active creatives for a brand like Nike . For your nearest competitor, it's usually 50–500. The catch: there's no download button. Just an interactive UI that paginates 40 creatives at a time. If you want this as a CSV — for a competitor sweep, a trademark audit, or a RAG corpus — you have to extract it yourself. Here's what that actually takes, and how I shortened it to one API call. What is the Google Ads Transparency Center? 🔎 The Google Ads Transparency Center is a public, Google-operated registry that shows the ad creatives any verified advertiser is running, the date range each ad was shown, and roughly where. Google built it to comply with ad-disclosure regulation, so the data is public by design — you're reading the same registry a regulator would. What it gives you per advertiser: Every ad creative currently or recently live (text, image, video) The landing domain each ad clicks through to First-shown / last-shown timestamps and a rough impression count A deep link to each creative inside the Transparency Center What it does not give you: a search-by-keyword mode, region-filtered results from the server, or — crucially — an API. Does the Google Ads Transparency Center have an A

I came across Scrapling through a recommendation on X and decided to put it through its paces — not against a demo page, but against Lazada Singapore, a production site with Google reCAPTCHA and a custom slider verification. The setup: a single 4GB VPS, no residential proxies, no credits, just open-source tools. Here's the full journey: installation pitfalls, wiring it into an AI agent, choosing the right browser for the job, and the real-world benchmarks that followed. What Is Scrapling? Scrapling is an adaptive web scraping framework for Python (BSD-3, v0.4.8). It handles everything from single HTTP requests to full-scale concurrent crawls. What sets it apart from the BeautifulSoup/Scrapy world: Adaptive element tracking — saves fingerprints of targeted elements and relocates them after site redesigns using similarity scoring. Your scrapers survive CSS changes without maintenance. Three fetchers, one API — HTTP ( Fetcher , curl_cffi), browser ( DynamicFetcher , Playwright Chromium), and stealth ( StealthyFetcher , Chromium + anti-bot patches). Swap with one line. Spider framework — Scrapy-like API with async, concurrent crawling, Ctrl+C pause/resume via checkpoint persistence, multi-session support. MCP server — 14 tools exposed natively for AI coding agents. Your agent can call mcp_scrapling_get , mcp_scrapling_fetch , mcp_scrapling_stealthy_fetch directly. It's open source, pip-installable, and designed to be the backbone of a scraping stack — not just another tool in the toolbox. Installation on a 4GB VPS This is where the real story starts. The VPS has 4GB RAM, 2 vCPUs, 77GB disk, and runs an AI agent gateway (615MB baseline). Every browser installation decision matters. What we installed pip install scrapling[fetchers,ai] # HTTP + Chromium + MCP server scrapling install # Downloads Playwright browsers This pulls in Playwright Chromium, Firefox, and WebKit (~1.3GB disk), plus curl_cffi for HTTP requests and patchright (Playwright fork) for browser automation.