标签：#webscraping

I built a small web scraping framework in Rust, mostly with an AI doing the typing. It's called ferrous — a Colly-style collector: register CSS selector callbacks, queue URLs, write JSONL. About 700 lines. The pitch I kept hearing, and half-believed, was that Rust and LLMs are a good match now: the borrow checker is a correctness oracle the model can lean on, so the class of bugs that plagues AI-written Python just won't compile. That's true. It's also where the story gets uncomfortable, because the build was green and the code was still wrong. How I worked I'm not a Rust native. ferrous was partly an excuse to get fluent — build something real instead of reading about lifetimes — and partly a test of how far an LLM could carry the typing while I drove. The loop was plain: describe the next change in English, let the model write the Rust, read what came back, run cargo , move on. It kept observations.md as a running design journal, one entry per change, each with a short rationale for the decision it made. That setup has a soft spot, and it's the whole point of this post. When you drive a language you don't fully know, the only reviewer you've got with real authority is the compiler. Everything past that — is this idiomatic, is it the right abstraction, does it actually do what the journal claims — depends on already knowing what correct looks like. Which is exactly the knowledge a learner doesn't have yet. Keep that in mind through the next part, because it's the difference between the one bug I could have caught and the six I couldn't. The bug that the toolchain told me wasn't there I ship two fetch backends. The default goes through the Zyte API; an optional one, gated behind a wreq feature, makes direct requests with browser TLS emulation. Each has an example. After a refactor that added URL resolution — ctx.resolve_and_visit(href) so callbacks stop hand-building absolute URLs — I had the model update the examples to use it. It did, and it wrote up the change in

Quick answer: The Google Ads Transparency Center is a public registry of every ad Google runs — but it ships no API and no bulk export . To get the data programmatically you scrape it. A Google Ads Transparency scraper sends the same RPC call the website uses and returns every ad creative for an advertiser as structured JSON. The Apify Actor below does it for $0.0012 per ad (~$1.20 per 1,000), with the TLS fingerprinting, proxy rotation, and pagination handled for you. Google's Ads Transparency Center is one of the most underused datasets in marketing. Launched in 2023 under the EU Digital Services Act and parallel US pressure, it indexes every ad campaign currently running on Search, YouTube, Display, Shopping, Maps, and Play — keyed by advertiser. Google's own counter lists 300,000+ active creatives for a brand like Nike . For your nearest competitor, it's usually 50–500. The catch: there's no download button. Just an interactive UI that paginates 40 creatives at a time. If you want this as a CSV — for a competitor sweep, a trademark audit, or a RAG corpus — you have to extract it yourself. Here's what that actually takes, and how I shortened it to one API call. What is the Google Ads Transparency Center? 🔎 The Google Ads Transparency Center is a public, Google-operated registry that shows the ad creatives any verified advertiser is running, the date range each ad was shown, and roughly where. Google built it to comply with ad-disclosure regulation, so the data is public by design — you're reading the same registry a regulator would. What it gives you per advertiser: Every ad creative currently or recently live (text, image, video) The landing domain each ad clicks through to First-shown / last-shown timestamps and a rough impression count A deep link to each creative inside the Transparency Center What it does not give you: a search-by-keyword mode, region-filtered results from the server, or — crucially — an API. Does the Google Ads Transparency Center have an A

I came across Scrapling through a recommendation on X and decided to put it through its paces — not against a demo page, but against Lazada Singapore, a production site with Google reCAPTCHA and a custom slider verification. The setup: a single 4GB VPS, no residential proxies, no credits, just open-source tools. Here's the full journey: installation pitfalls, wiring it into an AI agent, choosing the right browser for the job, and the real-world benchmarks that followed. What Is Scrapling? Scrapling is an adaptive web scraping framework for Python (BSD-3, v0.4.8). It handles everything from single HTTP requests to full-scale concurrent crawls. What sets it apart from the BeautifulSoup/Scrapy world: Adaptive element tracking — saves fingerprints of targeted elements and relocates them after site redesigns using similarity scoring. Your scrapers survive CSS changes without maintenance. Three fetchers, one API — HTTP ( Fetcher , curl_cffi), browser ( DynamicFetcher , Playwright Chromium), and stealth ( StealthyFetcher , Chromium + anti-bot patches). Swap with one line. Spider framework — Scrapy-like API with async, concurrent crawling, Ctrl+C pause/resume via checkpoint persistence, multi-session support. MCP server — 14 tools exposed natively for AI coding agents. Your agent can call mcp_scrapling_get , mcp_scrapling_fetch , mcp_scrapling_stealthy_fetch directly. It's open source, pip-installable, and designed to be the backbone of a scraping stack — not just another tool in the toolbox. Installation on a 4GB VPS This is where the real story starts. The VPS has 4GB RAM, 2 vCPUs, 77GB disk, and runs an AI agent gateway (615MB baseline). Every browser installation decision matters. What we installed pip install scrapling[fetchers,ai] # HTTP + Chromium + MCP server scrapling install # Downloads Playwright browsers This pulls in Playwright Chromium, Firefox, and WebKit (~1.3GB disk), plus curl_cffi for HTTP requests and patchright (Playwright fork) for browser automation.