标签：#scraping

If you have written more than a couple of scrapers, you already know the pattern. The first few hundred requests fly through. Then responses slow down, you start seeing 429 Too Many Requests , a captcha wall appears, and finally the target just returns empty pages or a hard 403 . Your code did not change. Your IP did. Scraping at any real volume is less about parsing HTML and more about managing where your requests come from. This post is a practical walk-through of how proxies fit into a scraping pipeline: why a single IP fails, what proxy types actually matter, how rotation works, and how to wire it all up in Python with requests , aiohttp , and Scrapy. There is code you can copy, plus the mistakes that cost me the most time. Why one IP is never enough Every site you scrape sees the same thing: a stream of requests from one address, arriving faster and more regularly than a human ever would. Anti-bot systems are built to spot exactly that. The signals they use are boring but effective: Request rate per IP. Too many hits in a short window trips a rate limiter. Volume over time. Even a slow scraper eventually stands out if every request comes from the same address for hours. Behavioral fingerprint. No mouse, no scroll, identical headers, requests in perfect intervals. Reputation. Datacenter ranges that have been abused before are pre-flagged. You can soften some of these with headers, delays, and a real browser, but there is a ceiling. Once a single IP has made enough requests, it gets throttled or blocked regardless of how polite you are. The only way past that ceiling is to spread requests across many addresses, so no single one crosses the threshold. That is the entire job of a proxy pool. The proxy landscape, minus the marketing Providers love to complicate this. For scraping, the distinctions that actually change your results are these: Shared vs private. Shared proxies are handed to many customers at once. You inherit everyone else's behavior, so an address ca

A scraper fails, someone swaps the User-Agent, someone else adds a proxy, then the job starts passing locally but fails again in CI. That usually happens because Cloudflare did not block “scraping” as one thing. It evaluated several signals, and each failure needs a different fix. This is about authorized automation: your own sites, customer-approved workflows, testing, monitoring, data access you are allowed to perform. If you do not have permission to automate against a site, changing fingerprints or rotating IPs does not make it okay. Start with the failure you actually see Cloudflare failures often get collapsed into “403”, but the page body matters. Common cases: Error 1020 : usually an access denied page from a Cloudflare rule or bot score decision. The HTTP status may still be 403, so inspect the HTML. 403 without a 1020 page : often IP reputation, firewall rules, geo restrictions, or an auth problem. 429 : rate limit exhaustion. Slowing down can help here, but it will not fix a fingerprint problem. Endless Just a moment... page : your client did not complete the browser-side challenge. CAPTCHA or Turnstile loop : Cloudflare still considers the session borderline after earlier checks. Add classification before you add workarounds. Even a basic classifier saves time: import time import requests CLOUDFLARE_MARKERS = { " 1020 " : " cloudflare_access_denied " , " Just a moment " : " cloudflare_js_challenge " , " cf-turnstile " : " cloudflare_turnstile " , " cf-error-code " : " cloudflare_error_page " , } def classify_response ( resp : requests . Response ) -> str : body = resp . text [: 5000 ] if resp . status_code == 429 : return " rate_limited " for marker , label in CLOUDFLARE_MARKERS . items (): if marker in body : return label if resp . status_code == 403 : return " forbidden_unknown " return " ok " if resp . ok else f " http_ { resp . status_code } " def get_with_backoff ( url : str , max_attempts = 4 ): for attempt in range ( max_attempts ): resp = request

Your customers have stopped Googling "best note-taking app." They're asking ChatGPT, Perplexity, and Gemini instead — and getting back a short list of three or four products. If your brand isn't on that list, you're invisible, and unlike a Google ranking you can't even see where you stand. That's the problem I set out to measure. This post is the build breakdown: five AI answer engines, one uniform result shape, a mention-detection core that doesn't lie to you, and the honest gotchas I hit around cost and billing. The whole thing runs as a paid Apify Actor written in async Python. The niche has a name now — GEO (Generative Engine Optimization) or AEO (Answer Engine Optimization). Think SEO, but the search engine is a language model and the "ranking" is whether you get named in the answer. The core question Give the tool a brand, its competitors, and the buyer-intent questions your customers actually type: { "brand" : "Notion" , "competitors" : [ "Obsidian" , "Coda" , "Evernote" ], "prompts" : [ "best note taking app for students" , "Notion vs Obsidian which should I use" ], "engines" : [ "perplexity" , "chatgpt" , "gemini" , "claude" , "aiOverview" ], "samplesPerPrompt" : 3 } It asks each engine each prompt (several times, because LLM answers vary run-to-run), then analyzes every answer for: were you mentioned, how early, were you recommended or just listed, what's the sentiment, who else got named, and — the part incumbents skip — which domains each engine cited. That last one is the actionable output: it tells you which websites the AI trusts for your category, i.e. where you need coverage. Architecture: one shape to rule them all The trick that keeps the whole thing sane is that every engine adapter — whether it's a clean REST API or a messy HTML scrape — returns the exact same record shape : { " engine " : " perplexity " , " prompt " : " best note taking app for students " , " sampleIndex " : 1 , " responseText " : " ... " , " citations " : [{ " url " : " ... "

If you've ever tried to pull job listings by scraping LinkedIn or Indeed, you know the pain: anti-bot systems, CAPTCHAs, rotating proxies, and scripts that silently break every few weeks. Here's the thing — you usually don't need any of that. Companies don't post jobs on LinkedIn first. They post them in their ATS (Applicant Tracking System) — Greenhouse, Lever, Ashby, Workday, etc. — and most ATS platforms expose the company's board as a public JSON endpoint . No key, no login, no browser. It's the company's own source of truth, so it's cleaner and fresher than any aggregator. The endpoints A few that work with a plain GET ( {company} = the company's slug): Greenhouse — https://boards-api.greenhouse.io/v1/boards/{company}/jobs?content=true Lever — https://api.lever.co/v0/postings/{company}?mode=json Recruitee — https://{company}.recruitee.com/api/offers/ Breezy HR — https://{company}.breezy.hr/json SmartRecruiters, Ashby, BambooHR and Personio have their own equivalents. Workday is the one annoying exception — it's a POST and needs the full board URL (tenant + datacenter + site), so you can't guess it from a bare company name. Example: pulling Stripe's open roles (Python) Stripe uses Greenhouse: import requests company = " stripe " url = f " https://boards-api.greenhouse.io/v1/boards/ { company } /jobs?content=true " jobs = requests . get ( url ). json ()[ " jobs " ] for j in jobs [: 5 ]: print ( j [ " title " ], " — " , j [ " location " ][ " name " ]) That's it. No Selenium, no proxy, no CAPTCHA solver. Runs in ~200ms and won't break next Tuesday because Cloudflare changed something. Auto-detecting the ATS If you don't know which ATS a company uses, just try them in order and take the first one that returns jobs. A bare 404 means "not this ATS, try the next." Greenhouse → Lever → Ashby → SmartRecruiters → Recruitee → Breezy covers a huge chunk of tech companies. Gotchas Rate limits are lenient but real — be polite, set a User-Agent . Descriptions : Greenhouse/Leve

"Cheapest residential proxy" is a search query with a hidden trap: the lowest price per GB and the lowest cost per successful request are not the same number. This post breaks down ten budget-to-mid-tier residential proxy providers from a cost-and-reliability angle, plus a script for measuring the metric that actually matters before you commit real traffic. The trap: price per GB vs. cost per success A proxy at $0.50/GB that fails half your requests is more expensive than one at $1.40/GB with a 98% success rate, because you're paying for retries, wasted bandwidth, and engineering time spent debugging "random" failures. Before comparing sticker prices, calculate: real_cost = traffic_price + failed_request_overhead + retries + setup_time + support_delays Concretely, here's a quick way to model it: def cost_per_success ( price_per_gb , success_rate , avg_response_kb = 50 , retry_overhead = 1.3 ): """ price_per_gb: advertised price success_rate: 0.0-1.0, measured against YOUR target site, not the vendor ' s claim retry_overhead: multiplier for bandwidth wasted on failed/retried requests """ effective_price = price_per_gb * retry_overhead gb_per_request = avg_response_kb / ( 1024 * 1024 ) cost_per_request = gb_per_request * effective_price return cost_per_request / success_rate # Example: cheap provider, mediocre success rate print ( cost_per_success ( 0.50 , 0.75 )) # looks cheap, isn't once failures are priced in # Example: pricier provider, high success rate print ( cost_per_success ( 1.40 , 0.98 )) # often cheaper in practice Run this with your own measured success rate (see the test harness further down), not the vendor's advertised uptime number. What to actually compare Before looking at price, check whether the provider covers: IP pool size and quality (pool size alone tells you nothing about freshness or block rate) Country vs. city-level targeting Sticky session support (for anything stateful) Rotation controls (for scraping/data collection) HTTP(S) and SOCKS5

If you've ever built a scraper that worked perfectly in dev and then got blocked or CAPTCHA'd the moment it hit production traffic volume, you already know why proxy choice matters. This post breaks down residential proxies from a practical, implementation-focused angle: what they are, when to use them vs. alternatives, how to wire them into common tools, and how the major providers stack up. TL;DR Residential proxies route requests through real ISP-assigned IPs, so they're harder for anti-bot systems to fingerprint than datacenter IPs. Rotating residential proxies are for scraping/data collection. Sticky sessions (or static ISP proxies) are for anything stateful — logins, checkout flows, long-lived account sessions. Nstproxy is a good default pick if you want residential, static ISP, and mobile proxies under one API/dashboard instead of juggling multiple vendors for different parts of your stack. For large-scale enterprise scraping, Oxylabs and Bright Data have the most mature tooling. For budget/prototype work, IPRoyal, DataImpulse, and Webshare are worth testing. Proxy types, quickly Type Use for Pros Watch out for Residential Scraping, SERP checks, ad verification Looks like real user traffic Usually billed per GB Static ISP Long-lived sessions, account workflows Fast + stable IP Less useful for high-volume rotation Datacenter Speed-sensitive, low-stakes tasks Cheap, fast Easiest to fingerprint/block Mobile Mobile-first platforms/apps Strongest trust signal Most expensive per GB A production-grade scraping/automation stack often uses more than one of these at once — e.g., rotating residential IPs for crawling, and static IPs pinned to specific browser profiles for anything that requires a login. Wiring a residential proxy into your code Most providers give you a host:port endpoint plus username:password auth, and let you control rotation/session stickiness through the username string. A typical setup looks like this: Python ( requests ): import requests proxy_ho

Web Scraping with Python in 2026: Best Libraries and Anti-Bot Strategies Web scraping in 2026 looks very different from 2020. Sites are smarter, anti-bot systems are more aggressive, and the legal landscape has evolved. Here's what actually works now. The 2026 Scraping Landscape Challenge 2020 Solution 2026 Solution Bot detection Rotate User-Agent Fingerprint randomization + residential proxies CAPTCHAs Manual solving Turnstile/hCaptcha solvers JavaScript rendering Selenium Playwright (faster, more reliable) Rate limiting Sleep between requests Adaptive pacing + request signing IP blocking VPN rotation Residential proxy pools Best Libraries in 2026 1. Playwright (Best for JS-heavy sites) from playwright.sync_api import sync_playwright def scrape_with_playwright ( url ): with sync_playwright () as p : browser = p . chromium . launch ( headless = True ) page = browser . new_page () page . goto ( url , wait_until = " networkidle " ) data = page . query_selector_all ( " .job-item " ) results = [] for item in data : title = item . query_selector ( " h2 " ). text_content () results . append ( title ) browser . close () return results 2. httpx + Selectolax (Fast, no JS needed) import httpx from selectolax.parser import HTMLParser def scrape_static ( url ): resp = httpx . get ( url , headers = { " User-Agent " : " Mozilla/5.0 " }) tree = HTMLParser ( resp . text ) for node in tree . css ( " .listing " ): print ( node . text ()) 3. API-First Approach (Always check first!) Many sites have hidden or public APIs that make scraping unnecessary: url = " https://www.freelancer.com/api/projects/0.1/projects/active/?query=python " data = httpx . get ( url ). json () Anti-Bot Strategies That Work 1. Request Fingerprint Randomization import random def get_random_headers (): browsers = [ " Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " , " Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 " , ] return { " User-Agent " : random . choice ( browsers ), " A

Why your Cloudflare Turnstile token works in the browser but 403s from requests You solved the Turnstile widget. You can see the token in the page. You copy it into your script, POST the form from requests, and the server hands you back a 403 — or a JSON body with "success": false. The token clearly worked a second ago in the browser, so what changed? Short answer: a Turnstile token is not a password you can carry around. It's a one-time, short-lived proof bound to a very specific context, and replaying it from a different context is exactly what it's designed to reject. Below is what that context is, how to tell which constraint you're hitting, and the fix for each. The real scenario You're automating a flow on a Cloudflare-protected site. There's a cf-turnstile widget on the form. You get a token one of two ways: you render the page in a real browser (Playwright/Selenium) and read cf-turnstile-response, or you hand the sitekey + page URL to a solving service and get a token back. Either way, you then submit the form with a plain HTTP client requests, httpx, axios) and it fails. The frustrating part: it's intermittent-looking. The reason it feels random is that there are four separate constraints, and you're usually tripping a different one each time. The four things a Turnstile token is bound to 1. It's single-use Once Cloudflare validates a token server-side (the siteverify call your target makes), that token is spent. Submit twice, retry, or test it once by hand, and the second use returns false. You get a fresh one per submission. 2. It has a short TTL Turnstile tokens expire fast — a few minutes. Solve early, do other work, submit later, and the token can be dead on arrival. The widget auto-refreshes in the browser precisely because tokens go stale; a script that grabs the token and sits on it loses that refresh. 3. It's bound to the sitekey and the page URL Multiple widgets. Some pages embed more than one Turnstile (login + newsletter). Solving the wrong site

Playwright and Selenium are great until you hit bot detection. Google OAuth, Cloudflare, and Vercel checkpoints all flag headless browsers. Here's how to control a real Chromium instance via CDP using Python and websockets. Why Not Playwright? Playwright launches a headless browser with automation flags. Even in headed mode with Xvfb, Google detects it. The CDP Approach Launch Chromium with remote debugging: chromium-browser --user-data-dir = /path/to/profile --remote-debugging-port = 9222 --no-first-run Connect via WebSocket in Python: import asyncio , json , websockets , urllib . request async def get_page_ws (): resp = urllib . request . urlopen ( ' http://localhost:9222/json ' ) targets = json . loads ( resp . read ()) for t in targets : if t [ ' type ' ] == ' page ' : return t [ ' webSocketDebuggerUrl ' ] async def cdp_call ( ws , method , params = None ): msg_id = cdp_call . id = getattr ( cdp_call , ' id ' , 0 ) + 1 msg = { ' id ' : msg_id , ' method ' : method } if params : msg [ ' params ' ] = params await ws . send ( json . dumps ( msg )) while True : resp = json . loads ( await ws . recv ()) if resp . get ( ' id ' ) == msg_id : return resp Key Advantages Real browser fingerprint, no automation flags Persistent sessions, cookies survive across runs Google OAuth works, existing sessions carry over No bot detection, it IS a real browser Follow for more tutorials on browser automation and AI agent architecture.

Hey Dev Community, If you are running enterprise-scale web scrapers, pricing monitors, or data ingestion pipelines for LLMs, you’ve probably spent sleepless nights dealing with network latency and sudden 403 blocks. When choosing an infrastructure partner, every provider pitches the same script: "99.9% uptime guarantees, millions of residential IPs, and lightning-fast response times." But in the trenches of real-world data collection, we all know that marketing numbers rarely match production reality. Last quarter, my team ran an exhaustive infrastructure audit to compare proxy providers pricing performance and infrastructure stability. If you want to dive straight into our live dataset, telemetry scripts, and interactive monitoring utilities, you can check out the full workbench at ProxyVero . Here is a technical breakdown of how we built our benchmarking matrix, and the architectural gaps we discovered across mainstream enterprise proxy services. 📊 1. The Core Metrics: Uptime vs. Success Rates The biggest lie in the networking industry is confusing Server Uptime with Request Success Rate . A proxy gateway server can maintain a 99.9% uptime while the underlying residential peer network is failing 20% of your data collection requests due to strict target WAFs or high peer churn. When conducting our proxy providers uptime guarantees performance benchmarks , we evaluated three core parameters: TCP Handshake Latency : The time it takes to establish a connection with the proxy endpoint. TTFB (Time to First Byte) : Critical for parsing dynamic JavaScript targets. HTTP Status Code Reliability : Tracking the exact ratio of 200 OK vs. 403 Forbidden / 429 Too Many Requests . ⚖️ 2. The Big Three: Oxylabs vs Bright Data vs SmartProxy Comparison To provide an objective proxy network performance benchmarks comparison , we deployed standard headless browser worker instances (Playwright/Puppeteer) routed through different enterprise gateways. Below is a high-level summary of our a

Every World Cup there's a moment. Some player nobody outside their domestic league had heard of scores an absolute screamer in a knockout match, and by the time they've finished celebrating, their follower count is climbing like a rocket. I always found that fascinating, but I could never see it happening. By the time the "X gained 3M followers!" tweets show up, the surge is already over. So this tournament I built a little tracker that snapshots player follower counts on a schedule and shows me the growth curve in near real-time. Here's how it works. The problem with doing this "properly" My first instinct was the official APIs. That died fast. Instagram's Graph API won't give you follower counts for accounts you don't own. TikTok's Research API is academics-only and takes weeks of applications. X's API now starts at $100/month and climbs steeply from there. I just wanted public follower counts — numbers anyone can see by opening the app. I didn't want a data partnership and a legal review. I ended up using the SociaVault API , which wraps public profile data from each platform behind one key. One request, one credit, JSON back. The shared client Everything runs through one tiny helper: // Node 18+ has fetch built in const API_KEY = process . env . SOCIAVAULT_API_KEY ; const BASE = " https://api.sociavault.com " ; async function sv ( path , params ) { const url = new URL ( BASE + path ); Object . entries ( params ). forEach (([ k , v ]) => url . searchParams . set ( k , v )); const res = await fetch ( url , { headers : { " X-API-Key " : API_KEY } }); if ( ! res . ok ) throw new Error ( ` ${ res . status } ${ await res . text ()} ` ); return res . json (); } Grabbing follower counts across platforms Each platform nests the count slightly differently, so I use fallback chains to stay defensive: async function instagramFollowers ( username ) { const data = await sv ( " /v1/scrape/instagram/profile " , { username }); const p = data . data ?. user ?? data . data ?? data

Introduction Extracting data from older websites is a technical challenge that goes beyond simple copy-pasting. The example website provided illustrates this perfectly: its outdated design, reliance on mouse-over interactions, and lack of structured export options create a perfect storm of extraction difficulties. This article dissects these challenges and provides a roadmap for extracting both visible content and mouse-over images while preserving data integrity. The Core Problem: Legacy Technology Meets Modern Needs The website's URL parameters ( screen_width=0&screen_height=0 ) immediately signal a legacy system likely built for a bygone era of fixed-width displays. This design choice breaks modern scraping tools that expect responsive layouts. The mouse-over images, critical to the site's content, are dynamically loaded via JavaScript , meaning they don't exist in the initial page source. This requires simulating user interactions to trigger their appearance, a task beyond basic HTML parsing. Why Manual Extraction Fails Attempting to manually save images or copy text from this site is a losing battle. The mouse-over images, for instance, are not directly downloadable – they're embedded in JavaScript events. Even if you could save them individually, maintaining their association with the corresponding visible content would be error-prone and time-consuming. This method also fails to scale for larger websites with hundreds of such elements. The Technical Solution: A Multi-Pronged Approach Effective extraction requires a combination of techniques: Browser Automation: Tools like Selenium or Puppeteer can simulate mouse movements to trigger hover events, capturing both visible and hidden content. This method mirrors human interaction , ensuring all dynamic elements are revealed. Network Request Inspection: Analyzing the website's backend requests using browser developer tools can reveal direct URLs for mouse-over images , bypassing the need for hover simulation. This

When I started building , I needed reliable access to Instagram data. Like many developers, my first instinct was to use a self-hosted solution such as instagrapi. It worked for experimenting, but once I started depending on it for production workflows, I spent more time maintaining the scraper than building features. Eventually I switched to HikerAPI, a hosted REST API for Instagram. This post isn't about saying one approach is universally better—it's about why it ended up being the better fit for my project. My use case I needed to fetch Instagram profile data for . The requirements were fairly simple: Look up public profiles Process structured JSON Integrate the results into my backend Avoid spending time maintaining login sessions I wasn't interested in reverse engineering Instagram every time something changed. Getting started One thing I liked was that it behaves like a normal REST API. Authentication is done through an x-access-key header, so integrating it into an existing Python backend took only a few minutes. import requests headers = {"x-access-key": "YOUR_KEY"} r = requests.get( " https://api.hikerapi.com/v2/user/by/username?username=instagram ", headers=headers, ) print(r.json()) That's enough to start requesting data and integrating it into your own application. If you want to explore the API, you can find it at HikerAPI. Why I moved away from self-hosted scraping I originally tried , including instagrapi. There wasn't a single issue that made me switch—it was the accumulation of small operational problems: Login sessions expiring Accounts getting challenged Temporary bans Instagram changing internal behavior Regular maintenance after updates None of those problems are impossible to solve. The question became whether solving them was the best use of my time. For my project, the answer was no. I'd rather focus on shipping features than maintaining scraping infrastructure. Tradeoffs Using a hosted API isn't free. Pricing starts at $0.001 per request, wi

If you've ever needed a list of local businesses - every dentist in Manchester, every plumber in Leeds - with their contact details , you've probably hit the same wall I did: Google's Places API is rate-limited, costs money once you scale, and annoyingly doesn't return email addresses at all. Copy-pasting from Maps by hand is soul-destroying past the first ten rows. Most "scrapers" give you the name and a phone number, then stop right where the value starts: the email . This guide shows a practical way to pull structured business data straight from Google Maps and auto-enrich each result with emails, extra phones, and social links - no Google API key, exportable to JSON/CSV/Excel, and callable from code. What you actually get per business { "name" : "Ringway Dental - Cheadle" , "address" : "187 Finney Ln, Heald Green, Cheadle SK8 3PX" , "phone" : "0161 437 2029" , "website" : "https://www.ringwaydental.com/" , "rating" : 5 , "reviewsCount" : 598 , "category" : "Dental clinic" , "lat" : 53.37 , "lng" : -2.22 , "emails" : [ "reception@ringwaydental.com" ], // ← enriched from the website "socialLinks" : { "facebook" : "..." , "instagram" : "..." }, "extraPhones" : [ "..." ] } The first block (name → coordinates) comes from Maps. The emails / socialLinks / extraPhones are the bit that makes a list actually usable for outreach - they're crawled from each business's own website. The fast way: a ready-made Actor Rather than build and babysit the scraping yourself, I packaged this as an Apify Actor: Google Maps Scraper . You give it search terms + locations; it returns enriched rows. Input: { "searchTerms" : [ "dentists" ], "locations" : [ "Manchester, UK" ], "maxPlacesPerSearch" : 50 , "scrapeContacts" : true , "relatedEmailsOnly" : true } That's it. scrapeContacts: true turns on the website crawl for emails/socials; relatedEmailsOnly keeps only emails that belong to the business's own domain (so you don't get random gmail noise). Call it from code (Python) Every Apify Act

If you scrape Weibo's hot-search board you get a snapshot: ~50 trending topics, ranked, right now. That's table stakes — and on its own it's almost useless as a signal. The value isn't what is trending; it's what's moving : which topic just jumped 30 places in 20 minutes, which is decaying, which is brand-new this hour. That's velocity , and velocity is where the signal lives — for brand-crisis teams, consumer-trend desks, and anyone modelling attention in China. The catch: a single scrape can't tell you velocity. You have to diff the board against its own past, reliably, run after run. That's a stateful pipeline, and it has a few non-obvious gotchas. Here's the shape of the problem and how to handle it. Why a snapshot isn't enough Rank-right-now tells you nothing about trajectory. "#7" could be a topic on its way to #1 or one fading out of the top 50 — same row, opposite meaning. To act on a trend you need the derivative : direction, speed, and how long it's been climbing. None of that is in a single pull. The trending-delta problem Three things make "just diff the board" harder than it looks: Key by identity, not position. You can't track a topic by its rank — rank is the thing that changes. Key by the topic itself (its text/keyword) or your deltas are nonsense. State has to survive between runs. A scheduled scrape is stateless by default — each run starts cold. To compute "this rose 12 places since 30 minutes ago," you must persist the previous board and reload it next run, keyed so independent schedules don't overwrite each other. The board churns. Topics appear, peak, and fall off. You want each tagged new / rising / falling / steady / dropped , plus how long it's been on the board and its running peak — none of which exist in the raw snapshot. How to handle it (the pattern) current = pull_board () # [{topic, rank, heat}, ...] previous = load_state ( key ) # durable store that persists across runs for t in current : prev = previous . get ( t . topic ) # match o

The Anti-Bot Detection Checklist I Use Before Every Scraping Project Every scraping project I take on starts with this checklist. Not because I'm paranoid — but because I've learned the hard way that production scrapers fail silently. They return 200 OK with garbage data, or they get rate-limited so gradually you don't notice for days. This is the systematic approach I've refined over 50+ scraping projects. Pre-Scraping: Know Your Target 1. Identify the CDN and Protection Stack Before writing a single line of code, check what you're up against: # Check CDN and headers curl -I https://target-site.com # Look for these common protection headers: # X-Engine: akamai-html-protection # X-Served-By: DataDome # cf-ray: Cloudflare # X-Bot-Status: blocked Common protection platforms: Cloudflare → Look for cf-ray and __cfduid cookies DataDome → Look for datadome in headers or scripts PerimeterX → Look for _pxff cookies Akamai → Look for akamai-html-protection headers 2. Check Robots.txt Respectfully curl https://target-site.com/robots.txt | grep -v "^#" Don't take this as gospel — but it's a good signal. If they explicitly disallow your use case, that's a flag. 3. Map the Site's JavaScript Rendering Some sites are fully static (fast, easy). Others render everything with JavaScript (need Playwright/Puppeteer). Check: // Quick check - fetch raw HTML vs rendered content // If they differ significantly, you need JS rendering const https = require ( ' https ' ); const html = await fetch ( ' https://target.com ' ). then ( r => r . text ()); const hasAngularVueReact = /ng-app|vue|react|__NEXT_DATA__/i . test ( html ); console . log ( ' Needs JS rendering: ' , hasAngularVueReact ); Code-Time: Defensive Patterns 4. Rotate User Agents const USER_AGENTS = [ ' Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/120 Safari ' , ' Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/120 Edge/120 ' , ' Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chro

A scraper can pass every check you wrote and still be wrong about the one thing you actually care about: how much it collected. No exception. No 500. No broken row. Exit code 0, logs green, every field valid. And the set on disk is a quarter of what the site actually has. I have run scrapers in production enough times to stop trusting a green run on its own, and this is the failure that taught me to count. TL;DR A paginated source can serve fewer rows than it claims and never throw — page caps, hidden offset limits, infinite scroll that "ends" early. Your status check (200), schema check (valid row), and byte check (you got data) all pass. None of them counts records. The tell: declared total vs unique ids collected. Or, when there's no declared total, the page that quietly repeats an earlier page. Below is a 40-line probe you can run right now. On a source that caps at 1,500 of a declared 4,000, it returned VERDICT: INCOMPLETE (missing 2500 rows) . This is a completeness check, not a correctness check. Different layer, different bug. What actually goes wrong You write the loop everyone writes. Walk ?page=1 , ?page=2 , keep going until a page comes back empty. Stop. Save. Done. The source has other plans. It says it has 4,000 records — the count is right there in the envelope, or in a "Showing 4,000 results" line in the HTML. But it only ever hands out real data for the first 30 pages. Page 31 doesn't error. It doesn't return empty either. It returns page 1 again. Still HTTP 200. Still 50 valid rows. Your loop has no reason to stop, so it grinds on until its own page budget runs out, collects a pile of rows, and exits clean. You now have 5,000 rows in hand and feel great about it. Looks like plenty. The catch: only 1,500 are unique. The page cap fed you the same first page over and over, and those duplicates hid the shortfall behind a big-looking row count. That is the exact shape of "50 rows passed every check while 4,000 existed" — the scraper saw a lot of rows an

Most web scraping projects are not unique snowflakes. Track competitor prices. Enrich a list of leads. Audit a site for SEO. Pull training data for a model. It is the same handful of recipes, over and over. A web scraping template is one of those recipes, pre-wired: a ready-to-use JSON config that chains the right tools in the right order, so you copy it, point it at your targets, and run. CrawlForge ships 24 of them in the templates gallery . This guide is about using them well — not just copy-paste, but read, adapt, and cost them out before you scale. TL;DR: A CrawlForge template is a copy-paste JSON config that chains multiple MCP tools into one workflow (price monitoring, lead enrichment, SEO audits, market research, AI training data). There are 24 across 9 categories, each costing 3–19 credits per run. Run them from Claude/Cursor, the crawlforge CLI, or the REST API. Free tier = 1,000 credits, no credit card. Table of Contents What Is a Web Scraping Template? Templates Gallery vs the scrape_template Tool How to Use a Template the Right Way 8 Templates Worth Copying First The Other 16 Templates Customizing or Building Your Own FAQ What Is a Web Scraping Template? A template is a saved configuration that orchestrates two or three CrawlForge tools into one workflow with a business outcome attached. Instead of wiring search_web then scrape_structured then analyze_content yourself — and guessing every parameter — you copy a config that already does it. Each template in the gallery carries: A category — E-commerce, Research, Data Collection, Monitoring, AI & LLM, Sales, SEO, Content, or Advanced Scraping (nine in total). A difficulty — beginner, intermediate, or advanced. The tool chain it runs and a fixed credit cost per run (3–19 credits). A copy-paste JSON config with sensible default parameters. You run that config from any MCP client (Claude, Cursor, Windsurf), the crawlforge CLI, or the REST API. Same config, same shape of result. Templates Gallery vs the scrap

The dangerous part of web extraction is not the error. The dangerous part is a clean JSON response that looks correct and is not. If an AI agent uses that output, the mistake does not stay inside a scraper. It moves into a report, a lead list, a price alert, a CRM update, or an automated decision. So before you trust any web extraction API in an agent workflow, test whether it fails honestly. Here is the checklist I use. 1. Test a real page, not a demo page Demo pages are usually polite little museum exhibits. Use pages that behave like the actual internet: a JavaScript-heavy product page a search results page a page with missing fields a login-gated page a blocked or rate-limited page a thin page that has layout but little useful content If the tool only looks good on clean static pages, you have not learned much. 2. Check whether the API separates fetch success from extraction success A page can return HTTP 200 and still be useless. Common examples: the final page is a login screen the visible content is a bot challenge the useful data loads after hydration and never appeared in the fetch the page contains a generic country selector instead of the product the source has navigation text but no actual item data The extractor should not treat these as success just because the request completed. A better response says something like: { "status" : "failed" , "failure_type" : "login_required" , "extracted" : null , "next_step" : "Use a public URL, authorised access, or sample content." } That is boring. Boring is good here. Boring means the agent can stop safely. 3. Ask for fields that are not on the page This is the easiest lie detector. Ask the extractor for something impossible: Extract the product name, price, stock status, warranty length, CEO name, office address, and refund policy. If the page only contains product name and price, the API should say the other fields are missing or low confidence. It should not politely invent them because the schema asked nicely.

Cloudflare Turnstile in Playwright: Why Your Tests Stall and How to Solve It in 8 Lines If you're running Playwright or Selenium against any site behind Cloudflare, you've already met Turnstile. It's the new "managed challenge" widget Cloudflare started shipping in 2023, and it now appears in front of login flows, contact forms, signup pages, and increasingly the entire site root. Here's the part most teams miss: Turnstile doesn't always show a checkbox. A lot of the time it just sits invisible, runs its scoring loop, and either issues a token silently or stalls forever. Your test doesn't crash. It just times out at the next page.click("button[type=submit]") . The CI log says "element not interactable." Nobody knows why. I work on CaptchaAI. I'm going to show you exactly what's happening, then drop in 8 lines that fix it. The real scenario You have a Playwright suite that runs every PR. One day a test starts failing on the signup flow. You re-run it. It fails again. Locally on your laptop it passes. On CI it doesn't. What's actually happening: Cloudflare flagged your CI runner's IP block (GitHub Actions, GitLab runners, Hetzner, OVH, DO — all of them are on Cloudflare's "elevated risk" list). Turnstile decides to switch from invisible mode to "managed challenge" mode. Now there's a widget in the DOM that needs a real token before the form submit will accept. Your test never interacted with the widget because last week it didn't exist. Why retries don't help The instinct is to add a retry: 2 and move on. Don't. Cloudflare's scoring is per-IP-per-fingerprint, and each retry from the same runner makes the next challenge harder, not easier. After ~3 attempts you'll get full block pages instead of the widget. The right move is to solve the widget once, inject the token, and submit normally — exactly what a human user does, just faster. How Turnstile actually issues a token The widget renders an iframe pointing at challenges.cloudflare.com . Inside the iframe it runs a fi