标签：#webscraping

Most football data scrapers on the market only extract high-level final scores (e.g. 2-1 ). But quantitative sports analysts, data scientists, and predictive betting modelers need granular data: Expected Goals (xG) , Official Referee Assignments , Goal Scorers paired with Assist Providers , and Half-Time vs Full-Time (1H/2H) statistical breakdowns . When I set out to build a professional-grade Flashscore scraper on Apify, I ran into two major engineering challenges: The Memory Problem : Keeping Puppeteer running to scrape hundreds of historical matches consumes over 1.5GB of RAM per run. The Protocol Problem : Flashscore serves its deep statistical feeds using a proprietary pipe-delimited data format ( ~ , ¬ , ÷ ) over CDN endpoints, rather than standard REST APIs. In this tutorial, I'll explain how I engineered the Flashscore Elite Statistics Extractor , how the hybrid Browser + HTTP/2 streaming pipeline drops RAM footprint from 1.5GB to 70MB , how to parse Flashscore's custom feed protocol, and how to pipe the resulting datasets directly into Python and Pandas. 🏛️ The Hybrid Pipeline Architecture To achieve zero proxy reliance for standard runs and ultra-low compute costs, the Actor splits execution into a 2-Phase Hybrid Pipeline : [ League & Season Selection ] │ ▼ ┌───────────────────────────────────────────┐ │ Phase 1: Browser Handshake (Puppeteer) │ │ - Captures x-fsign security tokens │ │ - Extracts countryId & tourId │ └─────────────────────┬─────────────────────┘ │ [ Immediate Browser Shutdown ] (RAM drops from 1.2GB -> 70MB) │ ▼ ┌───────────────────────────────────────────┐ │ Phase 2: Parallel HTTP/2 Feed Workers │ │ - got-scraping with JA3 TLS matching │ │ - Decodes df_st_1_ (Stats) & df_sui_1_ │ └─────────────────────┬─────────────────────┘ │ ▼ ┌───────────────────────────────────────────┐ │ Self-Healing Recovery Pass │ │ - Auto-retries skipped/failed matches │ └─────────────────────┬─────────────────────┘ │ ▼ ┌───────────────────────────────────────────┐

I built an OLX scraper for 24 countries — the boring version that actually ships OLX runs classifieds in about two dozen countries. Same brand, different domains, different anti-bot setups. Everyone scraping it does one country at a time. I got tired of forking. So I put 24 countries behind one input. country: "id" or country: "pl" or country: "br" — same schema out. It's live on Apify as primesieve/olx-global-scraper . One file. No browser. Here is the boring part that matters. What it does Input: { "country" : "id" , "keywords" : [ "iphone 13" ], "maxResults" : 50 , "maxPages" : 3 , "proxyConfiguration" : { "useApifyProxy" : true , "apifyProxyGroups" : [ "RESIDENTIAL" ] } } country — two-letter code ( id , pl , in , br , ua , pt , ro , bg , kz , uz , pk , za , ng , ke , eg , lb , ph , co , ar , pe , ec , gt , az , ma ). Default id . keywords — one or more search terms. Each runs sequentially. maxResults / maxPages — caps. Defaults 50 / 3, max 1000 / 30. proxyConfiguration — optional for Indonesia, required for the other 23. Output — same shape every country: { "listingId" : "123456789" , "title" : "iPhone 13 128GB mulus" , "price" : 6500000 , "priceText" : "Rp 6.500.000" , "currency" : "IDR" , "city" : "Jakarta Selatan" , "location" : "Tebet, Jakarta Selatan, DKI Jakarta" , "images" : [ "https://...jpg" ], "thumbnailUrl" : "https://...jpg" , "listingUrl" : "https://www.olx.co.id/item/123456789" , "country" : "id" } Title, price (numeric plus display text), currency, location, images, URL. No seller PII beyond what the listing page shows. No tricks. Try: https://apify.com/primesieve/olx-global-scraper The boring stack // no playwright, no puppeteer // apify + fetch + cheerio. That's it. The scraper is one file. Apify SDK for input, dataset, and pay-per-event. Native fetch for HTTP. cheerio for the HTML path. Undici ProxyAgent when a proxy is configured. Node 20, 512 MB, 600s timeout. I check the endpoint before I write the scraper. Indonesia answered with clean JSO

Google Trends has no public API. What it has is the same internal JSON endpoints the trends.google.com single-page app calls — and those endpoints do something most REST clients aren't built to survive: they answer with HTTP 200 and an empty body when Google decides you look like a bot. Quick answer A 200 OK from Google Trends' widgetdata endpoints does not mean you got data. If the response body is empty, Google soft-blocked the request without bothering to send a 429. The fix is to stop trusting the status code alone: check resp.text.strip() on every call that's supposed to return a payload, and if it's empty, rotate the proxy session and retry exactly like you would on a 429 — because that's what it functionally is. if status == 200 : if require_body and not resp . text . strip (): # Soft block: Google returns 200 with empty body when it detects bots. # Treat the same as 429 — rotate session and retry. logger . warning ( " %s: HTTP 200 but empty body (soft block, attempt %d/%d) " , ...) if proxy_cfg is not None : new_sid = _fresh_session_id () current_proxy_url = await proxy_cfg . new_url ( session_id = new_sid ) await asyncio . sleep ( delay ) continue return resp Why does a working response start with )]}' ? Every Trends JSON endpoint prepends an XSSI-protection prefix before the actual JSON body — a defence against cross-site script inclusion attacks that predates fetch() . Naive json.loads(resp.text) throws a JSONDecodeError on a perfectly healthy response. Worse, the prefix isn't even consistent: /trends/api/explore sends )]}'\n (no comma), some widgetdata endpoints send )]}',\n (with a comma). We check the longer variant first so a response using the short prefix doesn't get mis-stripped: XSSI_PREFIX = " )]} ' , \n " XSSI_PREFIX_NO_COMMA = " )]} ' \n " def _strip_xssi_prefix ( body : str ) -> str : if body . startswith ( XSSI_PREFIX ): return body [ len ( XSSI_PREFIX ):] if body . startswith ( XSSI_PREFIX_NO_COMMA ): return body [ len ( XSSI_PREFIX_NO_COMMA

Job postings are one of the most underrated public data sources on the internet. Recruiters use them to spot placement opportunities, B2B teams read them as buying signals (a new Head of Data means data-tooling budget), and job seekers want to apply on day one — not when a posting finally reaches the aggregators. The usual instinct is to scrape career pages. Don't. Most tech companies host their careers page on one of a handful of Applicant Tracking Systems (ATS), and the four biggest ones — Greenhouse, Lever, Ashby and SmartRecruiters — all expose public, documented JSON APIs . No auth. No proxies. No brittle HTML selectors. The career page itself loads the same JSON you're about to fetch. In this tutorial we'll build a single-file Python tool that: fetches every open job for a company from any of the four ATS, auto-detects which ATS a company uses, normalizes everything into one clean schema, monitors changes — run it on a schedule and get only new / removed / changed postings. The four endpoints ATS Endpoint Greenhouse GET https://boards-api.greenhouse.io/v1/boards/{slug}/jobs?content=true Lever GET https://api.lever.co/v0/postings/{slug}?mode=json Ashby GET https://api.ashbyhq.com/posting-api/job-board/{slug} SmartRecruiters GET https://api.smartrecruiters.com/v1/companies/{slug}/postings (paginated) The {slug} is the company identifier you see in career-page URLs: boards.greenhouse.io/stripe → stripe , jobs.lever.co/spotify → spotify , jobs.ashbyhq.com/linear → linear , careers.smartrecruiters.com/Visa → Visa . Try one right now — no API key needed: curl -s "https://api.ashbyhq.com/posting-api/job-board/linear" | head -c 400 Step 1 — fetchers, one per ATS Each API returns a different shape, so we normalize as we fetch. Here are all four (Python 3, only requests ): import requests UA = { " User-Agent " : " ats-jobs-tutorial/1.0 " } def get_json ( url , params = None ): r = requests . get ( url , params = params , headers = UA , timeout = 30 ) r . raise_for_statu

Most write-ups on this subject are about technique. This one is about the three decisions you should make before you write any code, because in my experience every project that went badly went badly for a reason that was decided on day one and not noticed until much later. I have built this kind of collection twice, for competitive analysis and for a partner-vetting workflow. Neither of them needed to touch anything behind a login, and I want to explain why that turned out to be the useful constraint rather than the limiting one. Line one: the login wall is a boundary A login wall is a statement about who the content is for. Treating it as an engineering obstacle to be routed around is the decision that puts a project on the wrong side of everything: terms of service, the platform's own detection, and in several jurisdictions the law. So the first line is simply: if it requires an account to see, it is out of scope. Not "hard," not "for later." Out of scope. I am not going to discuss techniques for getting past one, and I would be sceptical of any article that does. The interesting engineering question here is not how to see more. It is how much you can actually do with what is openly published, and the honest answer is: considerably more than people assume before they check. This constraint also has a practical benefit that is easy to miss. A pipeline built only on openly available data does not break when authentication changes, does not require credential management, and does not put an account at risk. Mine has survived two platform changes that took down colleagues' authenticated collectors. Line two: public does not mean unrestricted The second line is the one developers get wrong most often, and it has nothing to do with access. Data being publicly visible says nothing about whether you may store it, for how long, or what you may do with it. In the EU and UK, information about an identifiable person is personal data whether or not they published it themselves

I build a web service that preserves evidence of harassment on social platforms. The core feature is a single thing: automatically capture a real screenshot of the offending post. There was no substitute for it. I built an alternative that pulled the text through an API and rendered a tidy "evidence card" image, and threw it away. An image you can author freely afterwards proves nothing. Here's the conclusion first. Third-party wrappers eventually die, and when they do, the failure comes back as a plausible-looking image rather than an error. The first approach was refused by the other side I started with Cloudflare Browser Rendering. The wiring worked. The capture didn't. X blocks headless browsers. The request times out YouTube refuses script injection under a Trusted Types CSP. There's no way to make it render the comment Neither is a bug in my implementation — that is how they are built. So I declared Cloudflare alone impossible for this and moved to a service with a real browser and bot avoidance behind it. Both captures started working. For X, open the post page and clip the tweet element. For YouTube, open the URL with &lc= and screenshot just that comment element. Element screenshots have one trap worth knowing: selector_algorithm=clip returns a blank image when the element sits below the fold. The selector matches, the capture "succeeds," and the file is empty. That took a while to see. ytd-comment-thread-renderer :has ( a [ href *= "lc=ID" ]) A parameter that had worked started returning 400 I wanted timestamps rendered in Japan time, so I passed time_zone: Asia/Tokyo . One day every request started coming back 400. Every capture failed. The provider had narrowed which timezones they accept. Nothing changed on my side. I could diagnose it immediately only because I was storing the raw error body in the database. The response went into rawPayload.screenshotError , so opening one row told me why. Without that, this starts as "captures stopped working, no ide

If you clip short-form video for money, you know Whop Content Rewards: hundreds of live campaigns paying $0.15–$20 per 1,000 views. The discover page lets you sort by budget. That sort is quietly costing you nights of work. Here's the number that changed how I pick campaigns: on the live board right now, 21% of active campaigns have never paid out a single cent. Big banner budget, $0 actually spent. A "$30,000 budget" campaign that has paid nobody in three weeks is not a $30,000 opportunity — it's a landing page. The problem: the board doesn't show you payout speed. You can see budget and budget left , but not how fast the money is actually moving — and that's the only number that separates a campaign that pays from a campaign that poses. The trick: the page already contains everything you need Every campaign card on Whop publishes three things: when it was funded, how much has been spent, and how many creators joined. From one snapshot — no monitoring, no state between runs — you can derive: dailyBurnUsd = budgetSpent / daysSinceFunded → is money moving? estimatedDaysLeft = budgetLeft / dailyBurnUsd → will it still be there? payoutPerCreatorUsd = budgetSpent / creators → what did the average clipper earn? budgetPace = "draining" | "healthy" | "slow" | "stalled" That last field is the shortcut. On today's board of 456 campaigns: pace meaning what to do draining <3 days of budget left skip — gone before your clip gains traction healthy 3–60 days this is where you clip slow 60–180 days fine, but budget may outlive the campaign stalled >180 days at current burn the "big budget" mirage — money posted, almost nobody paid null zero paid out so far unproven; could be brand new, could be dead Real example from today: two campaigns, both showing ~$30K budget. One burns $255/day and has paid the average creator $75 . The other burns $19/day — at that rate its budget lasts four years , which is a polite way of saying nobody is getting paid. On the default board they look ident

A few months back I was running Sales Navigator searches for a client project — filtering down to "VP Sales, fintech, based in Italy or Spain" type lists — and the results were genuinely good. 60, 80 leads that actually matched. Then I hit the part nobody warns you about: there's no button on that page that says "save this." So I did what everyone does. Opened a spreadsheet, alt-tabbed back and forth, typed names and job titles by hand. Around profile 40 I gave up and went looking for a better way. This is what I found, roughly in the order I found it, including the tool I ended up building because none of the existing options quite fit what I needed. First: the export LinkedIn actually gives you LinkedIn has a real, built-in data export, and most people don't realize how narrow it is. It's under your profile photo → Settings & Privacy → Data Privacy → Get a copy of your data . From there you either tick specific categories (that email usually lands within minutes) or request the full archive, which takes closer to a day and sometimes arrives in two batches. Either way you get a download link that expires after 72 hours — and it's desktop only, the mobile app won't let you request one. What you get back is genuinely thorough: connections, messages, your own profile history, activity, even the ad-targeting data LinkedIn holds on you. A couple of quirks worth knowing before you rely on it: some connections' email addresses will just be missing, because sharing an email on download is something each person opts into individually, and you won't get a list of who viewed your profile or any "People You May Know" data. If you're in the EU, EEA, or Switzerland, LinkedIn also runs a separate API for pulling your data on a schedule rather than as a one-off request. Here's what this export is not built for, though: it has no idea what you searched for yesterday. It's an archive of your own account, not a way to capture a live search. Run a Sales Navigator query and pull 80 lea

Cómo solucionar el error “Enable JavaScript and cookies to continue” Este error aparece cuando Cloudflare (u otro proxy inverso de seguridad) detecta que el navegador del usuario no cumple con los requisitos mínimos para acceder al sitio: JavaScript está deshabilitado o las cookies no están permitidas . Pero en entornos reales, el problema suele ser más sutil: el navegador sí tiene JS y cookies habilitados, pero la configuración del entorno de ejecución (como un headless browser, test automation, o un scraper) no emula correctamente el comportamiento del cliente . 🔍 Causa raíz técnica Cloudflare emite un desafío (CAPTCHA o JS challenge) para verificar que el cliente es un navegador real. Si la respuesta no cumple con el desafío (por ejemplo, porque: El navegador no ejecuta el JS del desafío (headless sin soporte), Las cookies no se persisten entre solicitudes, El User-Agent o Accept-Language no coinciden con navegadores reales, Falta el Referer o Origin en headers, Se bloquean cookies de terceros (como las de Cloudflare), … entonces el servidor devuelve este mensaje estático en lugar de redirigir a la página solicitada. ⚠️ Nota crítica : Si estás usando herramientas como curl , requests de Python, o navegadores headless sin configuración especial, no pasarás el desafío de Cloudflare . Es intencional: Cloudflare bloquea tráfico no humano por diseño. ✅ Solución definitiva (por escenario) 🛠️ Caso 1: Navegador real (usuario final) Verifica que JavaScript esté habilitado : Chrome: Configuración → Privacidad y seguridad → Configuración de sitios → JavaScript → Permitido . Firefox: Preferencias → Privacidad y seguridad → Cookies y datos de sitios → Deshabilitar “Bloquear cookies y datos de sitios” . Limpia cookies y caché (especialmente para *.cloudflare.com ). Reinicia el navegador y vuelve a cargar la página. 🛠️ Caso 2: Automatización / Scraping (Python + Playwright/Selenium) No uses requests o urllib : no ejecutan JS. Usa un navegador real con soporte para Cloudflare. ✅

This article was originally published on e-shikumi-labo . Hello, I'm Shin from e-Shikumi-Labo. This is Part 3 of "Systematized Thinking," where we use AI to build our own tools and independently maintain them. Last time, I talked about creating a system to automatically output Markdown (.md) files to Google Drive simultaneously with appending to a spreadsheet. With list management in a spreadsheet and a comfortable viewing environment in Obsidian established, it was getting very close to completion as a tool. However, as I continued to use it practically, new challenges emerged on the operational front. This time, I will share the risks I faced while transitioning from a "manual button" to "full automation," and the process of evolving into safe code. 1. I Want to Eliminate the "Hassle of Pressing a Button" During the prototype stage, the system was designed so that logs were saved by pressing a button placed on the screen. However, as long as a human operates it manually, there are inevitably limitations. If you are concentrating on the conversation, you might forget to press the save button and close the screen. If the conversation gets long, you might miss past utterances that are no longer displayed on the screen. "If I have the screen open and am conversing, I want it to automatically save in the background without bothering human hands." Thinking this, I asked the AI to write the code for full automation. 2. The Code the AI Produced: "Patrolling the Screen Every 1.5 Seconds" When I consulted the AI, it immediately presented code for full automation. The mechanism was, "Start a timer every 1.5 seconds, check the entire screen in the background, and send any new utterances." When I actually tried it, the logs accumulated automatically as soon as I conversed without pressing the button, and at first glance, it looked like exceptionally well-done full automation. However, I felt something was slightly off regarding this "monitoring on a 1.5-second cycle." 3. The B

Key takeaways A request is judged on many layers at once — IP reputation, TLS fingerprint, HTTP/2 shape, headers, and how the browser is driven — and failing any one is enough for a 403. Your laptop passes because every layer is consistent with a real home browser; a server changes one (usually the IP) and the inconsistency is the tell. A 403 with no challenge page almost always means you were blocked at the network layer (IP/ASN reputation or TLS/JA3 fingerprint) before any HTML was served — not a credentials or rate-limit bug, so 'add a User-Agent' or 'slow down' won't fix it. The fix order that actually works: get off datacenter IPs (residential/ISP proxies), match a real browser's TLS fingerprint, and spin up a real-browser stealth setup only for the pages that truly need JavaScript — escalate, don't lead with a browser. A proxy only changes your IP; a Linux VPS still leaks a Linux-shaped TLS/JS fingerprint, so 'residential IP + datacenter everything-else' is a contradiction a real home machine never makes — which is why a proxied server can get blocked harder than your laptop. Your scraper runs perfectly on your laptop. You deploy it to a VPS or a CI runner, change nothing in the code, and suddenly every request comes back 403 . It feels like a bug — the code is identical — but it usually isn't. Anti-bot systems judge a request on many signals at once, and moving from your home machine to a datacenter flips several of them at the same time. This post breaks down exactly which signals change, how to tell which one is blocking you, and how to fix it — for authorized access to public data (we'll keep that framing honest throughout; nothing here is about defeating a protection). A request is judged in two stages It helps to know that detection happens in two stages: Stage 1 — before any HTML is served. IP reputation, your TLS handshake, your HTTP/2 settings, and header order are all inspected on the connection itself, passively and cheaply, before your request is e

A scraping job usually fails in boring ways: the browser hangs, a selector starts returning empty strings, a login expires, or the target site returns a captcha halfway through the run. The awkward part is that many platforms still bill you for the work done before the failure. If you run enough jobs, that difference shows up both in your invoice and in the amount of defensive code you need around the scraper. Billing by compute time changes how you build A lot of scraping platforms charge for runtime. Apify, for example, uses compute units: memory multiplied by time. A browser-heavy actor running for ten minutes with 2 GB of RAM consumes roughly a third of a compute unit before any actor-specific result fees. That model is reasonable from the provider side. Chromium processes are expensive. Proxies cost money. Retries use resources. But as the caller, you care about a different unit: did I get the rows I needed? The hard part is that runtime billing makes cost hard to know before execution. A job that normally takes 30 seconds might take 8 minutes when a site slows down. A job that returns malformed data can still count as successful from the platform's point of view. A job that fails after rendering 200 pages still consumed browser time. If your pipeline runs once a day, that may be fine. If it runs continuously, you probably want a local cost model that tracks outcomes, not just requests. type ScrapeRun = { jobId : string ; target : string ; startedAt : string ; finishedAt ?: string ; status : " queued " | " running " | " succeeded " | " failed " ; rowsExpected ?: number ; rowsReceived ?: number ; billedUnits ?: number ; }; function isUsefulResult ( run : ScrapeRun ) { if ( run . status !== " succeeded " ) return false ; if ( run . rowsExpected && ( run . rowsReceived ?? 0 ) < run . rowsExpected * 0.9 ) { return false ; } return ( run . rowsReceived ?? 0 ) > 0 ; } function costPerUsefulRow ( run : ScrapeRun ) { if ( ! isUsefulResult ( run )) return Infinity ; ret

When you scrape at any real volume, the bottleneck is rarely your code — it's the target site's rate limiting and IP bans. Rotating residential proxies solve this by routing each request through a different real-user IP. Here's how to wire them into requests and Scrapy in Python, including the sticky-session trick most tutorials skip. The proxy URL format A residential proxy is just an authenticated HTTP/SOCKS endpoint. With a pool gateway, you target a country and control session behavior through the username , not separate endpoints: http://USERNAME_country-us_session-a1b2c3_lifetime-30:PASSWORD@proxy.gproxy.net:1000 country-us — exit country (ISO code) session-a1b2c3 — a sticky-session id; reuse it to keep the same IP , change it to rotate lifetime-30 — how many minutes that session's IP stays fixed Basic request through a rotating proxy import requests USER = " USERNAME " PWD = " PASSWORD " def proxy ( country = " us " , session = None , lifetime = 30 ): tag = f " _country- { country } " if session : tag += f " _session- { session } _lifetime- { lifetime } " url = f " http:// { USER }{ tag } : { PWD } @proxy.gproxy.net:1000 " return { " http " : url , " https " : url } # New IP on every call (no session id): r = requests . get ( " https://api.ipify.org?format=json " , proxies = proxy (), timeout = 30 ) print ( r . json ()[ " ip " ]) Run it in a loop and you'll see a different IP each time — the gateway rotates automatically when no session id is present. Sticky sessions: keep one IP across requests Some flows (login, multi-step checkouts, paginated results behind a cookie) break if your IP changes mid-session. Pin the IP by passing a stable session id: import uuid sess = uuid . uuid4 (). hex [: 8 ] # one id for the whole flow p = proxy ( country = " de " , session = sess , lifetime = 30 ) s = requests . Session () s . proxies . update ( p ) s . get ( " https://example.com/login " ) s . post ( " https://example.com/login " , data = {...}) # same exit IP When you

For a long time I ran product extraction off a database of custom selector configs. Hundreds of retailers, each with its own set of CSS selectors mapped field by field: price here, title there, image over there. It worked. It was also a treadmill. Every retailer that redesigned its frontend silently broke its config, and the maintenance tax grew with every retailer I added. Hundreds of configs is hundreds of things that can rot without telling you, usually right before someone downstream asks why a brand's prices went null. At some point I stopped feeding it. The scraping platform I ran at my last engagement fed a 20M+ product catalogue at millions of pages a day, and the thing that made that survivable wasn't better per-site config. It was leaning on almost no per-site config at all. The pattern Instead of one careful parser per site, run several cheap generic extractors on every page, in parallel: JSON-LD. A huge share of e-commerce pages ship a schema.org/Product block because Google rewards it. It has name, price, currency, availability, images. It's the closest thing to a public API hiding in the HTML. OpenGraph tags. og:title , og:image , product:price:amount . Lower quality than JSON-LD, but present on sites too lazy for structured data, because everyone wants pretty link previews. Microdata / RDFa. Older sites, still surprisingly common outside the US. Embedded state. __NEXT_DATA__ , window.__INITIAL_STATE__ and friends. A JSON blob the site's own frontend hydrates from. Heuristics. A price-shaped string near a currency symbol inside the main content region. The largest above-the-fold image. The h1 . Dumb, and dumb works more often than you'd think. None of these is reliable alone. OG price tags go stale, JSON-LD sometimes describes the wrong variant, heuristics grab the crossed-out "was" price. The trick is you don't pick an extractor. You pick per field, and you let agreement between sources carry the decision. Confidence merge Each extractor emits candida

Every founder I know has burned a week building an investor list: digging through Crunchbase profiles tab by tab, copying partner names into a spreadsheet that is stale before the seed round closes. The data you want is simple, firms plus focus plus contacts, and it is weirdly hard to get in bulk. The shortcut I use now is the Startup Investors Data Scraper on Apify, a queryable investor database of 10,469 firms that returns filtered JSON in one call. Disclosure: the Apify links in this post are affiliate links. If you run the Actor, I may earn a referral commission at no extra cost to you. Is there a public API for investor data? Not really. The big commercial databases keep their APIs behind sales calls and paid plans sized for funds, not founders. Free sources are scattered lists and shared spreadsheets with no filters and no freshness guarantees. This Actor takes a different shape: a curated database of 10,469 investment firms (as of December 2025) that you query like an API, filtering by firm type, sector, stage, and country, and paying only for the records you pull. What the investor database API returns The investor database API returns one JSON record per firm: name, type, description, location, website, social links, assets under management, stages, and sector focus, with partner contacts when you ask for them. Field Example Notes firm_name Acme Ventures With firm_description alongside firm_type_name Venture Capital Investor One of 17 firm types firm_country Germany Plus firm_city and firm_state firm_website https://acme.vc Also firm_linkedin_url , crunchbase_url , twitter_url firm_aum $250M Assets under management when known investor_contacts [{ "job_title": "Partner", ... }] Names, titles, LinkedIn URLs, emails when available, and check sizes, with Include_Contacts on Who this is for Founders building a raise pipeline, sales teams selling into VC and PE back offices, and analysts mapping which firms fund a sector. If your CRM needs 200 seed funds with war

Heads-up: this post references a tool I built. It's a genuinely useful walkthrough either way — the technique applies to any Airbnb scraping project. If you've ever tried to scrape Airbnb, you already know the two walls you hit: the pages are rendered by JavaScript, and Airbnb aggressively blocks datacenter IPs. Below is the reliable way to get clean Airbnb data in 2026 — listing prices, ratings, coordinates, and discounts — without running a headless browser or babysitting proxies. The key insight: Airbnb ships its data in the HTML You don't need to render the page. Every Airbnb search response embeds the full result set as JSON inside a <script id="data-deferred-state-0"> tag. Parse that and you get structured data straight away — no DOM scraping, no selectors that break on the next redesign. The path to the results is: niobeClientData[*][1].data.presentation.staysSearch.results ├── searchResults[] // ~18 listings per page └── paginationInfo.pageCursors[] // all page cursors, upfront Each listing carries a base64-encoded ID in demandStayListing.id (decode it, take the segment after the last colon, and you have the numeric listing ID for airbnb.com/rooms/<id> ), a price line with discounts, avgRatingLocalized ("4.95 (123)"), and GPS coordinates. The two gotchas Datacenter IPs get blocked. You need residential proxies. If a response comes back without the data-deferred-state marker, you've been served a bot check — rotate to a fresh IP and retry. ~270 result cap per search. Airbnb won't paginate past ~15 pages. To cover a whole market, split into narrower searches (by price band or neighborhood) and dedupe by listing ID. The no-code way If you'd rather not maintain proxy pools and parsers, I published an Airbnb Scraper on Apify that does exactly the above. Paste a location or a full Airbnb search URL (every filter is honored), and get flat JSON/CSV back. curl -X POST "https://api.apify.com/v2/acts/ethanteague~airbnb-scraper/run-sync-get-dataset-items?token=YOUR_TOKE

If you have written more than a couple of scrapers, you already know the pattern. The first few hundred requests fly through. Then responses slow down, you start seeing 429 Too Many Requests , a captcha wall appears, and finally the target just returns empty pages or a hard 403 . Your code did not change. Your IP did. Scraping at any real volume is less about parsing HTML and more about managing where your requests come from. This post is a practical walk-through of how proxies fit into a scraping pipeline: why a single IP fails, what proxy types actually matter, how rotation works, and how to wire it all up in Python with requests , aiohttp , and Scrapy. There is code you can copy, plus the mistakes that cost me the most time. Why one IP is never enough Every site you scrape sees the same thing: a stream of requests from one address, arriving faster and more regularly than a human ever would. Anti-bot systems are built to spot exactly that. The signals they use are boring but effective: Request rate per IP. Too many hits in a short window trips a rate limiter. Volume over time. Even a slow scraper eventually stands out if every request comes from the same address for hours. Behavioral fingerprint. No mouse, no scroll, identical headers, requests in perfect intervals. Reputation. Datacenter ranges that have been abused before are pre-flagged. You can soften some of these with headers, delays, and a real browser, but there is a ceiling. Once a single IP has made enough requests, it gets throttled or blocked regardless of how polite you are. The only way past that ceiling is to spread requests across many addresses, so no single one crosses the threshold. That is the entire job of a proxy pool. The proxy landscape, minus the marketing Providers love to complicate this. For scraping, the distinctions that actually change your results are these: Shared vs private. Shared proxies are handed to many customers at once. You inherit everyone else's behavior, so an address ca

A scraper fails, someone swaps the User-Agent, someone else adds a proxy, then the job starts passing locally but fails again in CI. That usually happens because Cloudflare did not block “scraping” as one thing. It evaluated several signals, and each failure needs a different fix. This is about authorized automation: your own sites, customer-approved workflows, testing, monitoring, data access you are allowed to perform. If you do not have permission to automate against a site, changing fingerprints or rotating IPs does not make it okay. Start with the failure you actually see Cloudflare failures often get collapsed into “403”, but the page body matters. Common cases: Error 1020 : usually an access denied page from a Cloudflare rule or bot score decision. The HTTP status may still be 403, so inspect the HTML. 403 without a 1020 page : often IP reputation, firewall rules, geo restrictions, or an auth problem. 429 : rate limit exhaustion. Slowing down can help here, but it will not fix a fingerprint problem. Endless Just a moment... page : your client did not complete the browser-side challenge. CAPTCHA or Turnstile loop : Cloudflare still considers the session borderline after earlier checks. Add classification before you add workarounds. Even a basic classifier saves time: import time import requests CLOUDFLARE_MARKERS = { " 1020 " : " cloudflare_access_denied " , " Just a moment " : " cloudflare_js_challenge " , " cf-turnstile " : " cloudflare_turnstile " , " cf-error-code " : " cloudflare_error_page " , } def classify_response ( resp : requests . Response ) -> str : body = resp . text [: 5000 ] if resp . status_code == 429 : return " rate_limited " for marker , label in CLOUDFLARE_MARKERS . items (): if marker in body : return label if resp . status_code == 403 : return " forbidden_unknown " return " ok " if resp . ok else f " http_ { resp . status_code } " def get_with_backoff ( url : str , max_attempts = 4 ): for attempt in range ( max_attempts ): resp = request

Your customers have stopped Googling "best note-taking app." They're asking ChatGPT, Perplexity, and Gemini instead — and getting back a short list of three or four products. If your brand isn't on that list, you're invisible, and unlike a Google ranking you can't even see where you stand. That's the problem I set out to measure. This post is the build breakdown: five AI answer engines, one uniform result shape, a mention-detection core that doesn't lie to you, and the honest gotchas I hit around cost and billing. The whole thing runs as a paid Apify Actor written in async Python. The niche has a name now — GEO (Generative Engine Optimization) or AEO (Answer Engine Optimization). Think SEO, but the search engine is a language model and the "ranking" is whether you get named in the answer. The core question Give the tool a brand, its competitors, and the buyer-intent questions your customers actually type: { "brand" : "Notion" , "competitors" : [ "Obsidian" , "Coda" , "Evernote" ], "prompts" : [ "best note taking app for students" , "Notion vs Obsidian which should I use" ], "engines" : [ "perplexity" , "chatgpt" , "gemini" , "claude" , "aiOverview" ], "samplesPerPrompt" : 3 } It asks each engine each prompt (several times, because LLM answers vary run-to-run), then analyzes every answer for: were you mentioned, how early, were you recommended or just listed, what's the sentiment, who else got named, and — the part incumbents skip — which domains each engine cited. That last one is the actionable output: it tells you which websites the AI trusts for your category, i.e. where you need coverage. Architecture: one shape to rule them all The trick that keeps the whole thing sane is that every engine adapter — whether it's a clean REST API or a messy HTML scrape — returns the exact same record shape : { " engine " : " perplexity " , " prompt " : " best note taking app for students " , " sampleIndex " : 1 , " responseText " : " ... " , " citations " : [{ " url " : " ... "

If you've ever tried to pull job listings by scraping LinkedIn or Indeed, you know the pain: anti-bot systems, CAPTCHAs, rotating proxies, and scripts that silently break every few weeks. Here's the thing — you usually don't need any of that. Companies don't post jobs on LinkedIn first. They post them in their ATS (Applicant Tracking System) — Greenhouse, Lever, Ashby, Workday, etc. — and most ATS platforms expose the company's board as a public JSON endpoint . No key, no login, no browser. It's the company's own source of truth, so it's cleaner and fresher than any aggregator. The endpoints A few that work with a plain GET ( {company} = the company's slug): Greenhouse — https://boards-api.greenhouse.io/v1/boards/{company}/jobs?content=true Lever — https://api.lever.co/v0/postings/{company}?mode=json Recruitee — https://{company}.recruitee.com/api/offers/ Breezy HR — https://{company}.breezy.hr/json SmartRecruiters, Ashby, BambooHR and Personio have their own equivalents. Workday is the one annoying exception — it's a POST and needs the full board URL (tenant + datacenter + site), so you can't guess it from a bare company name. Example: pulling Stripe's open roles (Python) Stripe uses Greenhouse: import requests company = " stripe " url = f " https://boards-api.greenhouse.io/v1/boards/ { company } /jobs?content=true " jobs = requests . get ( url ). json ()[ " jobs " ] for j in jobs [: 5 ]: print ( j [ " title " ], " — " , j [ " location " ][ " name " ]) That's it. No Selenium, no proxy, no CAPTCHA solver. Runs in ~200ms and won't break next Tuesday because Cloudflare changed something. Auto-detecting the ATS If you don't know which ATS a company uses, just try them in order and take the first one that returns jobs. A bare 404 means "not this ATS, try the next." Greenhouse → Lever → Ashby → SmartRecruiters → Recruitee → Breezy covers a huge chunk of tech companies. Gotchas Rate limits are lenient but real — be polite, set a User-Agent . Descriptions : Greenhouse/Leve