标签：#rap

When I started building , I needed reliable access to Instagram data. Like many developers, my first instinct was to use a self-hosted solution such as instagrapi. It worked for experimenting, but once I started depending on it for production workflows, I spent more time maintaining the scraper than building features. Eventually I switched to HikerAPI, a hosted REST API for Instagram. This post isn't about saying one approach is universally better—it's about why it ended up being the better fit for my project. My use case I needed to fetch Instagram profile data for . The requirements were fairly simple: Look up public profiles Process structured JSON Integrate the results into my backend Avoid spending time maintaining login sessions I wasn't interested in reverse engineering Instagram every time something changed. Getting started One thing I liked was that it behaves like a normal REST API. Authentication is done through an x-access-key header, so integrating it into an existing Python backend took only a few minutes. import requests headers = {"x-access-key": "YOUR_KEY"} r = requests.get( " https://api.hikerapi.com/v2/user/by/username?username=instagram ", headers=headers, ) print(r.json()) That's enough to start requesting data and integrating it into your own application. If you want to explore the API, you can find it at HikerAPI. Why I moved away from self-hosted scraping I originally tried , including instagrapi. There wasn't a single issue that made me switch—it was the accumulation of small operational problems: Login sessions expiring Accounts getting challenged Temporary bans Instagram changing internal behavior Regular maintenance after updates None of those problems are impossible to solve. The question became whether solving them was the best use of my time. For my project, the answer was no. I'd rather focus on shipping features than maintaining scraping infrastructure. Tradeoffs Using a hosted API isn't free. Pricing starts at $0.001 per request, wi

Insta360's first gimbal camera is a high-powered Osmo Pocket rival that you can actually buy.

PostgreSQL 19 Beta has been announced, with general availability expected in September, following the project's yearly major-release cadence. This release introduces native SQL Property Graph Queries (SQL/PGQ), concurrent table repacking to reclaim storage without downtime, and a broad set of performance, observability, and administration improvements. By Renato Losio

If you've ever needed a list of local businesses - every dentist in Manchester, every plumber in Leeds - with their contact details , you've probably hit the same wall I did: Google's Places API is rate-limited, costs money once you scale, and annoyingly doesn't return email addresses at all. Copy-pasting from Maps by hand is soul-destroying past the first ten rows. Most "scrapers" give you the name and a phone number, then stop right where the value starts: the email . This guide shows a practical way to pull structured business data straight from Google Maps and auto-enrich each result with emails, extra phones, and social links - no Google API key, exportable to JSON/CSV/Excel, and callable from code. What you actually get per business { "name" : "Ringway Dental - Cheadle" , "address" : "187 Finney Ln, Heald Green, Cheadle SK8 3PX" , "phone" : "0161 437 2029" , "website" : "https://www.ringwaydental.com/" , "rating" : 5 , "reviewsCount" : 598 , "category" : "Dental clinic" , "lat" : 53.37 , "lng" : -2.22 , "emails" : [ "reception@ringwaydental.com" ], // ← enriched from the website "socialLinks" : { "facebook" : "..." , "instagram" : "..." }, "extraPhones" : [ "..." ] } The first block (name → coordinates) comes from Maps. The emails / socialLinks / extraPhones are the bit that makes a list actually usable for outreach - they're crawled from each business's own website. The fast way: a ready-made Actor Rather than build and babysit the scraping yourself, I packaged this as an Apify Actor: Google Maps Scraper . You give it search terms + locations; it returns enriched rows. Input: { "searchTerms" : [ "dentists" ], "locations" : [ "Manchester, UK" ], "maxPlacesPerSearch" : 50 , "scrapeContacts" : true , "relatedEmailsOnly" : true } That's it. scrapeContacts: true turns on the website crawl for emails/socials; relatedEmailsOnly keeps only emails that belong to the business's own domain (so you don't get random gmail noise). Call it from code (Python) Every Apify Act

In sandbox runtimes, "isolation" is the core requirement. qwrap (based on bwrap user namespace) and Container (podman/docker) are two mainstream backends. They solve the same problem — running code in a restricted environment — but take completely different paths. This article uses extensive analogies to help you understand the similarities and differences. Building Intuition: Two Ways to "Lock the Door" Imagine you need to confine someone you don't fully trust in a room to do work: qwrap approach : In your existing house, you put up a partition to wall off a corner, leaving only a small window to pass materials through. The walls are still the original walls, the floor is still the original floor, but the person can only see what's inside the partition. Container approach : You build a shipping container with its own independent power, water, and ventilation systems. Put the person inside, close the door. They feel like they're in a complete little house, completely unaware of what's outside. This is the most fundamental difference: qwrap is lightweight view isolation, Container is complete environment encapsulation . What is qwrap (bwrap user namespace) qwrap uses bubblewrap (bwrap) under the hood, a sandboxing tool that leverages Linux user namespaces. How it Works Host filesystem ├── /usr/bin/python3 ← Host's Python ├── /home/user/project/ ← User project └── /tmp/secrets/ ← Sensitive files qwrap sandbox view (what the process sees) ├── /usr/bin/python3 ← bind-mounted in, read-only ├── /workspace/ ← Only the project directory is exposed └── (/tmp/secrets/ doesn't exist) ← Completely invisible Key mechanisms: User Namespace : The process thinks it's root, but actually maps to an unprivileged user on the host Mount Namespace : Only bind-mounts necessary directories in, everything else is invisible No images, no layers, no network namespace (unless explicitly configured) Analogy: VPN Split Tunneling qwrap is like split-tunneling rules on your phone — you're not wrap

[Excerpted from THE QUANTUM COLLAPSE CHRONICLES — not science fiction, but a grounded forecast of what may come when quantum computation dismantles the cryptographic foundations of our digital civilization. These articles explore the collapse of computational trust and the brutal reconstruction of the world that follows.] The history of human civilization is often defined by sudden, violent shifts in the nature of power. We speak of the fall of empires, the industrial revolutions, and the splitting of the atom. But in the mid-2030s, the world experienced a collapse that was not made of steel or stone, but of mathematics. It was a quiet, clinical, and utterly devastating unraveling of the digital fabric that held modern society together. To understand The Quantum Collapse , one must look past the headlines of the era and into the humming, sub-Kelvin depths of the dilution refrigerators that changed everything. This is the story of how the transition from probabilistic experimentation to deterministic computation rendered the world's secrets transparent and its economies obsolete. The Death of Noise: The Rise of Dr. Aris Thorne For the first three decades of the 21st century, quantum computing was a game of chance. Scientists operated in the era of Noisy Intermediate-Scale Quantum (NISQ) devices—machines so temperamental and prone to error that every calculation was a desperate struggle against environmental noise. In those days, a single stray photon or a microscopic fluctuation in temperature could collapse a delicate superposition, turning a groundbreaking calculation into useless digital static. The turning point arrived in 2036 at the Institute for Advanced Quantum Engineering (IAQE) in the High Sierras. The air in the facility didn't vibrate with the erratic drone of the late 2020s; instead, it carried a heavy, rhythmic thrum—the sonic signature of the Lattice-Array-9 (LA-9). At the center of this revolution was Dr. Aris Thorne, the lead architect of the LA-9 pr

When you build Shopify apps or integrations, pagination becomes important very quickly. A small test store may have a few products and orders. A real merchant store can have thousands of products, variants, orders, customers, inventory items, metafields, and fulfillment records. You cannot fetch all of that data in one Shopify GraphQL request. You need pagination. More importantly, you need pagination that performs well. Poor Shopify GraphQL pagination can create slow syncs, API throttling, timeout errors, duplicate processing, and incomplete exports. This post explains how Shopify GraphQL pagination works and how to handle large Shopify datasets in a practical way. What Shopify GraphQL Pagination Solves Pagination lets your app retrieve data in smaller chunks. Instead of asking Shopify for 50,000 products at once, your app asks for 100 or 250 products per request. Shopify returns the data and gives your app information about the next page. This protects your app from huge responses and protects Shopify from heavy requests. It also gives your integration more control over retries, progress tracking, and background processing. Shopify Uses Cursor-Based Pagination Shopify GraphQL uses cursor-based pagination. That means you do not request data using page numbers. You request the next page using a cursor from the previous response. A basic product pagination query looks like this: query GetProducts ( $cursor : String ) { products ( first : 100 , after : $cursor ) { nodes { id title handle updatedAt } pageInfo { hasNextPage endCursor } } } The first time you run this query, pass cursor as null. Shopify returns the first 100 products and gives you an endCursor . Use that endCursor as the after value in the next request. Keep doing this until hasNextPage is false. Why Cursors Work Better Than Page Numbers Offset pagination usually works like this: page=1 page=2 page=3 or: offset=5000&limit=100 This approach becomes inefficient when datasets grow. The system may need to sk

They swear they haven’t peeked at the closely guarded secret and that they’ll keep the cryptographic competition going.

The funding round was led by Norwest, with participation S Capital VC, Cerca Partners, and Oceans Ventures. Snowflake Ventures also participated as a strategic investor.

If you scrape Weibo's hot-search board you get a snapshot: ~50 trending topics, ranked, right now. That's table stakes — and on its own it's almost useless as a signal. The value isn't what is trending; it's what's moving : which topic just jumped 30 places in 20 minutes, which is decaying, which is brand-new this hour. That's velocity , and velocity is where the signal lives — for brand-crisis teams, consumer-trend desks, and anyone modelling attention in China. The catch: a single scrape can't tell you velocity. You have to diff the board against its own past, reliably, run after run. That's a stateful pipeline, and it has a few non-obvious gotchas. Here's the shape of the problem and how to handle it. Why a snapshot isn't enough Rank-right-now tells you nothing about trajectory. "#7" could be a topic on its way to #1 or one fading out of the top 50 — same row, opposite meaning. To act on a trend you need the derivative : direction, speed, and how long it's been climbing. None of that is in a single pull. The trending-delta problem Three things make "just diff the board" harder than it looks: Key by identity, not position. You can't track a topic by its rank — rank is the thing that changes. Key by the topic itself (its text/keyword) or your deltas are nonsense. State has to survive between runs. A scheduled scrape is stateless by default — each run starts cold. To compute "this rose 12 places since 30 minutes ago," you must persist the previous board and reload it next run, keyed so independent schedules don't overwrite each other. The board churns. Topics appear, peak, and fall off. You want each tagged new / rising / falling / steady / dropped , plus how long it's been on the board and its running peak — none of which exist in the raw snapshot. How to handle it (the pattern) current = pull_board () # [{topic, rank, heat}, ...] previous = load_state ( key ) # durable store that persists across runs for t in current : prev = previous . get ( t . topic ) # match o

Earlier this year, the “Because I Got High” rapper went viral for winning a case against the cops. Now he’s crypto’s free-speech hero, even though he isn’t quite sure how the digital currency works.

The Anti-Bot Detection Checklist I Use Before Every Scraping Project Every scraping project I take on starts with this checklist. Not because I'm paranoid — but because I've learned the hard way that production scrapers fail silently. They return 200 OK with garbage data, or they get rate-limited so gradually you don't notice for days. This is the systematic approach I've refined over 50+ scraping projects. Pre-Scraping: Know Your Target 1. Identify the CDN and Protection Stack Before writing a single line of code, check what you're up against: # Check CDN and headers curl -I https://target-site.com # Look for these common protection headers: # X-Engine: akamai-html-protection # X-Served-By: DataDome # cf-ray: Cloudflare # X-Bot-Status: blocked Common protection platforms: Cloudflare → Look for cf-ray and __cfduid cookies DataDome → Look for datadome in headers or scripts PerimeterX → Look for _pxff cookies Akamai → Look for akamai-html-protection headers 2. Check Robots.txt Respectfully curl https://target-site.com/robots.txt | grep -v "^#" Don't take this as gospel — but it's a good signal. If they explicitly disallow your use case, that's a flag. 3. Map the Site's JavaScript Rendering Some sites are fully static (fast, easy). Others render everything with JavaScript (need Playwright/Puppeteer). Check: // Quick check - fetch raw HTML vs rendered content // If they differ significantly, you need JS rendering const https = require ( ' https ' ); const html = await fetch ( ' https://target.com ' ). then ( r => r . text ()); const hasAngularVueReact = /ng-app|vue|react|__NEXT_DATA__/i . test ( html ); console . log ( ' Needs JS rendering: ' , hasAngularVueReact ); Code-Time: Defensive Patterns 4. Rotate User Agents const USER_AGENTS = [ ' Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/120 Safari ' , ' Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/120 Edge/120 ' , ' Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chro

This is the story of how a mundane complaint — "the VPN is slow" — turned into a US patent application. Not a granted patent. An application . I want to be precise about that from the start, because the distance between the two is the whole point of this post. It started with a slow VPN The company I work for had an internal VPN that everyone routed through. It lived in the Tokyo office, it was old, and it was not something I built. Then the complaints started arriving — from a lot of people, all saying the same thing: it's slow. I work from Thailand most of the time. That detail matters. If that aging box in Tokyo had fallen over, I would have been the person furthest from the power button, in the worst position to fix it. A slow VPN is annoying. An unreachable VPN, when you're a few thousand kilometers away, is a real problem. So I started moving it to the cloud. I stood up a WireGuard VPN — modern, fast, and something I could actually reason about and operate remotely instead of inheriting a black box. Down the WireGuard rabbit hole Around that time I was deep into building my own iPhone apps. So the cloud migration turned into a personal project on the side: I built my own server and wired WireGuard into an iPhone app of my own. And to do that properly, I started studying how WireGuard actually works under the hood — the Noise protocol, the handshake, the key exchange. That study is where everything else came from. I wasn't trying to invent anything. I was just trying to understand the thing I was now responsible for. The SYN flood that primed my brain Not long before, the same company had been hit with a SYN flood attack. If you've dealt with one, you know it lodges the mechanics of connection handshakes firmly in your head — the back-and-forth, the round trips, the cost of every "hello" before any real data moves. So I had handshakes on the brain. And then, reading through how WireGuard establishes a session, a thought stopped me: Wait — does it really handsha

Sigma’s new entry is both a bold design experiment and a pretty decent camera.

A scraper can pass every check you wrote and still be wrong about the one thing you actually care about: how much it collected. No exception. No 500. No broken row. Exit code 0, logs green, every field valid. And the set on disk is a quarter of what the site actually has. I have run scrapers in production enough times to stop trusting a green run on its own, and this is the failure that taught me to count. TL;DR A paginated source can serve fewer rows than it claims and never throw — page caps, hidden offset limits, infinite scroll that "ends" early. Your status check (200), schema check (valid row), and byte check (you got data) all pass. None of them counts records. The tell: declared total vs unique ids collected. Or, when there's no declared total, the page that quietly repeats an earlier page. Below is a 40-line probe you can run right now. On a source that caps at 1,500 of a declared 4,000, it returned VERDICT: INCOMPLETE (missing 2500 rows) . This is a completeness check, not a correctness check. Different layer, different bug. What actually goes wrong You write the loop everyone writes. Walk ?page=1 , ?page=2 , keep going until a page comes back empty. Stop. Save. Done. The source has other plans. It says it has 4,000 records — the count is right there in the envelope, or in a "Showing 4,000 results" line in the HTML. But it only ever hands out real data for the first 30 pages. Page 31 doesn't error. It doesn't return empty either. It returns page 1 again. Still HTTP 200. Still 50 valid rows. Your loop has no reason to stop, so it grinds on until its own page budget runs out, collects a pile of rows, and exits clean. You now have 5,000 rows in hand and feel great about it. Looks like plenty. The catch: only 1,500 are unique. The page cap fed you the same first page over and over, and those duplicates hid the shortfall behind a big-looking row count. That is the exact shape of "50 rows passed every check while 4,000 existed" — the scraper saw a lot of rows an

Most web scraping projects are not unique snowflakes. Track competitor prices. Enrich a list of leads. Audit a site for SEO. Pull training data for a model. It is the same handful of recipes, over and over. A web scraping template is one of those recipes, pre-wired: a ready-to-use JSON config that chains the right tools in the right order, so you copy it, point it at your targets, and run. CrawlForge ships 24 of them in the templates gallery . This guide is about using them well — not just copy-paste, but read, adapt, and cost them out before you scale. TL;DR: A CrawlForge template is a copy-paste JSON config that chains multiple MCP tools into one workflow (price monitoring, lead enrichment, SEO audits, market research, AI training data). There are 24 across 9 categories, each costing 3–19 credits per run. Run them from Claude/Cursor, the crawlforge CLI, or the REST API. Free tier = 1,000 credits, no credit card. Table of Contents What Is a Web Scraping Template? Templates Gallery vs the scrape_template Tool How to Use a Template the Right Way 8 Templates Worth Copying First The Other 16 Templates Customizing or Building Your Own FAQ What Is a Web Scraping Template? A template is a saved configuration that orchestrates two or three CrawlForge tools into one workflow with a business outcome attached. Instead of wiring search_web then scrape_structured then analyze_content yourself — and guessing every parameter — you copy a config that already does it. Each template in the gallery carries: A category — E-commerce, Research, Data Collection, Monitoring, AI & LLM, Sales, SEO, Content, or Advanced Scraping (nine in total). A difficulty — beginner, intermediate, or advanced. The tool chain it runs and a fixed credit cost per run (3–19 credits). A copy-paste JSON config with sensible default parameters. You run that config from any MCP client (Claude, Cursor, Windsurf), the crawlforge CLI, or the REST API. Same config, same shape of result. Templates Gallery vs the scrap

The dangerous part of web extraction is not the error. The dangerous part is a clean JSON response that looks correct and is not. If an AI agent uses that output, the mistake does not stay inside a scraper. It moves into a report, a lead list, a price alert, a CRM update, or an automated decision. So before you trust any web extraction API in an agent workflow, test whether it fails honestly. Here is the checklist I use. 1. Test a real page, not a demo page Demo pages are usually polite little museum exhibits. Use pages that behave like the actual internet: a JavaScript-heavy product page a search results page a page with missing fields a login-gated page a blocked or rate-limited page a thin page that has layout but little useful content If the tool only looks good on clean static pages, you have not learned much. 2. Check whether the API separates fetch success from extraction success A page can return HTTP 200 and still be useless. Common examples: the final page is a login screen the visible content is a bot challenge the useful data loads after hydration and never appeared in the fetch the page contains a generic country selector instead of the product the source has navigation text but no actual item data The extractor should not treat these as success just because the request completed. A better response says something like: { "status" : "failed" , "failure_type" : "login_required" , "extracted" : null , "next_step" : "Use a public URL, authorised access, or sample content." } That is boring. Boring is good here. Boring means the agent can stop safely. 3. Ask for fields that are not on the page This is the easiest lie detector. Ask the extractor for something impossible: Extract the product name, price, stock status, warranty length, CEO name, office address, and refund policy. If the page only contains product name and price, the API should say the other fields are missing or low confidence. It should not politely invent them because the schema asked nicely.

You've been building REST APIs — one endpoint for users, another for posts, another for comments. The client makes three requests, stitches the data together, and half of it gets thrown away because it wasn't needed in the first place. GraphQL was built to fix exactly that. It gives the client full control over what data it receives. One request. Exactly what you asked for. Nothing more, nothing less. The Problem REST Couldn't Solve Before understanding GraphQL, you need to understand the two problems that drove its creation. Over-fetching The server returns more data than the client needs. GET /users/123 Response: { "id": 123, "name": "Anne", "email": "anne@example.com", "phone": "...", "address": "...", "createdAt": "...", ← you didn't need any of this "updatedAt": "..." ← but the server sent it anyway } The client only needed name and email — but it downloaded the whole object every time. Under-fetching One endpoint doesn't return enough, so the client has to make multiple requests. GET /users/123 → gets the user GET /users/123/posts → gets their posts GET /users/123/followers → gets their followers Three round trips to the server just to render one screen. On a mobile network, that cost is real. GraphQL's answer: Let the client write the query. The server returns exactly what was asked. What Is GraphQL? GraphQL is a query language for your API and a runtime for executing those queries. It was created by Facebook in 2012, open-sourced in 2015, and is now maintained by the GraphQL Foundation . Unlike REST, which exposes multiple URL endpoints, GraphQL exposes a single endpoint — typically POST /graphql . The client sends a query in the request body describing exactly what it wants, and the server responds with only that data. Key characteristics: Single endpoint — everything goes through POST /graphql Client-driven — the client defines the shape of the response Strongly typed — every field has a declared type in the schema Introspective — clients can query the API

Cloudflare Turnstile in Playwright: Why Your Tests Stall and How to Solve It in 8 Lines If you're running Playwright or Selenium against any site behind Cloudflare, you've already met Turnstile. It's the new "managed challenge" widget Cloudflare started shipping in 2023, and it now appears in front of login flows, contact forms, signup pages, and increasingly the entire site root. Here's the part most teams miss: Turnstile doesn't always show a checkbox. A lot of the time it just sits invisible, runs its scoring loop, and either issues a token silently or stalls forever. Your test doesn't crash. It just times out at the next page.click("button[type=submit]") . The CI log says "element not interactable." Nobody knows why. I work on CaptchaAI. I'm going to show you exactly what's happening, then drop in 8 lines that fix it. The real scenario You have a Playwright suite that runs every PR. One day a test starts failing on the signup flow. You re-run it. It fails again. Locally on your laptop it passes. On CI it doesn't. What's actually happening: Cloudflare flagged your CI runner's IP block (GitHub Actions, GitLab runners, Hetzner, OVH, DO — all of them are on Cloudflare's "elevated risk" list). Turnstile decides to switch from invisible mode to "managed challenge" mode. Now there's a widget in the DOM that needs a real token before the form submit will accept. Your test never interacted with the widget because last week it didn't exist. Why retries don't help The instinct is to add a retry: 2 and move on. Don't. Cloudflare's scoring is per-IP-per-fingerprint, and each retry from the same runner makes the next challenge harder, not easier. After ~3 attempts you'll get full block pages instead of the widget. The right move is to solve the widget once, inject the token, and submit normally — exactly what a human user does, just faster. How Turnstile actually issues a token The widget renders an iframe pointing at challenges.cloudflare.com . Inside the iframe it runs a fi

Researchers are using machine learning algorithms to decrypt historical pencil-and-paper ciphers.