标签：#rap

Web Scraping with Python in 2026: Best Libraries and Anti-Bot Strategies Web scraping in 2026 looks very different from 2020. Sites are smarter, anti-bot systems are more aggressive, and the legal landscape has evolved. Here's what actually works now. The 2026 Scraping Landscape Challenge 2020 Solution 2026 Solution Bot detection Rotate User-Agent Fingerprint randomization + residential proxies CAPTCHAs Manual solving Turnstile/hCaptcha solvers JavaScript rendering Selenium Playwright (faster, more reliable) Rate limiting Sleep between requests Adaptive pacing + request signing IP blocking VPN rotation Residential proxy pools Best Libraries in 2026 1. Playwright (Best for JS-heavy sites) from playwright.sync_api import sync_playwright def scrape_with_playwright ( url ): with sync_playwright () as p : browser = p . chromium . launch ( headless = True ) page = browser . new_page () page . goto ( url , wait_until = " networkidle " ) data = page . query_selector_all ( " .job-item " ) results = [] for item in data : title = item . query_selector ( " h2 " ). text_content () results . append ( title ) browser . close () return results 2. httpx + Selectolax (Fast, no JS needed) import httpx from selectolax.parser import HTMLParser def scrape_static ( url ): resp = httpx . get ( url , headers = { " User-Agent " : " Mozilla/5.0 " }) tree = HTMLParser ( resp . text ) for node in tree . css ( " .listing " ): print ( node . text ()) 3. API-First Approach (Always check first!) Many sites have hidden or public APIs that make scraping unnecessary: url = " https://www.freelancer.com/api/projects/0.1/projects/active/?query=python " data = httpx . get ( url ). json () Anti-Bot Strategies That Work 1. Request Fingerprint Randomization import random def get_random_headers (): browsers = [ " Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " , " Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 " , ] return { " User-Agent " : random . choice ( browsers ), " A

This continues from Part ① . In Part ②, we'll configure the common settings and the internal-processing Power Automate flows. Common Settings Setting Up Connections Open Power Automate ( https://make.powerautomate.com ) Go to "Data" → "Connections" → "New connection" and create a Microsoft Dataverse connection Do the same to create an Office 365 Outlook connection Basic Flow Creation Steps Click "Create" → select "Automated cloud flow" (event-triggered) or "Scheduled cloud flow" (recurring) Name flows in the format [Zone]-[Number] [Description] (e.g., "A-1 Opportunity Stage Stall Alert") Always run a test after creating a flow to verify it works 2. Internal-Processing PA Flows — 4 Flows (Write-back portions of A-4, C-5, C-6, D-3) Once the common settings are done, it's time to build. A-4: Write Back Stage Changed Date Without this flow, the stall-day calculations in A-1 and B-1 will not work. Implement this first. In Microsoft Dynamics 365 (D365), a "stage" refers to a major milestone in a process — such as a sales deal or customer engagement — that guides the responsible person through what needs to happen next. It's how a series of activities is visualized and managed. From here, all work is done in Power Automate. Step Task Details 1 Create the flow "Automated cloud flow" → select trigger "When a row is added, modified or deleted (Dataverse)" 2 Configure trigger Table: Opportunities / Change type: Modified 3 Add condition Add a "Condition" action: "When Status Reason (statuscode) has changed" 4 Write-back action "Update a row (Dataverse)" → set cr917_stage_changed_date to utcNow() C-5: Auto-Set Renewal Date + Auto-Create Renewal Opportunity (on Won) On Won close, two things happen: ① auto-set the renewal date to close date + 365 days, and ② auto-create a new Opportunity for the renewal cycle and add it to the pipeline. Step Task Details 1 Create the flow "Automated cloud flow" → trigger "When a row is added, modified or deleted (Dataverse)" 2 Configure trigger Ta

Interesting research on a new class of weak RSA keys: keys with lots of zeros. It turns out that these keys are out in the wild. The badkeys project is an open-source service that checks public keys for known vulnerabilities. While developing this tool, Hanno collected a massive number of real-world keys from public sources, including Certificate Transparency logs, internet-wide TLS and SSH scans, PGP keys, and many others. By searching this dataset for unexpectedly sparse RSA moduli, we uncovered a large number of keys in the wild with the patterns in Figure 1...

After writing drivers for NVMe storage, my bare-metal kernel could load files and run JIT code. However, I was still typing commands into a text-only COM1 serial terminal. I needed a graphical interface. Last night, the second agent took over to build a double-buffered visual rendering compositor on top of the UEFI Graphics Output Protocol (GOP) framebuffer. The V.E.L.O.C.I.T.Y.-OS 12-Part Roadmap We are building a bare-metal, self-healing operating system running entirely inside the CPU's L3 cache. Here is the roadmap for this 12-part series: Part 1: The Spark — Exposing the "Safe-Room" security leak and building the compiler gate. Part 2: The NDA Language — Designing a content-addressed triplet representation to cure context bloat. Part 3: Ditching the Web Stack — Building a native 30MB IDE with 1,500,000x IPC latency drops. Part 4: The Closure JIT — Compiling AST blocks to nested closures and bypassing borrow checker limits. Part 5: JIT Math Optimizations — Replacing division operations with precomputed 16-bit lookup tables. Part 6: x86-64 Assembler & SCEV-Lite — Compiling scalar loops directly to native code in constant time. Part 7: Classic Compiler Passes — Implementing inter-procedural Dead Code Elimination and loop unrolling. Part 8: Reclaiming Ring 0 — Exiting UEFI boot services and transitioning the kernel to Ring 0. Part 9: Bare-Metal Drivers — Writing a PCI scanner, NVMe block storage controller, and FAT32 parser. Part 10: Synaptic Canvas — Rendering a spatial, force-directed GUI based on model token activation vectors. (You are here) Part 11: Swarms & Hot-Patching — Building multi-agent scheduling and zero-downtime RCU driver updates. Part 12: Self-Evolution — Handing system control over to a local LLM Terminal that self-optimizes via telemetry. This led to the design of the Synaptic Canvas GUI . The Swappable GUI Engines I started by mapping the physical screen buffer pointer discovered by UEFI GOP. I implemented a double-buffering scheme: drawing elem

Why your Cloudflare Turnstile token works in the browser but 403s from requests You solved the Turnstile widget. You can see the token in the page. You copy it into your script, POST the form from requests, and the server hands you back a 403 — or a JSON body with "success": false. The token clearly worked a second ago in the browser, so what changed? Short answer: a Turnstile token is not a password you can carry around. It's a one-time, short-lived proof bound to a very specific context, and replaying it from a different context is exactly what it's designed to reject. Below is what that context is, how to tell which constraint you're hitting, and the fix for each. The real scenario You're automating a flow on a Cloudflare-protected site. There's a cf-turnstile widget on the form. You get a token one of two ways: you render the page in a real browser (Playwright/Selenium) and read cf-turnstile-response, or you hand the sitekey + page URL to a solving service and get a token back. Either way, you then submit the form with a plain HTTP client requests, httpx, axios) and it fails. The frustrating part: it's intermittent-looking. The reason it feels random is that there are four separate constraints, and you're usually tripping a different one each time. The four things a Turnstile token is bound to 1. It's single-use Once Cloudflare validates a token server-side (the siteverify call your target makes), that token is spent. Submit twice, retry, or test it once by hand, and the second use returns false. You get a fresh one per submission. 2. It has a short TTL Turnstile tokens expire fast — a few minutes. Solve early, do other work, submit later, and the token can be dead on arrival. The widget auto-refreshes in the browser precisely because tokens go stale; a script that grabs the token and sits on it loses that refresh. 3. It's bound to the sitekey and the page URL Multiple widgets. Some pages embed more than one Turnstile (login + newsletter). Solving the wrong site

Playwright and Selenium are great until you hit bot detection. Google OAuth, Cloudflare, and Vercel checkpoints all flag headless browsers. Here's how to control a real Chromium instance via CDP using Python and websockets. Why Not Playwright? Playwright launches a headless browser with automation flags. Even in headed mode with Xvfb, Google detects it. The CDP Approach Launch Chromium with remote debugging: chromium-browser --user-data-dir = /path/to/profile --remote-debugging-port = 9222 --no-first-run Connect via WebSocket in Python: import asyncio , json , websockets , urllib . request async def get_page_ws (): resp = urllib . request . urlopen ( ' http://localhost:9222/json ' ) targets = json . loads ( resp . read ()) for t in targets : if t [ ' type ' ] == ' page ' : return t [ ' webSocketDebuggerUrl ' ] async def cdp_call ( ws , method , params = None ): msg_id = cdp_call . id = getattr ( cdp_call , ' id ' , 0 ) + 1 msg = { ' id ' : msg_id , ' method ' : method } if params : msg [ ' params ' ] = params await ws . send ( json . dumps ( msg )) while True : resp = json . loads ( await ws . recv ()) if resp . get ( ' id ' ) == msg_id : return resp Key Advantages Real browser fingerprint, no automation flags Persistent sessions, cookies survive across runs Google OAuth works, existing sessions carry over No bot detection, it IS a real browser Follow for more tutorials on browser automation and AI agent architecture.

If you ship a desktop app outside an app store, you eventually hit the same wall: how do you check a license when the user is on a plane, behind a corporate firewall, or just offline? Calling your server on every launch isn't an option. Here's how offline activation actually works, without the hand-waving. The naive version, and why it breaks The first thing everyone reaches for is "call home on launch, get back yes/no." It works in the demo and fails in the wild: No network = no app. Fail-closed locks out paying customers. Fail-open means anyone who blocks your domain runs free. Both are bad. A boolean is forgeable. If your app trusts a {"valid": true} response, a proxy or a patched DNS entry returns that for free. The fix isn't a better endpoint. It's moving the trust off the network and onto cryptography. The model that works: signed leases The durable pattern is a cryptographically signed lease (Keygen calls these license files, Keylight calls them leases — same idea): On first activation, the device talks to the server once . The server returns a small signed document: the license state, an expiry, the device binding, and any entitlements (which features/tiers are unlocked). The document is signed with the server's private key (Ed25519 is the modern choice — small, fast, boring in the good way). Your app ships the matching public key and verifies the signature locally on every launch. No network needed. Because the app only ever verifies with a public key, there's nothing secret in the binary to steal, and a forged lease fails the signature check. That's the whole trick: the server vouches once, math vouches forever after. first launch ──► server signs lease (Ed25519, private key) ──► stored on device every launch ──► app verifies signature (public key) ──► no network Device binding (so one key isn't infinite installs) A lease is bound to a device so a single license can't be pasted onto a thousand machines. The lease embeds a device fingerprint, and the SDK ch

Diagrid has announced the release of Dapr 1.18, introducing what it calls Verifiable Execution, a new set of capabilities designed to bring cryptographic trust, provenance, and tamper-evident execution records to distributed applications and AI agents. By Craig Risi

Cross-posted from noflattery.com/decide — where I ran this exact question through a council of four different frontier models and let them argue it out. You're a solo founder at ~$8K MRR. You have runway for exactly one full-time hire. Which role unlocks the most growth? (A) a second engineer to ship features faster (B) a marketer to build a real acquisition channel (C) a customer-success / support hire to cut churn and free your time (D) a salesperson to chase larger deals The intuitive answer for most technical founders is A — more shipping velocity. The case below is for C , and it's stronger than it looks. (With one caveat that can flip the whole thing — stick around for it.) TL;DR: At ~$8K MRR solo, hire customer success first if churn is real or support is eating your week . If voluntary churn is under ~3% and support is light, hire a marketer instead. Engineer and sales come later. The case for customer success first 1. Churn quietly eats growth before features can add it. At $8K MRR, 5% monthly churn is ~$400/month bleeding out before you grow an inch. Across bootstrapped SaaS in the $5–15K MRR band, the strongest predictor of reaching $50K isn't feature velocity or channel — it's net revenue retention above 90% . That's a customer-success function, not an engineering one. 2. You are the bottleneck, and support is eating you. As a solo founder you're doing product, sales, billing, and support. If support takes ~15 hours a week, that's nearly 40% of your capacity — and it's the cheapest thing to hand off. A CS hire costs less than a senior engineer or an experienced salesperson, and it buys back the hours (and the headspace) you need to think strategically again. 3. It's a research department in disguise. A CS hire generates the highest volume of qualitative signal: why people leave, what they actually use, what they'd pay more for. An engineer builds what you think users want. CS tells you what they actually need — which means the engineer you hire next buil

Sony's A7R VI finally serves up speed and resolution together thanks to the new 67MP stacked sensor.

Hey Dev Community, If you are running enterprise-scale web scrapers, pricing monitors, or data ingestion pipelines for LLMs, you’ve probably spent sleepless nights dealing with network latency and sudden 403 blocks. When choosing an infrastructure partner, every provider pitches the same script: "99.9% uptime guarantees, millions of residential IPs, and lightning-fast response times." But in the trenches of real-world data collection, we all know that marketing numbers rarely match production reality. Last quarter, my team ran an exhaustive infrastructure audit to compare proxy providers pricing performance and infrastructure stability. If you want to dive straight into our live dataset, telemetry scripts, and interactive monitoring utilities, you can check out the full workbench at ProxyVero . Here is a technical breakdown of how we built our benchmarking matrix, and the architectural gaps we discovered across mainstream enterprise proxy services. 📊 1. The Core Metrics: Uptime vs. Success Rates The biggest lie in the networking industry is confusing Server Uptime with Request Success Rate . A proxy gateway server can maintain a 99.9% uptime while the underlying residential peer network is failing 20% of your data collection requests due to strict target WAFs or high peer churn. When conducting our proxy providers uptime guarantees performance benchmarks , we evaluated three core parameters: TCP Handshake Latency : The time it takes to establish a connection with the proxy endpoint. TTFB (Time to First Byte) : Critical for parsing dynamic JavaScript targets. HTTP Status Code Reliability : Tracking the exact ratio of 200 OK vs. 403 Forbidden / 429 Too Many Requests . ⚖️ 2. The Big Three: Oxylabs vs Bright Data vs SmartProxy Comparison To provide an objective proxy network performance benchmarks comparison , we deployed standard headless browser worker instances (Playwright/Puppeteer) routed through different enterprise gateways. Below is a high-level summary of our a

One of the most powerful upgrades you can give any AI developer Introduction AI coding assistants are dazzling on a single file and surprisingly lost on a large one. Point a capable agent at a mature, multi-package codebase and you watch the same pattern every session: it greps for a symbol, opens a dozen files to work out how they fit together, and burns a large slice of its context window simply rediscovering the shape of the system before it can do any actual work. That orientation phase — the crawling, the grepping, the file-by-file reconstruction of structure the codebase already encodes — is the single biggest waste of tokens in most AI-assisted workflows. And it repeats every session, because nothing persists. The fix is straightforward: give the agent a map. Model your codebase as a knowledge graph and let the agent query the map instead of crawling the territory. This article explains what that looks like, why it works, and what it actually finds when you run it on a real system. TL;DR A code-graph should map your architecture not just your code. Converting grep to graph minimises token usage and saves you time. Find bugs, security mistakes, and omissions in your codebase in seconds. Plan upgrades quickly and with far more accuracy. Using deep-memory's vocabulary simplifies usage for your AI. Use deep-memory free. Check out the full example code-graph-guide.md What is a code-graph A code-graph is a graph database representation of your system. I use the word system and not code deliberately — the real power comes from driving architectural insights, not just building a faster file search. Code graphs are becoming popular. There are some impressive repositories where the author has scripted a process to mirror a codebase into a graph DB, capturing files, imports, and call relationships. That approach is useful for dependency visualisation. But mirroring syntax only scratches the surface. What separates a useful code-graph from an elaborate directory listing

Order warns of national security risks if post-quantum cryptography isn't adopted in time.

A static-analysis approach to unifying 46 repositories (37 air-closet-side + 9 mall-side) of legacy production code into one knowledge graph. Why simply 'letting AI read the code' isn't enough, why I had to chase down boundary nodes (API endpoints, DB tables, Event topics), how I dealt with framework and library diversity, and what 3 months of trial and error solved or didn't solve — looking back through actual git history.

Every World Cup there's a moment. Some player nobody outside their domestic league had heard of scores an absolute screamer in a knockout match, and by the time they've finished celebrating, their follower count is climbing like a rocket. I always found that fascinating, but I could never see it happening. By the time the "X gained 3M followers!" tweets show up, the surge is already over. So this tournament I built a little tracker that snapshots player follower counts on a schedule and shows me the growth curve in near real-time. Here's how it works. The problem with doing this "properly" My first instinct was the official APIs. That died fast. Instagram's Graph API won't give you follower counts for accounts you don't own. TikTok's Research API is academics-only and takes weeks of applications. X's API now starts at $100/month and climbs steeply from there. I just wanted public follower counts — numbers anyone can see by opening the app. I didn't want a data partnership and a legal review. I ended up using the SociaVault API , which wraps public profile data from each platform behind one key. One request, one credit, JSON back. The shared client Everything runs through one tiny helper: // Node 18+ has fetch built in const API_KEY = process . env . SOCIAVAULT_API_KEY ; const BASE = " https://api.sociavault.com " ; async function sv ( path , params ) { const url = new URL ( BASE + path ); Object . entries ( params ). forEach (([ k , v ]) => url . searchParams . set ( k , v )); const res = await fetch ( url , { headers : { " X-API-Key " : API_KEY } }); if ( ! res . ok ) throw new Error ( ` ${ res . status } ${ await res . text ()} ` ); return res . json (); } Grabbing follower counts across platforms Each platform nests the count slightly differently, so I use fallback chains to stay defensive: async function instagramFollowers ( username ) { const data = await sv ( " /v1/scrape/instagram/profile " , { username }); const p = data . data ?. user ?? data . data ?? data

It's unclear whether the system is currently intact.

Maggot therapy lacks robust data, but it has fans and a fail-safe "bacon therapy."

GraphQL: A Complete Guide for Developers in 2026 NEWS: MY GAME JUST LAUNCHED Flip Duel Card Battle - Apps on Google Play Outsmart rivals in 1v1 card duels. Joker, bluff, ranked PvP. 5 rounds. play.google.com If you have built more than a couple of APIs, you have probably felt the friction of REST at scale. You ship an endpoint, the frontend team asks for one more field, you version the route, the mobile team needs a different shape of the same data, and six months later you are maintaining /v3/users/:id/full next to /v2/users/:id/summary and nobody remembers which one the Android app actually calls. GraphQL was built to kill that exact pain. It is a query language and runtime that lets clients ask for precisely the data they need — no more, no less — from a single endpoint, against a strongly typed schema that doubles as living documentation. This guide walks through GraphQL from first principles to production concerns. It is aimed at working developers, so expect schema definitions, resolvers, real queries, the N+1 problem, federation, security, and the parts of the ecosystem that actually matter in 2026. By the end you should be able to decide whether GraphQL belongs in your stack and how to build it without shooting yourself in the foot. What GraphQL Actually Is GraphQL is a specification, not a library or a framework. It was created at Facebook in 2012 to power their mobile apps, open-sourced in 2015, and is now governed by the GraphQL Foundation under the Linux Foundation. The spec defines a query language, a type system, and an execution model — but it deliberately says nothing about which database you use, which programming language you implement it in, or how you transport requests over the wire. That last point trips people up, so let it sink in: GraphQL is transport-agnostic and storage-agnostic. Most implementations run over HTTP with JSON, but that is a convention, not a requirement. Your resolvers can pull data from PostgreSQL, a REST microservice, a gR

Gene therapy company UniQure had another FDA meeting after Vinay Prasad's exit.

Introduction Extracting data from older websites is a technical challenge that goes beyond simple copy-pasting. The example website provided illustrates this perfectly: its outdated design, reliance on mouse-over interactions, and lack of structured export options create a perfect storm of extraction difficulties. This article dissects these challenges and provides a roadmap for extracting both visible content and mouse-over images while preserving data integrity. The Core Problem: Legacy Technology Meets Modern Needs The website's URL parameters ( screen_width=0&screen_height=0 ) immediately signal a legacy system likely built for a bygone era of fixed-width displays. This design choice breaks modern scraping tools that expect responsive layouts. The mouse-over images, critical to the site's content, are dynamically loaded via JavaScript , meaning they don't exist in the initial page source. This requires simulating user interactions to trigger their appearance, a task beyond basic HTML parsing. Why Manual Extraction Fails Attempting to manually save images or copy text from this site is a losing battle. The mouse-over images, for instance, are not directly downloadable – they're embedded in JavaScript events. Even if you could save them individually, maintaining their association with the corresponding visible content would be error-prone and time-consuming. This method also fails to scale for larger websites with hundreds of such elements. The Technical Solution: A Multi-Pronged Approach Effective extraction requires a combination of techniques: Browser Automation: Tools like Selenium or Puppeteer can simulate mouse movements to trigger hover events, capturing both visible and hidden content. This method mirrors human interaction , ensuring all dynamic elements are revealed. Network Request Inspection: Analyzing the website's backend requests using browser developer tools can reveal direct URLs for mouse-over images , bypassing the need for hover simulation. This