标签：#t

If you have written more than a couple of scrapers, you already know the pattern. The first few hundred requests fly through. Then responses slow down, you start seeing 429 Too Many Requests , a captcha wall appears, and finally the target just returns empty pages or a hard 403 . Your code did not change. Your IP did. Scraping at any real volume is less about parsing HTML and more about managing where your requests come from. This post is a practical walk-through of how proxies fit into a scraping pipeline: why a single IP fails, what proxy types actually matter, how rotation works, and how to wire it all up in Python with requests , aiohttp , and Scrapy. There is code you can copy, plus the mistakes that cost me the most time. Why one IP is never enough Every site you scrape sees the same thing: a stream of requests from one address, arriving faster and more regularly than a human ever would. Anti-bot systems are built to spot exactly that. The signals they use are boring but effective: Request rate per IP. Too many hits in a short window trips a rate limiter. Volume over time. Even a slow scraper eventually stands out if every request comes from the same address for hours. Behavioral fingerprint. No mouse, no scroll, identical headers, requests in perfect intervals. Reputation. Datacenter ranges that have been abused before are pre-flagged. You can soften some of these with headers, delays, and a real browser, but there is a ceiling. Once a single IP has made enough requests, it gets throttled or blocked regardless of how polite you are. The only way past that ceiling is to spread requests across many addresses, so no single one crosses the threshold. That is the entire job of a proxy pool. The proxy landscape, minus the marketing Providers love to complicate this. For scraping, the distinctions that actually change your results are these: Shared vs private. Shared proxies are handed to many customers at once. You inherit everyone else's behavior, so an address ca

At Mehran University of Engineering and Technology (MUET), Jamshoro, results are traditionally announced via large, static PDF tables. But the main issue is: Every semester, the same story. Need to check your result? Open your laptop. Connect to the university network... or set up a VPN. Want to know your actual class or batch rank? Good luck guessing. That frustration became my latest project. To solve this, I set out to build the MUET Results Portal ( https://muetresults.vercel.app )—an independent, open-source lookup engine and administrative compiler that provides students with instant semester results, CGPA calculations, batch standings, and interactive academic calendars. Here is an engineering deep-dive into how I built it using a serverless GitOps pipeline, vanilla JavaScript SPA, and Gemini AI. 🛠️ The Architecture & Data Pipeline To keep the platform hosting costs at absolute zero while maintaining lighting-fast page loads, I designed a pre-rendered static pipeline. Rather than querying a database at runtime, all student data is compiled statically. Here is the GitOps workflow: Official PDF Release : The Mehran University Examination Department publishes a new results PDF. LLM OCR Parsing : Via a secure administrative panel ( /mokshadmin ), I upload the scanned PDF/image. A serverless backend function streams the document to the Google Gemini 1.5 Flash API , which returns structured JSON student records. Git Database Update : The approved JSON records are committed back to the repository's git-tracked database ( muet_student_gpa_dataset.csv ) using the GitHub REST API. CI/CD Pre-rendering Build : The new commit triggers a Vercel build hook. Node compilation scripts read the CSV database and: Group records and compile them into static runtime JSON structures. Pre-render complete static HTML folder structures for all batch rankings and departments. Regenerate SEO sitemaps ( sitemap.xml ). Instant Deployment : Vercel serves the pre-rendered static files instan

The problem I've been working with Manticore Search for about two years at EricaPRO , building the search layer for two financial data platforms. For the past year, that work has been a Laravel API. Manticore was never the problem. It's fast and it's stable. The problem was the gap between Manticore and Laravel. I had already built a package for this — laravel-manticore-search , a fluent builder over Manticore's HTTP/JSON API. It works, and it's still in use. But it's a client wrapper. Every feature had to be implemented manually. Every new filter or facet meant more custom architecture around the client, and none of the things Laravel gives you for free — models, migrations, pagination, casts — applied to it. Scout doesn't close that gap either. Scout gives you search() . No full query builder, no migrations for your indexes, sync is your problem. That's not a criticism — Scout abstracts over engines with completely different APIs, so it exposes the lowest common denominator. It just wasn't what I needed. What I needed was simple to describe and annoying to not have: something plug and play. Something Laravel way. Point Eloquent at Manticore and use Eloquent. The insight Manticore speaks the MySQL wire protocol. Out of the box, port 9306. You can connect to it with any MySQL client and run SQL. I had been using that port for two years without thinking about what it meant for Laravel. Because here's the thing: all of Eloquent — models, query builder, migrations, pagination, chunking — sits on top of a Connection and a Grammar . The grammar compiles builder calls into SQL for a specific dialect. That's the entire mechanism behind Laravel supporting MySQL, Postgres, SQLite and SQL Server with one codebase. So the real question was never "how do I re-implement Eloquent on top of Manticore's client?" It was "how thin can a Manticore grammar be?" If Manticore accepts MySQL-protocol connections and mostly-MySQL SQL, then a Laravel database driver — a custom connection plu

This is today’s edition of The Download, our weekday newsletter that provides a daily dose of what’s going on in the world of technology. What Anthropic’s latest AI discovery does—and doesn’t—show —James O’Donnell When Anthropic announced last week that it had found a new window into its models’ “internal thoughts” as they reason through answers,…

If you want to stream local media, this free and open source media server is just as good as Plex. But if you rely on remote access or live TV, prepare to tinker.

The Linkerd community has announced the release of Linkerd 2.20, introducing a series of performance, observability, and traffic management enhancements that further strengthen the CNCF-graduated service mesh's position as a lightweight alternative for Kubernetes networking. By Craig Risi

Demis Hassabis thinks the world needs an AI watchdog with the power to hit the brakes if frontier models become too dangerous. Writing in a blog post, the Google DeepMind CEO and cofounder said the US should lead the initiative, arguing that the country is the best place to set global standards "given its economic […]

Hello, I'm Maneshwar. I'm building git-lrc, a Micro AI code reviewer that runs on every commit. It is...

RGB TVs are the latest hotness in the world of televisions, and I’ve tested many of the latest models to see which you should buy.

There’s more to Google’s messaging app than you might realize.

Reflect Orbital has been given the green light to launch its first space mirror that aims to redirect sunlight down to Earth at night. The US Federal Communications Commission (FCC) has authorized the California-based startup to build and operate a single prototype satellite in low-Earth orbit later this year, despite concerns over how the technology […]

It’s not the upgrade Ring 4 owners have been waiting for, but it’s easily the best smart ring Oura has ever made.

FIFA’s network was vulnerable to anyone with even minimal access.

Build landing pages with Aceternity UI Discussion | Link

Armed with high-end knives and digital calipers, Germanbreadcutter has entranced thousands of fans, one loaf at a time.

An increasing number of people seem to agree the internet is terrible for children - allegedly addictive, destructive to self-esteem, possibly a portal to predators. Over the past year, several countries have started requiring stringent age verification or outright bans for minors. At the end of June in the US, the House of Representatives passed […]

75 ready-to-use email templates for work & job search Discussion | Link

On this week’s episode of The Big Interview podcast, WIRED’s senior culture editor Manisha Krishnan talks to Gowanus about eschewing Big Tech, going outside, and rejection in the age of dating apps.

Google released the Genkit Agents API in preview for TypeScript and Go. The open-source framework packages message history, tool loops, streaming, and state persistence behind a single chat() interface. Detached turns let agents work after clients disconnect. Interruptible tools provide human-in-the-loop control with anti-forgery validation on resume. By Steef-Jan Wiggers

The Architecture Shift: SPA vs. Framework Internationalization (i18n) is one of those features that feels straightforward in a Single Page Application (SPA). You install react-i18next , wrap your app in a provider, and you're good to go. However, when you decide to migrate that Vite-based React app to Next.js for better SEO and performance, the strategy for i18n changes fundamentally. In a Vite SPA, i18n is typically client-side. In Next.js, i18n happens at the routing and server level. If you don't plan the migration carefully, you'll end up with hydration mismatches, flashing text, or broken search engine indexing. Here is how to navigate the transition. 1. Defining the Routing Strategy In Vite, your translations often live in the same bundle, and you swap them out using a state hook. Next.js, particularly with the App Router, prefers sub-path routing (e.g., /en/about or /es/about ). This is crucial for SEO because it allows search engines to crawl localized versions of your pages individually. Instead of relying on localStorage to remember a user's language, you should now rely on the URL. Most teams moving from Vite use a middleware approach to detect the user's preferred locale and redirect them to the correct sub-path. 2. Choosing the Right Library If you were using react-i18next in your Vite project, you have two main paths in Next.js: next-i18next (Pages Router): The traditional choice for the Pages Router. next-intl or i18next + i18next-resources-to-backend (App Router): These are modern solutions that leverage Server Components. When handling complex migrations involving many components, using a specialized tool like ViteToNext.AI can help automate the transformation of your Vite project structure into a Next.js-ready architecture, saving you hours of manual refactoring. 3. Handling Server Components vs. Client Components one of the biggest hurdles is that useTranslation() hooks from standard i18n libraries are "Client hooks." In the App Router, you'll wan