标签：#Python

Hello DEV Community! 👋 I wanted to share my latest open-source project: Hunter-Base-Intelligence (v17 Plus). It is a fully local-only cryptocurrency intelligence dashboard that scans DEX tokens on the Base blockchain, scores them using a multi-factor logic, and simulates a paper-trading shadow portfolio. 🛡️ Why Local-Only? Most on-chain analytics tools require sensitive private keys, leak user data, or rely heavily on slow, paid external infrastructure. I engineered this tool to be fully local —it requires no wallets, no seed phrases, and sends your data nowhere. Pure local analysis using Python , Flask , and SQLite . ⚙️ How It Works (Core Architecture) The ecosystem runs on a continuous ~60-second scan cycle: scanner.py : Discovers active and newly created tokens using DexScreener, BaseScan, and direct EVM RPC factory logs. scorer.py : Every token is evaluated across 6 independent dimensions (Momentum, Manual Trade Feasibility, Execution Reality, Money Flow, Multi-Timeframe Pulse, and Composite Rank). hunter_court.py : A proprietary "Court" analytics engine that runs a risk-free paper-trading shadow portfolio with realistic gas, fee, and slippage simulation. It evaluates its own past decisions to continuously calibrate scoring thresholds! 📊 System Features Adaptive Exit Parameters: Automated position sizing and execution simulation ( exit_engine.py ). System Guardian: Keeps the system running 24/7 with auto-restart on crashes and automatic local database backups ( system_guardian.py ). Beautiful Dashboard: Clean, real-time local web interface for tracking active simulated trades and market analytics. 📂 Explore and Contribute The project is licensed under the MIT License and is open for contributions. Whether you want to optimize the scoring algorithms, expand the web API endpoints, or improve the dashboard frontend, feel free to dive in! 👉 Check out the Repository here: https://github.com/shbadrconsulting-source/Hunter-Base-Intelligence I would love to hear your fe

Every LLM-powered app I'd built up to this point followed the same recipe (pun intended): call an API, write a good prompt, wrap it in a nice UI. That's a legitimate way to build things, but at some point I wanted to actually understand what was happening inside the model I was calling and not just how to prompt one. So for my recipe app Rasaveda , I decided to skip the API entirely. Intially, I had one made, but then I felt like I was not making any clear progress in actual machine building. So I ditched the entire external API callings. No OpenAI, no HuggingFace inference endpoint, no pretrained weights. I wrote a decoder-only transformer from scratch in PyTorch, trained it on a single Colab T4, and shipped it as the actual language model powering the app in production. This post is a lazy attempt at what that looked like. The architecture, the training runs, the mistakes, and what I'd tell someone about to try the same thing (do at your own risk). What Rasaveda actually does Rasaveda is a full-stack recipe intelligence app: you give it the ingredients sitting in your kitchen, it does a semantic vector search (ChromaDB + all-MiniLM-L6-v2 ) over 365 recipes to find the best matches, tells you exactly what you're missing, and can critique or explain any cooking step conversationally. It also has a somewhat unnecessary but delightful feature where you pick a theme by clicking one of 36 Indian states on a geographically accurate SVG map (original idea lol). The part I actually want to talk about is RasavedaGPT , the model that generates every word of AI output in the app, running in-process inside the FastAPI backend. Why build the model instead of calling one Two reasons, one practical and one selfish. The practical one: I wanted a fully self-contained, dependency-free inference path without any API keys, no rate limits, no per-token cost, no vendor to go down at 2am. For a small, domain-specific task like "reason about recipes," a giant general-purpose model is over

An MCP server is a small app that extends an AI model's capabilities by giving it access to custom tools, a particular set of data or workflows. It's based on the Model Context Protocol, which is an open standard for connecting AI apps with these external sources. The most straightforward way to create an MCP server is to use the official SDK, implement a single function and mark it as a tool and then expose it through stdio (standard input/output) which you can register in Claude Code; it basically boils down to a single Python file with a single tool and connecting it end-to-end took us around 10 minutes. Background Generally, the Model Context Protocol defines two sides: The server — it's the app you write that you use to publish tools/data The client — for example Claude Code; it finds and calls available tools based on your permission As for the main purpose of the Model Context Protocol — before it was introduced, every AI app needed its own custom integration with every tool; the Model Context Protocol replaces this with a single standard connector, so to say it's like USB-C for the AI world — you have a single standardised port instead of having to use a separate cable with every device. In terms of the protocol, a tool is just a function that the model can decide to call. So if you want to build an MCP server, you do it when you want your model to have access to some resources you have (like your internal API or database for example) which aren't available through any of the already-published servers. Let's have a look at a minimal example of what such server might look like — a single Python file with a single tool that returns the number of words, characters and lines in the input text. Scaffold the project To set up the project we used uv (a CLI for managing Python projects) and installed the official SDK: uv init word-count-mcp cd word-count-mcp uv add "mcp[cli]" uv init word-count-mcp — initialises a new project called "word-count-mcp" with an uv proje

int(os.environ.get("PORT", "8080")) fails constantly in ways that waste your time: ValueError: invalid literal for int() with base 10: 'abc' . No variable name. No hint about what a valid value looks like. You grep the codebase for PORT to even find where the read happened. specenv is a zero-runtime-dependency Python library for typed environment variable loading — casting, validation, schema grouping, prefix namespacing. All of that is useful, but none of it is the actual design decision worth writing about. The decision that shaped everything else was: every error must name the variable and say how to fix it, unconditionally, with no opt-out. Decision 1: The error message is generated at the failure site, not templated afterward It would be easy to build one generic EnvCastError(var_name, raw_value, target_type) and format a message from those three fields in __str__ . specenv doesn't do that — each cast failure builds its own message inline, at the point where the specific failure is known: if cast_type is int : try : return int ( raw ) except ValueError : raise EnvCastError ( f ' Cannot cast { name } = { raw !r} to int. \n ' f ' → Set { name } to a valid integer (e.g. { name } =8080) ' ) from None if cast_type is bool : ... raise EnvCastError ( f ' Cannot cast { name } = { raw !r} to bool. \n ' f ' → Set { name } to one of: 1/0, true/false, yes/no, on/off ' ) The generic version would produce "Cannot cast PORT='abc' to int" and stop there. The inline version gets to add (e.g. PORT=8080) for ints, 1/0, true/false, yes/no, on/off for bools, a namespaced hint for prefixed variables — because at the point of failure, you know exactly what a correct value looks like for that type, and a generic formatter three calls up the stack doesn't. The cost is a few lines of duplication across _caster.py 's type branches. That's a fair trade for every single error message being genuinely actionable instead of generically accurate. Decision 2: Missing-and-required collapses to t

Most ad-hoc timing code in Python looks like this: start = time . perf_counter () result = do_work () elapsed = time . perf_counter () - start stats [ name ]. append ( elapsed ) It works, until do_work() raises. Then the line that records the timing never runs, the exception propagates, and the one call that was probably slowest — the one that failed — is silently missing from your stats. If you're using timing data to find what's expensive, the failing case is exactly the one you can least afford to lose. timerx is a small, dependency-free Python timing library — a decorator, a context manager, and named stopwatches, all backed by one stats store. The one rule that shapes the whole implementation: a timing gets recorded whether or not the timed code raised. Decision 1: finally , everywhere, no exceptions to the rule @functools.wraps ( target ) def wrapper ( * args : Any , ** kwargs : Any ) -> Any : started = self . _clock () try : return target ( * args , ** kwargs ) finally : elapsed = self . _clock () - started with self . _lock : self . _record ( label , elapsed ) return wrapper The async wrapper is the identical shape with await added. The context manager ( _Lap ) does the same thing structurally, just split across __enter__ / __exit__ instead of try / finally : def __exit__ ( self , * exc_info : object ) -> bool : if self . _started is None : raise RuntimeError ( " timerx lap exited before it was entered " ) elapsed = self . _timer . _clock () - self . _started with self . _timer . _lock : self . _timer . _record ( self . _name , elapsed ) return False Note the return False — __exit__ deliberately never swallows the exception. It records the timing and lets the exception continue propagating unchanged, because a timing library has exactly one job here: observe, not intervene. A version that suppressed exceptions to "clean up" would be actively dangerous to drop into someone else's codebase. Three entry points — decorator, context manager, stopwatch — and all t

GitHub热门项目 | 📑 PageIndex: Document Index for Vectorless, Reasoning-based RAG | Stars: 34,513 | 222 stars today | 语言: Python

GitHub热门项目 | Dark Web OSINT Tool | Stars: 4,414 | 35 stars today | 语言: Python

GitHub热门项目 | Apache Superset is a Data Visualization and Data Exploration Platform | Stars: 73,970 | 23 stars today | 语言: Python

GitHub热门项目 | Easy Data Preparation with latest LLMs-based Operators and Pipelines. | Stars: 6,962 | 139 stars today | 语言: Python

GitHub热门项目 | Open-source NotebookLM alternative. Research the open web with live data(Reddit, YT, IG, TikTok, Google Search, Maps etc) through one platform, API or MCP server. Join our Discord: https://discord.gg/ejRNvftDp9 | Stars: 15,451 | 35 stars today | 语言: Python

Before you can truly understand how AI systems think, learn, and generate responses, you need to understand the math that powers them. This guide covers the essential mathematical concepts that form the backbone of modern Artificial Intelligence and Large Language Models (LLMs). Why does this matter? Every aspect of AI — from how text is encoded, to how a model predicts the next word, to how it improves itself during training — is driven by mathematics. Skipping this foundation means you will only ever use AI as a black box, without understanding why it works. 🔄 How an LLM Actually Works — The Complete Pipeline Before diving into each math concept individually, here's the big picture of how text flows through a Large Language Model from input to output. Every section in this guide maps to a step in this pipeline: ┌─────────────────────┐ │ Your Prompt │ "What is gravity?" └──────────┬──────────┘ ↓ ┌─────────────────────┐ │ Tokenizer │ Splits text into chunks (BPE algorithm) └──────────┬──────────┘ → Section 1: Number Systems & Encoding ↓ ┌─────────────────────┐ │ Token IDs │ Each token → a number (e.g., "gravity" → 17942) └──────────┬──────────┘ → Section 1: Number Systems & Encoding ↓ ┌─────────────────────┐ │ Embedding Model │ Each token ID → a dense vector of numbers └──────────┬──────────┘ → Section 3: Vectors & Embeddings ↓ ┌─────────────────────┐ │ Vectors │ [0.12, -0.87, 0.45, ...] per token │ + Positional Info │ → Section 3 & 6: Embeddings & Linear Algebra └──────────┬──────────┘ ↓ ┌─────────────────────┐ │ Transformer │ Multi-Head Attention + Feed-Forward layers │ (×N layers) │ repeated 32-96+ times └──────────┬──────────┘ → Section 4, 6: Algebra & Linear Algebra ↓ ┌─────────────────────┐ │ Probability │ Softmax converts final output to │ Distribution │ probabilities over entire vocabulary └──────────┬──────────┘ → Section 2 & 6: Probability & Softmax ↓ ┌─────────────────────┐ │ Next Token │ Sampling picks one token │ (Sampling) │ (using Temperature, Top-K,

Hash the password, hand out a token, and make absolutely sure no one can read someone else's expenses. So Phase 2 gave my app a mouth. It could finally talk — create, read, update, and delete expenses over real HTTP endpoints, all clicking together through /docs . But there was a giant, deliberately-ignored problem sitting in the middle of it: the door had no lock. Anyone who could reach the server could read, edit, or delete anything. And every expense I created was quietly stamped with the same hardcoded owner — a "dev user" whose id I'd nailed into the code with a # TEMP note and a promise to fix it "in Phase 3." Well. It's Phase 3. Time to pay that debt. This is where the app grows a bouncer. The buzzword is auth , which actually hides two jobs that sound the same and aren't: authentication ["who are you?"] and authorization ["okay, but are you allowed to touch this ?"]. I went in thinking auth was "add a login form" and came out having learned about one-way hashing, signed tokens, a security bug with the excellent name IDOR , and why the same password can produce two different hashes. Let me dump what I learned [and the parts that tripped me up, because — as usual — there were several]. Auth is the one phase where you have to stop thinking like a builder and start thinking like the person trying to rob you. So every step below is really "here's a way an attacker wins, and here's the gap I closed to stop them." The structure. Let's call it PHASE 3 — The Lock: Give the User table somewhere to store a password [a hashed one, never the real thing] Password hashing helpers — turn a password into something safe to store POST /auth/register — sign up with a hashed password Understand what a JWT actually is [it's just a signed string, and it's readable] POST /auth/login — check the password, hand back a token get_current_user — the gatekeeper that turns a token back into a user Lock every expense endpoint and scope it to the logged-in owner Retire the hardcoded DEV_USE

Streamlit makes it remarkably fast to transform raw Python scripts into interactive, web-based data applications without needing any frontend knowledge in HTML, CSS, or JavaScript. In this tutorial, we will build a full-featured **Sales Analytics Dashboard** complete with real-time sidebar filtering, custom KPI metric cards, dynamic line/bar charts, and expandable data preview tables. --- ## Prerequisites To follow along, make sure you have Python 3.9+ installed along with the required libraries: bash pip install streamlit pandas numpy --- ## Step 1: Setting Up the Page & Mock Data with Caching First, we import the necessary libraries, set up the layout, and create a function to generate mock sales records. We use Streamlit’s `@st.cache_data` decorator so the data is only generated once per session, keeping the app snappy during user interactions. python import streamlit as st import pandas as pd import numpy as np Set layout configuration st.set_page_config(page_title="Sales Dashboard", layout="wide") Cache data loading for performance optimization @st .cache_data def load_data(): dates = pd.date_range("2025-01-01", periods=180) regions = ["North", "South", "East", "West"] df = pd.DataFrame({ "date": np.random.choice(dates, 500), "region": np.random.choice(regions, 500), "product": np.random.choice(["A", "B", "C"], 500), "sales": np.random.randint(100, 5000, 500), "units": np.random.randint(1, 50, 500), }) return df.sort_values("date") df = load_data() --- ## Step 2: Adding Interactive Sidebar Filters Next, we add controls inside the sidebar to let users filter the dataset by region, product type, and date range. A boolean mask applies those selections dynamically. python --- Sidebar filters --- st.sidebar.header("Filters") region_filter = st.sidebar.multiselect("Region", df["region"].unique(), default=df["region"].unique()) product_filter = st.sidebar.multiselect("Product", df["product"].unique(), default=df["product"].unique()) date_range = st.sidebar.date_input(

A few weeks ago I wrote about Siagnos , a personal taste engine for fanfiction that learns from reading behavior instead of matching tags. I was three stages in: scraper done, schema designed, embeddings working as a proof of concept. Then I got a two-week internship window to build something deployable, and I made a call. Instead of pushing Siagnos forward stage by stage, I built Opsis : a scoped-down, content-based recommender that answers one specific question. Given a fic, what else in a real, collected corpus is closest to it in content? Opsis doesn't do taste modeling. It doesn't touch my reading behavior at all. It's the layer underneath that, and it's live right now. Why not just keep building Siagnos directly Two weeks isn't enough time to get a reading tracker, a feature pipeline, and a trained preference model all working end to end. It is enough time to take the scraper and schema I already had and turn them into something real: a working recommender, deployed, with a UI, that someone else can actually use today. So I scoped down on purpose. No personal taste model yet. No behavior tracking yet. Just: can I take one fic and find genuinely similar ones, from AO3 metadata alone, using content instead of tags? What Opsis actually does Scrapes AO3 metadata under conditions the OTW Communications Committee confirmed were acceptable before I collected anything: one persistent session, randomized delays, capped retries Cleans and validates the raw data, log-and-skip instead of all-or-nothing, so one malformed row doesn't take down a 7,000-fic load Normalizes everything into PostgreSQL: fics, six lookup tables, six join tables, idempotent upserts so re-running the loader is always safe Embeds every fic's summary with sentence-transformers/all-MiniLM-L6-v2 Ranks candidates with a blended score: 0.70 embedding cosine similarity, 0.15 fandom overlap, 0.10 relationship overlap, 0.05 popularity If you submit a fic that isn't in the database yet, Opsis scrapes it, cle

LSTMs are still the go-to architecture for a lot of time series work, but they're annoying to trust. You get a number out the other end and no real sense of why the model landed there. This tutorial walks through training an LSTM on daily temperature data, then pulling it apart with three explainability methods: permutation importance, SHAP, and Integrated Gradients. Who this is for: people who already know some Keras and want to add interpretability to a forecasting model, not a from-scratch intro to neural nets. 1. Getting the data into shape LSTMs want a 3D tensor — (samples, timesteps, features) — so before anything else we need to turn a flat column of temperatures into overlapping 7-day windows, each one paired with the value on day 8. import numpy as np import pandas as pd from sklearn.preprocessing import MinMaxScaler # 1. Load data df = pd . read_csv ( " weather_data.csv " ) data = df [ ' Temperature ' ]. values . reshape ( - 1 , 1 ) # 2. Scale the data for stable neural network training scaler = MinMaxScaler ( feature_range = ( 0 , 1 )) scaled_data = scaler . fit_transform ( data ) # 3. Create sequences: 7 days of lag to predict the 8th day X , y = [], [] for i in range ( 7 , len ( scaled_data )): X . append ( scaled_data [ i - 7 : i ]) y . append ( scaled_data [ i ]) X , y = np . array ( X ), np . array ( y ) print ( f " Input shape: { X . shape } " ) # Output: (Samples, 7, 1) Scaling matters more than it sounds like it should — LSTMs trained on unscaled temperature values are prone to exploding gradients, and training just falls apart. The windowing step is really the whole trick here: every prediction only ever sees the past seven days, nothing more. 2. Building the model Two stacked LSTM layers, dropout after each one, early stopping so we don't have to babysit the epoch count. from tensorflow.keras.models import Sequential from tensorflow.keras.layers import LSTM , Dense , Dropout , Input from tensorflow.keras.callbacks import EarlyStopping # 1. Build

I’ve always disliked writing documentation. Not because it’s hard, but because it’s repetitive. You write a function, you describe what it does, you give an example, and then you realize you need the same thing in another language because half your users don’t speak English. So I decided to automate it. The result is an API that takes source code as input and returns a clean Markdown README, API reference, or inline comments — in any of 13 languages. No templates, no manual translation. curl -X POST "https://ai-code-documentation-generator.p.rapidapi.com/demo" \ -H "x-rapidapi-host: ai-code-documentation-generator.p.rapidapi.com" \ -H "x-rapidapi-key: YOUR_KEY" \ -H "Content-Type: application/json" \ -d '{"code":"def add(a,b): return a+b","code_language":"python","doc_language":"en"}' { "success" : true , "documentation" : "# Add Utility \n\n ## Overview \n A simple function to add two numbers..." , "quality_score" : 9 , "target_language" : "en" } It auto-detects the programming language (Python, JavaScript, Go, Rust…) and spits out a polished doc. The English output is solid, but seeing it generate accurate Japanese or German READMEs from the same code still feels like magic. The Tech Behind It Backend: Python + FastAPI, hosted on Northflank. AI Model: DeepSeek (via API). The model actually understands code structure, so generated docs aren’t just generic wrappers. Language Detection: Pygments for syntax highlighting + language guessing. Caching: 24-hour cache to avoid redundant calls and save cost. Security: Sensitive strings (API keys, passwords) are automatically redacted from the output. The whole thing is open source: https://github.com/zhaochangbo888/docgen-api Why I Didn’t Just Use ChatGPT You could absolutely paste your code into ChatGPT and ask for docs. But integrating an LLM directly into a CI/CD pipeline, or a VS Code extension, or a platform that needs programmatic access gets messy with rate limits, authentication, and output consistency. This API giv

How we tackled 精准翻译与语言识别 (precise translation and language identification) for AI-powered book translation. The Problem: Garbage In, Garbage Out When we first launched LectuLibre, our AI book translation service, we thought the hardest part would be fine-tuning LLM prompts for literary quality. But we quickly discovered a more fundamental hurdle: if the source language of an uploaded book is misidentified, no amount of prompt engineering can salvage the translation. Users upload EPUBs and PDFs from all over the world. Some contain metadata specifying the language, but many don't. Others are multilingual books, or have prefaces in a different language. Our initial language detection using Python's langdetect library was correct only about 85% of the time on real-world uploads. That 15% error rate meant entirely garbled translations, frustrated users, and wasted LLM API credits. We needed something far more robust—what we internally call 精准翻译与语言识别 (precise translation and language identification). Here’s how we built it. The Language Detection Pipeline: From 85% to 98% Accuracy Our first instinct was to try heavier models like fastText's pre-trained language identification model, which is known for high accuracy. But when we tested it on book excerpts, we hit a new problem: short paragraphs or dialogues in one language embedded in a book of another language (e.g., French phrases in an English novel) would throw off chunk-level detection. We realized that we needed a two-tier approach: book-level language detection with confidence scoring, and per-chunk verification before translation. Combining Multiple Detectors with Voting We created a LanguageDetector class that runs several detectors and picks the majority vote, with a fallback to user-specified language when available. The detectors we use are: fastText with the official lid.176.bin model (loaded once, not per request) langdetect , which is lightweight and good for long texts cld3 (Compact Language Detector 3) fr

1. Introduction Hello from Japan! 🇯🇵 I am an active truck driver in Japan self-studying Python and Web technologies, leveraging my logistics domain knowledge to transition into a Web Engineer. (English is my second language, but I'm excited to share my progress with developers worldwide!) I started my learning journey on May 12, 2026. As of June 4, I have logged 24 days and 86 cumulative hours of study. This article documents my progress, learning roadmap, and project milestones chronologically. Check out my GitHub here👈️ (Note: Most repository documentation and commit messages are currently in Japanese.) 2. The 86-Hour Learning Roadmap A quick breakdown of my 86 hours: 80 hours were dedicated to hands-on development and implementation, and 6 hours were spent on environment configuration, documentation, and workflow optimization. Stages 1–3: Automation & Scraping Projects ① Fundamentals & Web Scraping (25 hours) Learned core Python syntax and built automated scripts to collect targeted web data. ② Puoppo Auto-Analysis System (15 hours) Implemented automated poll data retrieval and AI-powered text analysis on Lubuntu. ③ Bakery Sales Aggregation System (12 hours) Integrated web scraping with automated Excel processing to streamline sales data aggregation. Stage 4: Practical Ruby on Rails ④ rails_practice (23 hours) Explored MVC architecture in Ruby on Rails. Practiced configuration management and Git rebasing to build a solid foundation in web framework operations. Stage 5: Real-World System Integration ⑤ hiroshima-logistics-hub (5 hours) Built a web system to aggregate real-time weather and traffic conditions in the Hiroshima area. Technical Fix : To bypass build crashes on Render (Free Tier / 512MB RAM), I precompiled assets locally and configured SQLite3 database storage in writable directories before deployment. 3. Key Takeaways for Resource-Constrained Development Through these projects, I established three operational principles for developing efficiently within

If your app sends email — transactional or marketing — three DNS records decide whether it lands in the inbox or the spam folder: SPF , DKIM , and DMARC . Here's how to look them up and sanity-check them in Python, no third-party API required. Install the one dependency: pip install dnspython SPF: who is allowed to send SPF lives in a TXT record on the domain itself and starts with v=spf1 . import dns.resolver def get_spf ( domain ): for rec in dns . resolver . resolve ( domain , " TXT " ): txt = b "" . join ( rec . strings ). decode () if txt . startswith ( " v=spf1 " ): return txt return None print ( get_spf ( " github.com " )) # v=spf1 ip4:... include:_spf.google.com ~all A quick gotcha worth checking: SPF allows at most 10 DNS-querying mechanisms ( include , a , mx , ptr , exists , redirect ). Go over and receivers return permerror , which quietly breaks authentication: def spf_lookup_count ( spf ): return sum ( spf . count ( m ) for m in ( " include: " , " a: " , " mx: " , " ptr " , " exists: " , " redirect= " )) spf = get_spf ( " example.com " ) if spf and spf_lookup_count ( spf ) > 10 : print ( " ⚠️ SPF exceeds the 10-lookup limit " ) DMARC: the policy that ties it together DMARC is a TXT record on the _dmarc. subdomain and starts with v=DMARC1 . def get_dmarc ( domain ): try : for rec in dns . resolver . resolve ( f " _dmarc. { domain } " , " TXT " ): txt = b "" . join ( rec . strings ). decode () if txt . startswith ( " v=DMARC1 " ): return dict ( kv . strip (). split ( " = " , 1 ) for kv in txt . split ( " ; " ) if " = " in kv ) except dns . resolver . NXDOMAIN : return None print ( get_dmarc ( " github.com " )) # {'v': 'DMARC1', 'p': 'reject', 'rua': 'mailto:...'} The key field is p : none (monitor only), quarantine (spam folder), or reject (bounce). If a domain sends real mail but has p=none , it's not protected against spoofing yet. DKIM: the signature key DKIM is trickier because you need the selector — a label chosen by the sender that lives at SELECT

When you scrape at any real volume, the bottleneck is rarely your code — it's the target site's rate limiting and IP bans. Rotating residential proxies solve this by routing each request through a different real-user IP. Here's how to wire them into requests and Scrapy in Python, including the sticky-session trick most tutorials skip. The proxy URL format A residential proxy is just an authenticated HTTP/SOCKS endpoint. With a pool gateway, you target a country and control session behavior through the username , not separate endpoints: http://USERNAME_country-us_session-a1b2c3_lifetime-30:PASSWORD@proxy.gproxy.net:1000 country-us — exit country (ISO code) session-a1b2c3 — a sticky-session id; reuse it to keep the same IP , change it to rotate lifetime-30 — how many minutes that session's IP stays fixed Basic request through a rotating proxy import requests USER = " USERNAME " PWD = " PASSWORD " def proxy ( country = " us " , session = None , lifetime = 30 ): tag = f " _country- { country } " if session : tag += f " _session- { session } _lifetime- { lifetime } " url = f " http:// { USER }{ tag } : { PWD } @proxy.gproxy.net:1000 " return { " http " : url , " https " : url } # New IP on every call (no session id): r = requests . get ( " https://api.ipify.org?format=json " , proxies = proxy (), timeout = 30 ) print ( r . json ()[ " ip " ]) Run it in a loop and you'll see a different IP each time — the gateway rotates automatically when no session id is present. Sticky sessions: keep one IP across requests Some flows (login, multi-step checkouts, paginated results behind a cookie) break if your IP changes mid-session. Pin the IP by passing a stable session id: import uuid sess = uuid . uuid4 (). hex [: 8 ] # one id for the whole flow p = proxy ( country = " de " , session = sess , lifetime = 30 ) s = requests . Session () s . proxies . update ( p ) s . get ( " https://example.com/login " ) s . post ( " https://example.com/login " , data = {...}) # same exit IP When you