Rotating Residential Proxies in Python: requests, Scrapy & Sticky Sessions

When you scrape at any real volume, the bottleneck is rarely your code — it's the target site's rate limiting and IP bans. Rotating residential proxies solve this by routing each request through a different real-user IP. Here's how to wire them into requests and Scrapy in Python, including the sticky-session trick most tutorials skip. The proxy URL format A residential proxy is just an authenticated HTTP/SOCKS endpoint. With a pool gateway, you target a country and control session behavior through the username , not separate endpoints: http://USERNAME_country-us_session-a1b2c3_lifetime-30:PASSWORD@proxy.gproxy.net:1000 country-us — exit country (ISO code) session-a1b2c3 — a sticky-session id; reuse it to keep the same IP , change it to rotate lifetime-30 — how many minutes that session's IP stays fixed Basic request through a rotating proxy import requests USER = " USERNAME " PWD = " PASSWORD " def proxy ( country = " us " , session = None , lifetime = 30 ): tag = f " _country- { country } " if session : tag += f " _session- { session } _lifetime- { lifetime } " url = f " http:// { USER }{ tag } : { PWD } @proxy.gproxy.net:1000 " return { " http " : url , " https " : url } # New IP on every call (no session id): r = requests . get ( " https://api.ipify.org?format=json " , proxies = proxy (), timeout = 30 ) print ( r . json ()[ " ip " ]) Run it in a loop and you'll see a different IP each time — the gateway rotates automatically when no session id is present. Sticky sessions: keep one IP across requests Some flows (login, multi-step checkouts, paginated results behind a cookie) break if your IP changes mid-session. Pin the IP by passing a stable session id: import uuid sess = uuid . uuid4 (). hex [: 8 ] # one id for the whole flow p = proxy ( country = " de " , session = sess , lifetime = 30 ) s = requests . Session () s . proxies . update ( p ) s . get ( " https://example.com/login " ) s . post ( " https://example.com/login " , data = {...}) # same exit IP When you