# Rotating residential proxies in Python: requests, httpx and aiohttp

> Rotating residential proxies in Python: requests with retries, httpx sync and async, aiohttp, sticky sessions and byte counts. Syntax-checked code.

- URL: https://crawlrelay.com/docs/python/
- Last updated: 2026-10-13
- Publisher: CrawlRelay (https://crawlrelay.com)

Production patterns, not one-liners: retries that respect Retry-After, a new IP per request or one IP per flow, per-host concurrency in asyncio, and the bytes each page really costs.

**Quick answer — How do I use rotating residential proxies in Python?**

Pass `http://USERNAME:PASSWORD@gw.crawlrelay.com:7777` as the proxy for both `http` and `https` in requests, as `proxy=` in httpx 0.26+, or as `proxy=` per request in aiohttp. **CrawlRelay** residential proxies rotate on every new connection and stay sticky with `-session-<id>` in the username. They cost $1.50 for 1 GB and $0.50/GB from 50 GB. Paid in crypto, with no KYC.

## Python setup at a glance

| Attribute | Value |
| --- | --- |
| requests | `proxies={"http": url, "https": url}`; retries through urllib3 `Retry` |
| httpx | `httpx.Client(proxy=url)`; the old `proxies=` argument was removed in httpx 0.28 |
| aiohttp | `session.get(url, proxy="http://USER:PASS@host:port")`, credentials in the proxy URL |
| SOCKS5 | `pip install "requests[socks]"` or `httpx[socks]`, then `socks5h://` on port 7778 |
| New IP per request | Disable keep-alive: `Connection: close`, `max_keepalive_connections=0` or `force_close=True` |
| Same IP for a flow | `-session-<id>-ttl-<minutes>` in the username, up to 30 minutes |
| Proxy URL scheme | Always `http://` for the HTTP port, even for HTTPS websites |

## How do I use a proxy with requests, with retries that respect Retry-After?

Mount an `HTTPAdapter` with urllib3's `Retry` on a `Session`. It retries connection errors and the statuses you list, waits with exponential backoff, and honours the `Retry-After` header that sites send with 429 and 503. Set `trust_env = False` so a leftover `HTTPS_PROXY` variable cannot override your proxy.

*requests + urllib3 Retry*

```python
import os

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

USER = os.environ.get("CRAWLRELAY_USER", "USERNAME-country-us")
PASSWORD = os.environ.get("CRAWLRELAY_PASSWORD", "PASSWORD")
PROXY = "http://%s:%s@gw.crawlrelay.com:7777" % (USER, PASSWORD)

retry = Retry(
    total=4,
    connect=2,
    backoff_factor=1.0,  # waits of 0 s, 2 s, 4 s, 8 s (Retry-After wins on 429/503)
    status_forcelist=(429, 502, 503, 504),
    allowed_methods=frozenset({"GET", "HEAD"}),
    respect_retry_after_header=True,
)

session = requests.Session()
session.trust_env = False  # ignore HTTP(S)_PROXY from the environment
session.mount("http://", HTTPAdapter(max_retries=retry))
session.mount("https://", HTTPAdapter(max_retries=retry))
session.proxies = {"http": PROXY, "https": PROXY}
session.headers["User-Agent"] = "ExampleCrawler/1.0 (+https://example.com/crawler-info)"
session.headers["Connection"] = "close"  # new connection per request, so usually a new exit IP

r = session.get("https://api.ipify.org", timeout=(10, 30))
print(r.status_code, r.text)
```

## How do I build targeted proxy URLs safely?

Generate the username in one function and validate it. A typo in a country code or an unencoded character in the password produces a [407](https://crawlrelay.com/docs/errors/407-proxy-authentication-required/) that looks like a credentials problem. The helper below keeps the documented order (`USERNAME-country-{cc}-state-{state}-city-{city}-zip-{zip}-asn-{asn}-session-{id}-ttl-{minutes}`), refuses unknown keys and URL-encodes both credentials.

*Username builder and proxy URL*

```python
import re
from urllib.parse import quote

ORDER = ("country", "state", "city", "zip", "asn", "session", "ttl")
VALUE = re.compile(r"^[a-z0-9_]+$")


def build_username(base, **params):
    """build_username("USERNAME", country="us", city="new_york", session="job1", ttl=30)"""
    parts = [base]
    for key in ORDER:
        value = params.pop(key, None)
        if value is None:
            continue
        value = str(value).lower()
        if not VALUE.match(value):
            raise ValueError("bad value for %s: %r" % (key, value))
        if key == "ttl" and not 1 <= int(value) <= 30:
            raise ValueError("ttl must be 1 to 30 minutes")
        parts.append("%s-%s" % (key, value))
    if params:
        raise ValueError("unknown parameters: %s" % ", ".join(sorted(params)))
    return "-".join(parts)


def proxy_url(username, password, socks=False):
    scheme, port = ("socks5h", 7778) if socks else ("http", 7777)
    return "%s://%s:%s@gw.crawlrelay.com:%d" % (
        scheme, quote(username, safe=""), quote(password, safe=""), port)


print(proxy_url(build_username("USERNAME", country="us", state="ny", city="new_york", session="job1", ttl=20), "p@ss:word"))
```

## How do I get a new IP on every request with httpx?

Turn off keep-alive. httpx pools connections by default, and a reused connection keeps its exit IP. With `max_keepalive_connections=0`, each request opens a new connection to the gateway and usually leaves through a different residential IP. `num_bytes_downloaded` gives the bytes received on the wire, which is what traffic billing is closest to.

*httpx: rotating client*

```python
import httpx

PROXY = "http://USERNAME-country-gb:PASSWORD@gw.crawlrelay.com:7777"

# No keep-alive: every request opens a new connection, so it usually gets a new exit IP
with httpx.Client(
    proxy=PROXY,
    timeout=httpx.Timeout(30.0, connect=10.0),
    limits=httpx.Limits(max_keepalive_connections=0),
    follow_redirects=True,
) as client:
    for _ in range(3):
        r = client.get("https://api.ipify.org")
        print(r.text, r.num_bytes_downloaded, "bytes on the wire")
```

## How do I scrape concurrently with asyncio without hammering one site?

Cap concurrency **per host**, not only globally. A global limit of 50 lets all 50 requests hit the same site when your URL list is sorted by domain. A semaphore per hostname keeps each site at a polite level while the crawl as a whole stays fast. Start at 2 to 4 per host and read the [concurrency guide](https://crawlrelay.com/docs/concurrency/) before you raise it.

*httpx.AsyncClient with a semaphore per host*

```python
import asyncio
from collections import defaultdict
from urllib.parse import urlsplit

import httpx

PROXY = "http://USERNAME-country-us:PASSWORD@gw.crawlrelay.com:7777"
PER_HOST = 4  # parallel requests per target host
host_slots = defaultdict(lambda: asyncio.Semaphore(PER_HOST))


async def fetch(client, url):
    async with host_slots[urlsplit(url).hostname]:
        try:
            r = await client.get(url)
            return url, r.status_code, r.num_bytes_downloaded
        except httpx.HTTPError as exc:
            return url, type(exc).__name__, 0


async def main(urls):
    limits = httpx.Limits(max_connections=50, max_keepalive_connections=0)
    timeout = httpx.Timeout(30.0, connect=10.0)
    async with httpx.AsyncClient(proxy=PROXY, limits=limits, timeout=timeout) as client:
        for result in await asyncio.gather(*(fetch(client, u) for u in urls)):
            print(*result)


asyncio.run(main(["https://example.com/", "https://example.org/", "https://example.net/"]))
```

## How do I use a proxy with aiohttp?

Pass the gateway as `proxy=` on each request, with the credentials in the proxy URL. That works on every aiohttp 3.x release without warnings; `BasicAuth` and the `proxy_auth=` parameter are deprecated since aiohttp 3.14 and will be removed in 4.0. A `TCPConnector(force_close=True)` closes every connection after its response, so each request usually gets a new exit. Without it, aiohttp reuses connections and you keep the same IP for a while.

*aiohttp with credentials in the proxy URL*

```python
import asyncio

import aiohttp

# Credentials go in the proxy URL; percent-encode a password that contains @ or :
PROXY = "http://USERNAME-country-fr:PASSWORD@gw.crawlrelay.com:7777"


async def main():
    # force_close=True: one connection per request, so each request usually gets a new exit IP
    connector = aiohttp.TCPConnector(limit=20, force_close=True)
    timeout = aiohttp.ClientTimeout(total=60, sock_connect=10)
    async with aiohttp.ClientSession(connector=connector, timeout=timeout) as session:
        for _ in range(3):
            async with session.get("https://api.ipify.org", proxy=PROXY) as resp:
                print(resp.status, await resp.text())


asyncio.run(main())
```

## How do I keep one IP for a multi-step flow?

Generate one session ID per flow and put it in the username with a TTL longer than the flow. Keep-alive is fine here: you want the same exit. If the residential device behind the session goes offline, the IP can change before the TTL ends, so make the flow restartable from its last checkpoint.

*Sticky session with requests*

```python
import uuid

import requests

session_id = uuid.uuid4().hex[:10]  # one ID per flow, lowercase letters and digits
user = "USERNAME-country-us-state-ca-session-%s-ttl-20" % session_id
proxy = "http://%s:PASSWORD@gw.crawlrelay.com:7777" % user

with requests.Session() as s:
    s.proxies = {"http": proxy, "https": proxy}
    first = s.get("https://api.ipify.org", timeout=(10, 30)).text
    second = s.get("https://api.ipify.org", timeout=(10, 30)).text
    print(first, second, "same exit" if first == second else "exit changed")
```

## How much traffic does a Python scraper use per page?

Measure it on 100 real pages before you buy volume. Log `num_bytes_downloaded` in httpx, or the length of the compressed body in your own middleware, and multiply by your page count. HTML-only requests usually cost tens of kilobytes per page; the [bandwidth guide](https://crawlrelay.com/guides/bandwidth-calculator/) gives the numbers and the cost per 1,000 pages at CrawlRelay prices.

Three habits cut traffic without changing your results:

- Send `Accept-Encoding: gzip, br` (httpx and requests send gzip by default) so pages travel compressed.
- Fetch the JSON endpoint that a page loads, when the site documents or allows it, instead of the full HTML.
- Skip unchanged pages with `If-None-Match` or `If-Modified-Since`: a `304 Not Modified` costs a few hundred bytes.

## Python and proxies: questions

### requests, httpx or aiohttp: which should I use for scraping?

Use requests for simple scripts and small crawls. Use httpx when you want the same API in sync and async code, and HTTP/2 through `httpx[http2]`. Use aiohttp for very high concurrency in pure asyncio. For a crawl with scheduling, deduplication and pipelines, use [Scrapy](https://crawlrelay.com/docs/scrapy/) instead of writing those parts yourself.

### Why do I get SSL: WRONG_VERSION_NUMBER with my proxy in Python?

You probably wrote `https://` in front of the proxy address. The gateway's port 7777 speaks plain HTTP and opens a CONNECT tunnel for HTTPS sites, so the proxy URL must start with `http://` in both the `http` and the `https` keys. See the [error page](https://crawlrelay.com/docs/errors/ssl-wrong-version-number/).

### How do I use SOCKS5 proxies in Python?

Install the extra (`pip install "requests[socks]"` or `pip install "httpx[socks]"`), then use `socks5h://USERNAME:PASSWORD@gw.crawlrelay.com:7778`. The `h` sends hostnames to the proxy, so DNS resolves at the exit and you see the same regional answers as local visitors.

### Why does my rotating proxy return the same IP twice?

Your client reused a kept-alive connection, and rotation happens per new connection. Send `Connection: close` in requests, set `max_keepalive_connections=0` in httpx, or use `force_close=True` in aiohttp. Also check that the username has no `session-` part, which keeps one IP on purpose.

### How do I set timeouts for proxied requests?

Set them explicitly: requests has no timeout by default. A connect timeout of about 10 seconds and a read timeout of 30 seconds suit HTML pages; mobile exits and slow targets need 45 to 60 seconds. Retry a timeout once on a new connection rather than waiting longer. See [read timeouts](https://crawlrelay.com/docs/errors/read-timeout/).

### Can I use these proxies in Jupyter or with pandas?

Yes. Anything that uses requests or httpx underneath takes the same proxy settings. For `pandas.read_html` or `read_csv` on a URL, download the content with a proxied requests call first, then pass the text or bytes to pandas.

**Run your Python scraper through residential IPs** $1.50 for 1 GB, $0.50/GB from 50 GB. [See pricing](https://crawlrelay.com/pricing/) · [Scrapy guide](https://crawlrelay.com/docs/scrapy/)

---

CrawlRelay: CrawlRelay is a developer-first rotating residential and 4G/5G mobile proxy service for scrapers and AI data pipelines: non-expiring pay-as-you-go traffic from $0.50/GB, paid in crypto, with no ID upload. No KYC — no ID, no selfie, no documents. Pay in crypto.