Quick answer
How many concurrent requests can I run through residential proxies?
As many as each target tolerates: CrawlRelay bills residential traffic per GB ($0.50/GB from 50 GB), not per thread or connection. Size concurrency with Little's law, requests in flight = pages per second × seconds per page, then cap it per host, starting at 2 to 4. Lower it when 429 or 503 responses appear. Paid in crypto, with no KYC.
Concurrency at a glance
| Billing | Per GB of traffic; threads and connections are not billed |
|---|---|
| Sizing rule | In flight = throughput × latency (Little's law) |
| Starting point per host | 2–4 parallel requests; 1 with a delay for small sites |
| Signals to slow down | 429, 503, rising response times, challenge pages |
| Bandwidth math | Pages per second × bytes per page × 3,600 = bytes per hour |
| Residential median response | 0.93 s on the proxy network, 3.21 s on real targets (Proxyway 2026) |
How do I size concurrency from the throughput I need?
Use Little's law: the number of requests in flight equals the rate you want multiplied by how long each request takes. Proxyway's Proxy Market Research 2026 measured a median of 3.21 seconds per request on real targets through residential proxies, so plan with 2 to 4 seconds for real sites, not with the sub-second figure of the network alone.
| Goal | Time per request | Requests in flight |
|---|---|---|
| 5 pages per second | 2 s | 10 |
| 20 pages per second | 2 s | 40 |
| 20 pages per second | 4 s | 80 |
| 100 pages per second | 3 s | 300 |
Then spread those requests over hosts. Forty requests in flight across 200 sites is gentle; forty on one small site is a load test it never agreed to.
How much bandwidth does that throughput use?
Multiply pages per second by bytes per page. At 20 HTML pages per second and 50 KB per page, the crawl moves 1 MB per second, which is 3.6 GB per hour and about 86 GB per day. At 2 MB per rendered page, the same rate would move 40 MB per second.
Multiply the result by your plan's price per GB, $0.50/GB from 50 GB at CrawlRelay, to get the hourly cost. The bandwidth calculator guide gives typical page weights by page type.
What per-host limits should I start with?
| Target | Parallel requests per host | Delay between requests | Notes |
|---|---|---|---|
| Small site (blog, local shop) | 1 | 2–5 s | Honour Crawl-delay in robots.txt if present |
| Medium site | 2–4 | 0.5–2 s | Watch response times; back off when they double |
| Large platform with CDN | 4–16 | AutoThrottle or adaptive | Raise only while 429 and 5xx rates stay flat |
| Public API | The documented limit | As documented | Use the API key and limits the provider gives you |
How do I adapt concurrency automatically?
Use additive increase and multiplicative decrease, the rule TCP uses for congestion: add one slot after a run of clean responses, halve the limit on a 429 or 503. The limiter below keeps one limit per host, pauses every request to that host for the Retry-After delay, retries once, and never exceeds a ceiling you set. Network errors come back as values, so one timeout does not stop the batch.
import asyncio
import random
from urllib.parse import urlsplit
import httpx
PROXY = "http://USERNAME-country-us:[email protected]:7777"
class AdaptiveLimit:
"""One per host: +1 slot after 20 clean responses, halve on 429/503, pause for Retry-After."""
def __init__(self, start=2, low=1, high=16):
self.limit, self.low, self.high = start, low, high
self.in_flight, self.clean = 0, 0
self.pause_until = 0.0
self.cond = asyncio.Condition()
async def __aenter__(self):
loop = asyncio.get_running_loop()
while True:
async with self.cond:
await self.cond.wait_for(lambda: self.in_flight < self.limit)
delay = self.pause_until - loop.time()
if delay <= 0:
self.in_flight += 1
return self
await asyncio.sleep(delay) # the host asked for a pause: wait outside the lock
async def __aexit__(self, *exc):
async with self.cond:
self.in_flight -= 1
self.cond.notify_all()
def record(self, status, retry_after=None):
if status in (429, 503):
self.limit, self.clean = max(self.low, self.limit // 2), 0
wait = int(retry_after) if retry_after and retry_after.isdigit() else 5 + random.random()
now = asyncio.get_running_loop().time()
self.pause_until = max(self.pause_until, now + wait)
elif status < 400:
self.clean += 1
if self.clean >= 20 and self.limit < self.high:
self.limit, self.clean = self.limit + 1, 0
gates = {}
def gate_for(url):
host = urlsplit(url).hostname
if host not in gates:
gates[host] = AdaptiveLimit()
return gates[host]
async def fetch(client, url, attempts=2):
gate = gate_for(url)
status = None
for _ in range(attempts):
try:
async with gate:
r = await client.get(url)
except httpx.HTTPError as exc:
return "error: %s" % type(exc).__name__
status = r.status_code
gate.record(status, r.headers.get("Retry-After"))
if status not in (429, 503):
break # the retry, if any, waits for the host's pause in __aenter__
return status
async def main(urls):
timeout = httpx.Timeout(30.0, connect=10.0)
async with httpx.AsyncClient(proxy=PROXY, timeout=timeout) as client:
print(await asyncio.gather(*(fetch(client, u) for u in urls)))
print({host: gate.limit for host, gate in gates.items()})
asyncio.run(main(["https://example.com/"] * 10))What slows a proxied crawl down besides the target?
- TLS handshakes. A new connection per request, which you need for a new IP each time, adds a handshake with the site every time. Keep a sticky session with keep-alive for pages of one site when IP diversity does not matter.
- Slow exits. Residential and mobile exits are real home and cellular links. Use timeouts and retry on a new connection instead of waiting.
- Narrow targeting. A small city or a single ASN has fewer exits than a country. Target narrowly only where the data changes.
- Rendering. A headless browser costs far more CPU, memory and traffic per page than an HTTP client. Render only the pages that need JavaScript.
- DNS. With
socks5h://or the HTTP port, DNS resolves at the exit; you do not need a local resolver cache for target hosts.
Concurrency questions
Does CrawlRelay limit the number of threads or connections?
Plans are billed per GB of traffic, not per thread, connection or request, and the published plans set no thread count. The practical limits are each target's tolerance and your own machine. If you plan a very large crawl, write to [email protected] first so we can check coverage for your targeting.
Why did my crawl get slower when I added more threads?
Usually because the target started to throttle you: response times rise, 429 or 503 responses appear, and retries add more load. The other common cause is your own machine running out of CPU or sockets, especially with headless browsers. Measure per-host latency and error rates before adding threads.
Is one connection per request slower than keep-alive?
Yes, by one TCP and TLS handshake per request, typically a few hundred milliseconds through a residential exit. That is the price of a new IP on every request. For many pages of one site where the IP does not matter, a sticky session with keep-alive is faster and uses less traffic.
How do I respect robots.txt crawl-delay in a concurrent crawler?
Parse robots.txt once per host, cache it for up to 24 hours as RFC 9309 suggests, and turn a Crawl-delay value into a per-host minimum gap between requests with a concurrency of 1. Python's urllib.robotparser exposes it through crawl_delay().
Scale on per-GB pricing
$0.50/GB from 50 GB to 1 TB, traffic that never expires.