Quick answer
What does 502 Bad Gateway mean with a proxy?
502 Bad Gateway means a gateway or proxy received an invalid response, or none, from the next server in the chain. Through CrawlRelay residential proxies, a 502 usually means the exit connection or the target failed mid-request. Retry it once on a new connection. If 502s continue for one site only, the site is the problem; if they hit every site, check your setup.
502 Bad Gateway at a glance
| Error | 502 Bad Gateway |
|---|---|
| Type | HTTP status code (5xx, server error) |
| Sent by | The proxy gateway or a reverse proxy in front of the target |
| Defined in | RFC 9110, section 15.6.3 |
| Retry it? | Yes, on a new connection or session |
| First check | Retry once on a new IP; if it persists, test the target without the proxy |
What does 502 Bad Gateway mean?
RFC 9110 describes 502 as a gateway's way of saying that the server behind it answered badly. In a scraping chain there are at least two gateways: the proxy and, very often, a CDN or load balancer in front of the website. Either one can produce the 502, and the response body usually tells you which.
Residential networks add one more link. The exit is a real household connection, and a device can leave the network in the middle of a request. The gateway then has nothing valid to send back.
What does a 502 look like in Python, curl, Node.js and browsers?
The same failure prints differently in every tool. These are the messages to search your logs for:
- Python requests:
status_code == 502, orProxyError('Unable to connect to proxy', OSError('Tunnel connection failed: 502 Bad Gateway'))for HTTPS targets - curl:
curl: (56) CONNECT tunnel failed, response 502 - Browsers:
net::ERR_TUNNEL_CONNECTION_FAILEDon HTTPS, or a plain error page on HTTP - Scrapy: retried by default, since 502 is in
RETRY_HTTP_CODES
What causes a 502?
Most likely first:
- The exit left the pool during the request, which happens on any residential network.
- The target closed the connection, crashed or returned something malformed.
- The target's CDN failed to reach its own origin server.
- DNS failure for the target at the exit side, for example a typo in the hostname.
- Very narrow targeting that leaves few exits, so failures repeat on the same small set.
How do I diagnose a 502 step by step?
- Retry the request on a new connection. A single 502 that disappears was a dropped exit.
- Read the response body and headers: a CDN error page names the CDN; a short proxy message names the gateway.
- Test the target from your own connection without the proxy.
- Widen targeting from city to country and compare the failure rate.
# Use the URL that fails; with https://api.ipify.org instead, you test the gateway alone.
# "HTTP/1.1 200 Connection established" first, then the site's own status: the site answered.
curl -v -x http://USERNAME:[email protected]:7777 https://TARGET_URL
# Same test through the SOCKS5 port, with DNS resolved at the exit
curl -v --socks5-hostname gw.crawlrelay.com:7778 --proxy-user USERNAME:PASSWORD https://TARGET_URLHow do I fix a 502?
- Retry with a new session ID or without one, so the request leaves through another exit.
- Set sensible timeouts so a dead exit fails fast instead of hanging.
- Log 502s per host and pause a host whose 502 rate jumps: it is probably down.
import random
import time
import requests
PROXY = "http://USERNAME:[email protected]:7777"
PROXIES = {"http": PROXY, "https": PROXY}
RETRY_STATUSES = {429, 502, 503, 504}
def fetch(url, attempts=5, timeout=(10, 30)):
for attempt in range(attempts):
try:
r = requests.get(url, proxies=PROXIES, timeout=timeout)
except (requests.exceptions.ConnectionError, requests.exceptions.Timeout):
r = None # dropped exit, reset or timeout: a new connection gets a new exit
if r is not None and r.status_code not in RETRY_STATUSES:
return r
retry_after = r.headers.get("Retry-After", "") if r is not None else ""
if retry_after.isdigit():
wait = int(retry_after)
else:
wait = min(60, 2 ** attempt) + random.uniform(0, 1)
time.sleep(wait)
raise RuntimeError("giving up on %s after %d attempts" % (url, attempts))Should I retry after a 502?
Yes, retry a 502, but on a fresh exit and with a limit. Two or three attempts with a short backoff are enough. A 502 that survives several exits points to the target, and more retries will not help. Track the rate per host so that one failing site does not fill your retry queue.
How does 502 Bad Gateway apply to CrawlRelay?
With CrawlRelay, drop the session ID or generate a new one to leave, usually, through a different residential IP. A sticky session ties you to one exit, so a 502 inside a sticky session should end that session in your code. The sessions guide explains how to rotate a session cleanly.
Gateway: gw.crawlrelay.com, port 7777 (HTTP, HTTPS through CONNECT) and port 7778 (SOCKS5). Username format: USERNAME-country-{cc}-state-{state}-city-{city}-zip-{zip}-asn-{asn}-session-{id}-ttl-{minutes}. Related: 504 Gateway Timeout · Connection reset · 503 Service Unavailable.
502 Bad Gateway: questions developers ask
Is a 502 the proxy's fault or the website's?
Either. Read the response body: CDN error pages name the CDN, which points to the website. A 502 on every site you try points to the proxy or your setup, while a 502 on one site points to that site. Test the same URL without the proxy to be sure.
Why are 502 errors more common with residential proxies?
Residential exits are real home connections, and devices go offline, change networks or lose signal. A request in flight when that happens ends with a 502 or a reset. A good scraper treats these as normal and retries on a new exit, with a small cap.
Should I switch to sticky sessions to reduce 502s?
No. A sticky session keeps you on one exit, so if that exit fails, every request in the session fails with it. Use rotating connections for independent pages and sticky sessions only for multi-step flows. End a sticky session after a 502 and start a new one.
Other proxy errors in this reference
HTTP status codes
- 401 Unauthorized vs 407
- 403 Forbidden
- 407 Proxy Authentication Required
- 429 Too Many Requests
- 502 Bad Gateway
- 503 Service Unavailable
- 504 Gateway Timeout
Browser and client errors
Connection and TLS errors
Residential proxies with docs that cover the failure cases
CrawlRelay residential proxies from $0.50/GB, traffic that never expires.