Quick answer
What does 504 Gateway Timeout mean with a proxy?
504 Gateway Timeout means a gateway, such as the proxy or the site's CDN, waited too long for the next server and gave up. Through CrawlRelay residential proxies, the slow link is usually the target or the exit's home connection. Retry once on a new exit, keep pages light, and set your own client timeouts so a slow request fails on your terms.
504 Gateway Timeout at a glance
| Error | 504 Gateway Timeout |
|---|---|
| Type | HTTP status code (5xx, server error) |
| Sent by | The proxy gateway or a CDN in front of the target |
| Defined in | RFC 9110, section 15.6.5 |
| Retry it? | Yes, once or twice on a new exit |
| First check | Time the same request without the proxy |
What does 504 Gateway Timeout mean?
RFC 9110 defines 504 as a gateway that did not get a timely response from the upstream server it needed. It is the timeout twin of 502: 502 means a bad answer, 504 means no answer in time.
A residential proxy adds a hop that a direct request does not have: your traffic goes to the gateway, then to a home connection, then to the site. Each hop adds delay, and the slowest one decides the total.
What does a 504 look like in Python, curl, Node.js and browsers?
The same failure prints differently in every tool. These are the messages to search your logs for:
- Python requests:
status_code == 504, orTunnel connection failed: 504 Gateway Timeoutfor HTTPS targets - curl:
curl: (56) CONNECT tunnel failed, response 504 - Scrapy: retried by default (504 is in
RETRY_HTTP_CODES) - CDN-served sites: an error page from the CDN with status 504
What causes a 504?
Most likely first:
- A slow target, for example a search page that takes many seconds to build.
- A slow exit, such as a weak home connection or a congested mobile cell.
- Large responses that take longer than the gateway's patience to arrive.
- The site's own origin timing out behind its CDN, which then answers 504.
How do I diagnose a 504 step by step?
- Time the request without the proxy. If it is slow at home too, the target is slow.
- Retry on a new exit and compare the timing.
- Check the response size: a 5 MB page is a different problem from a 50 KB page.
- Read the body to see whether the CDN or the proxy wrote the error page.
# Use the URL that fails; with https://api.ipify.org instead, you test the gateway alone.
# "HTTP/1.1 200 Connection established" first, then the site's own status: the site answered.
curl -v -x http://USERNAME:[email protected]:7777 https://TARGET_URL
# Same test through the SOCKS5 port, with DNS resolved at the exit
curl -v --socks5-hostname gw.crawlrelay.com:7778 --proxy-user USERNAME:PASSWORD https://TARGET_URLHow do I fix a 504?
- Set explicit timeouts: about 10 seconds to connect and 30 to 60 seconds to read for normal pages.
- Retry on a new exit once or twice.
- Request less: skip images and media when you only need the HTML or the JSON API behind the page.
- Schedule heavy pages at off-peak hours for the target's country.
import httpx
proxy = "http://USERNAME:[email protected]:7777"
timeout = httpx.Timeout(connect=10.0, read=30.0, write=10.0, pool=10.0)
with httpx.Client(proxy=proxy, timeout=timeout) as client:
try:
r = client.get("https://api.ipify.org")
print(r.status_code, r.text)
except httpx.TimeoutException as exc:
print("timed out:", type(exc).__name__) # ConnectTimeout, ReadTimeout...Should I retry after a 504?
Retry a 504 once or twice on a fresh exit, with a short backoff. If the same URL times out across several exits, the target is slow or overloaded: lower your rate on that host or try later. Raising your client timeout helps only if the gateway's timeout is not the limit.
How does 504 Gateway Timeout apply to CrawlRelay?
Each new connection to CrawlRelay without a session ID usually gets a different residential exit, so a retry normally tests another home connection. For slow targets, keep payloads small and measure time to first byte separately from total download time.
Gateway: gw.crawlrelay.com, port 7777 (HTTP, HTTPS through CONNECT) and port 7778 (SOCKS5). Username format: USERNAME-country-{cc}-state-{state}-city-{city}-zip-{zip}-asn-{asn}-session-{id}-ttl-{minutes}. Related: 502 Bad Gateway · Read timed out · 503 Service Unavailable.
504 Gateway Timeout: questions developers ask
What is the difference between 502 and 504?
Both come from a gateway. A 502 means the server behind it sent an invalid response or dropped the connection. A 504 means it sent nothing in time. With residential proxies, 502 often points to a dropped exit, and 504 to a slow target or a slow home connection.
Can I raise the proxy's timeout?
You control your client's timeouts, not the gateway's internal ones. Set your connect and read timeouts explicitly so slow requests fail predictably, retry on a new exit, and make pages lighter. If a target needs very long responses, fetch its API or a lighter page instead.
Are mobile proxies slower than residential ones?
They can be, because a mobile exit shares a cell with other users and its speed changes with signal and load. Use mobile IPs for targets that need them and residential IPs for bulk crawling. Measure your own response times per exit type before you decide.
Other proxy errors in this reference
HTTP status codes
- 401 Unauthorized vs 407
- 403 Forbidden
- 407 Proxy Authentication Required
- 429 Too Many Requests
- 502 Bad Gateway
- 503 Service Unavailable
- 504 Gateway Timeout
Browser and client errors
Connection and TLS errors
Residential proxies with docs that cover the failure cases
CrawlRelay residential proxies from $0.50/GB, traffic that never expires.