Proxy error reference · HTTP status codes

429 Too Many Requests when scraping: what it means and how to fix it

Q: How long should I wait after a 429?

Wait for the number of seconds in the Retry-After header, or until the date it gives. Without the header, start at one or two seconds and double the delay after each new 429, with a random extra and a cap of about a minute. Resume at a lower rate, not at the rate that caused the 429.

HTTP status code (4xx, client error): the exact message in each tool, the likely causes in order, and the fix that works.

Last updated: October 13, 2026 · 5 min read

Quick answer

What does 429 Too Many Requests mean with a proxy?

429 Too Many Requests means the website is rate-limiting you: you sent more requests than it accepts in a time window. The fix is to send fewer, not to hide more. With CrawlRelay residential proxies, honour the Retry-After header, cap concurrency per host, and back off exponentially with random jitter. Rotating IPs to exceed a site's published limit breaks most terms of use.

429 Too Many Requests at a glance

Error	429 Too Many Requests
Type	HTTP status code (4xx, client error)
Sent by	The target website or its API gateway
Defined in	RFC 6585, section 4 (Retry-After: RFC 9110, section 10.2.3)
Retry it?	Yes, after the Retry-After delay or an exponential backoff
First check	Read the Retry-After header and your requests per second per host

What does 429 Too Many Requests mean?

RFC 6585 introduced 429 for rate limiting. The server may add a Retry-After header, either a number of seconds or an HTTP date, to say when to come back. APIs often add their own headers too, such as X-RateLimit-Remaining.

A 429 is the most polite answer a site can give a scraper. It says the content is available, just not at your current pace. Sites that see their 429s ignored usually escalate to 403s, CAPTCHAs or permanent blocks.

What does a 429 look like in Python, curl, Node.js and browsers?

The same failure prints differently in every tool. These are the messages to search your logs for:

Python requests / httpx: status_code == 429, often with Retry-After: 30
Scrapy: handled by RetryMiddleware, since 429 is in the default RETRY_HTTP_CODES
curl: HTTP/2 429 followed by retry-after: in the response headers
APIs: a JSON body such as {"error": "rate_limited"}

What causes a 429?

Most likely first:

Too much concurrency per host. Fifty parallel workers on one domain look like an attack to most sites.
No delay between requests, or a fixed delay that lines up many workers at the same instant.
Retries without backoff. A retry storm after the first 429 multiplies the load.
Shared limits. Some sites count requests per account, per API key or per session cookie, not per IP.
Heavy endpoints. Search and filter URLs often have stricter limits than static pages.

How do I diagnose a 429 step by step?

Log the Retry-After value and any rate-limit headers on the first 429.
Measure your real request rate per host, not your total rate.
Check whether the limit follows the IP, the cookie or the API key: change one at a time.
Read the site's API or robots.txt documentation; a Crawl-delay line states the pace it asks for.

Test the failing URL with curl (HTTP port, verbose)

# Use the URL that fails; with https://api.ipify.org instead, you test the gateway alone.
# "HTTP/1.1 200 Connection established" first, then the site's own status: the site answered.
curl -v -x http://USERNAME:[email protected]:7777 https://TARGET_URL

# Same test through the SOCKS5 port, with DNS resolved at the exit
curl -v --socks5-hostname gw.crawlrelay.com:7778 --proxy-user USERNAME:PASSWORD https://TARGET_URL

How do I fix a 429?

Honour Retry-After. Wait at least the time the server asks for.
Cap concurrency per host, not only globally. Scrapy has CONCURRENT_REQUESTS_PER_DOMAIN and AUTOTHROTTLE_ENABLED.
Back off exponentially with jitter: 1 s, 2 s, 4 s and so on, plus a random fraction, capped at a maximum.
Fetch less. Cache responses, send If-Modified-Since or If-None-Match, and skip pages that did not change.
Use the official API when the site offers one; its limits are usually documented.

Python: retry with Retry-After, exponential backoff and jitter

import random
import time

import requests

PROXY = "http://USERNAME:[email protected]:7777"
PROXIES = {"http": PROXY, "https": PROXY}
RETRY_STATUSES = {429, 502, 503, 504}


def fetch(url, attempts=5, timeout=(10, 30)):
    for attempt in range(attempts):
        try:
            r = requests.get(url, proxies=PROXIES, timeout=timeout)
        except (requests.exceptions.ConnectionError, requests.exceptions.Timeout):
            r = None  # dropped exit, reset or timeout: a new connection gets a new exit
        if r is not None and r.status_code not in RETRY_STATUSES:
            return r
        retry_after = r.headers.get("Retry-After", "") if r is not None else ""
        if retry_after.isdigit():
            wait = int(retry_after)
        else:
            wait = min(60, 2 ** attempt) + random.uniform(0, 1)
        time.sleep(wait)
    raise RuntimeError("giving up on %s after %d attempts" % (url, attempts))

Should I retry after a 429?

Yes, a 429 is meant to be retried, later. Wait for the Retry-After delay if the server sends one; otherwise use exponential backoff with jitter and a cap. Count consecutive 429s per host and lower that host's concurrency when the count grows. The Python function below does the basic part.

How does 429 Too Many Requests apply to CrawlRelay?

A 429 that reaches you through CrawlRelay carries the target's own headers and body: the limit is the site's, and per-host pacing is your job. Use sticky sessions when a site counts requests per session, so that your pace stays measurable, and see the concurrency guide for numbers to start with.

Gateway: gw.crawlrelay.com, port 7777 (HTTP, HTTPS through CONNECT) and port 7778 (SOCKS5). Username format: USERNAME-country-{cc}-state-{state}-city-{city}-zip-{zip}-asn-{asn}-session-{id}-ttl-{minutes}. Related: 403 Forbidden · 503 Service Unavailable · Concurrency guide.

429 Too Many Requests: questions developers ask

Should I rotate IPs to avoid 429 errors?

No. A 429 is the site's request to slow down. Spreading the same request rate over more IPs to stay under a per-IP limit is evasion, and most terms of use forbid it. Rotation is useful for load spreading and for location, not for exceeding a published limit. Lower your rate and cache what you can.

How long should I wait after a 429?

Wait for the number of seconds in the Retry-After header, or until the date it gives. Without the header, start at one or two seconds and double the delay after each new 429, with a random extra and a cap of about a minute. Resume at a lower rate, not at the rate that caused the 429.

Why do I get 429 with only a few requests per minute?

The limit may be per account, cookie or API key rather than per IP, or the endpoint may be expensive, like a search page. Some sites also set low limits for clients they classify as bots. Check the response headers and the site's API documentation, and ask the site for access if you need more.

Other proxy errors in this reference

HTTP status codes

Browser and client errors

Connection and TLS errors

Residential proxies with docs that cover the failure cases

CrawlRelay residential proxies from $0.50/GB, traffic that never expires.

See pricing Quick start