# 429 Too Many Requests when scraping: what it means and how to fix it

> What HTTP 429 Too Many Requests means for a scraper, how to read Retry-After, and Python code for backoff with jitter. Slow down; do not evade.

- URL: https://crawlrelay.com/docs/errors/429-too-many-requests/
- Last updated: 2026-10-13
- Publisher: CrawlRelay (https://crawlrelay.com)

HTTP status code (4xx, client error): the exact message in each tool, the likely causes in order, and the fix that works.

**Quick answer — What does 429 Too Many Requests mean with a proxy?**

**429 Too Many Requests** means the website is rate-limiting you: you sent more requests than it accepts in a time window. The fix is to send fewer, not to hide more. With CrawlRelay residential proxies, honour the `Retry-After` header, cap concurrency per host, and back off exponentially with random jitter. Rotating IPs to exceed a site's published limit breaks most terms of use.

## 429 Too Many Requests at a glance

| Attribute | Value |
| --- | --- |
| Error | 429 Too Many Requests |
| Type | HTTP status code (4xx, client error) |
| Sent by | The target website or its API gateway |
| Defined in | RFC 6585, section 4 (Retry-After: RFC 9110, section 10.2.3) |
| Retry it? | Yes, after the Retry-After delay or an exponential backoff |
| First check | Read the Retry-After header and your requests per second per host |

## What does 429 Too Many Requests mean?

RFC 6585 introduced 429 for rate limiting. The server may add a `Retry-After` header, either a number of seconds or an HTTP date, to say when to come back. APIs often add their own headers too, such as `X-RateLimit-Remaining`.

A 429 is the most polite answer a site can give a scraper. It says the content is available, just not at your current pace. Sites that see their 429s ignored usually escalate to 403s, CAPTCHAs or permanent blocks.

## What does a 429 look like in Python, curl, Node.js and browsers?

The same failure prints differently in every tool. These are the messages to search your logs for:

- **Python requests / httpx:** `status_code == 429`, often with `Retry-After: 30`
- **Scrapy:** handled by `RetryMiddleware`, since 429 is in the default `RETRY_HTTP_CODES`
- **curl:** `HTTP/2 429` followed by `retry-after:` in the response headers
- **APIs:** a JSON body such as `{"error": "rate_limited"}`

## What causes a 429?

Most likely first:

- **Too much concurrency per host.** Fifty parallel workers on one domain look like an attack to most sites.
- **No delay between requests**, or a fixed delay that lines up many workers at the same instant.
- **Retries without backoff.** A retry storm after the first 429 multiplies the load.
- **Shared limits.** Some sites count requests per account, per API key or per session cookie, not per IP.
- **Heavy endpoints.** Search and filter URLs often have stricter limits than static pages.

## How do I diagnose a 429 step by step?

1. Log the `Retry-After` value and any rate-limit headers on the first 429.
2. Measure your real request rate per host, not your total rate.
3. Check whether the limit follows the IP, the cookie or the API key: change one at a time.
4. Read the site's API or robots.txt documentation; a `Crawl-delay` line states the pace it asks for.

*Test the failing URL with curl (HTTP port, verbose)*

```bash
# Use the URL that fails; with https://api.ipify.org instead, you test the gateway alone.
# "HTTP/1.1 200 Connection established" first, then the site's own status: the site answered.
curl -v -x http://USERNAME:PASSWORD@gw.crawlrelay.com:7777 https://TARGET_URL

# Same test through the SOCKS5 port, with DNS resolved at the exit
curl -v --socks5-hostname gw.crawlrelay.com:7778 --proxy-user USERNAME:PASSWORD https://TARGET_URL
```

## How do I fix a 429?

- **Honour Retry-After.** Wait at least the time the server asks for.
- **Cap concurrency per host**, not only globally. Scrapy has `CONCURRENT_REQUESTS_PER_DOMAIN` and `AUTOTHROTTLE_ENABLED`.
- **Back off exponentially with jitter**: 1 s, 2 s, 4 s and so on, plus a random fraction, capped at a maximum.
- **Fetch less.** Cache responses, send `If-Modified-Since` or `If-None-Match`, and skip pages that did not change.
- **Use the official API** when the site offers one; its limits are usually documented.

*Python: retry with Retry-After, exponential backoff and jitter*

```python
import random
import time

import requests

PROXY = "http://USERNAME:PASSWORD@gw.crawlrelay.com:7777"
PROXIES = {"http": PROXY, "https": PROXY}
RETRY_STATUSES = {429, 502, 503, 504}


def fetch(url, attempts=5, timeout=(10, 30)):
    for attempt in range(attempts):
        try:
            r = requests.get(url, proxies=PROXIES, timeout=timeout)
        except (requests.exceptions.ConnectionError, requests.exceptions.Timeout):
            r = None  # dropped exit, reset or timeout: a new connection gets a new exit
        if r is not None and r.status_code not in RETRY_STATUSES:
            return r
        retry_after = r.headers.get("Retry-After", "") if r is not None else ""
        if retry_after.isdigit():
            wait = int(retry_after)
        else:
            wait = min(60, 2 ** attempt) + random.uniform(0, 1)
        time.sleep(wait)
    raise RuntimeError("giving up on %s after %d attempts" % (url, attempts))
```

## Should I retry after a 429?

Yes, a 429 is meant to be retried, later. Wait for the `Retry-After` delay if the server sends one; otherwise use exponential backoff with jitter and a cap. Count consecutive 429s per host and lower that host's concurrency when the count grows. The Python function below does the basic part.

## How does 429 Too Many Requests apply to CrawlRelay?

A 429 that reaches you through CrawlRelay carries the target's own headers and body: the limit is the site's, and per-host pacing is your job. Use sticky sessions when a site counts requests per session, so that your pace stays measurable, and see the [concurrency guide](https://crawlrelay.com/docs/concurrency/) for numbers to start with.

Gateway: `gw.crawlrelay.com`, port 7777 (HTTP, HTTPS through CONNECT) and port 7778 (SOCKS5). Username format: `USERNAME-country-{cc}-state-{state}-city-{city}-zip-{zip}-asn-{asn}-session-{id}-ttl-{minutes}`. Related: [403 Forbidden](https://crawlrelay.com/docs/errors/403-forbidden/) · [503 Service Unavailable](https://crawlrelay.com/docs/errors/503-service-unavailable/) · [Concurrency guide](https://crawlrelay.com/docs/concurrency/).

## 429 Too Many Requests: questions developers ask

### Should I rotate IPs to avoid 429 errors?

No. A 429 is the site's request to slow down. Spreading the same request rate over more IPs to stay under a per-IP limit is evasion, and most terms of use forbid it. Rotation is useful for load spreading and for location, not for exceeding a published limit. Lower your rate and cache what you can.

### How long should I wait after a 429?

Wait for the number of seconds in the `Retry-After` header, or until the date it gives. Without the header, start at one or two seconds and double the delay after each new 429, with a random extra and a cap of about a minute. Resume at a lower rate, not at the rate that caused the 429.

### Why do I get 429 with only a few requests per minute?

The limit may be per account, cookie or API key rather than per IP, or the endpoint may be expensive, like a search page. Some sites also set low limits for clients they classify as bots. Check the response headers and the site's API documentation, and ask the site for access if you need more.

## Other proxy errors in this reference

- [401 Unauthorized vs 407](https://crawlrelay.com/docs/errors/401-unauthorized/)
- [403 Forbidden](https://crawlrelay.com/docs/errors/403-forbidden/)
- [407 Proxy Authentication Required](https://crawlrelay.com/docs/errors/407-proxy-authentication-required/)
- [429 Too Many Requests](https://crawlrelay.com/docs/errors/429-too-many-requests/)
- [502 Bad Gateway](https://crawlrelay.com/docs/errors/502-bad-gateway/)
- [503 Service Unavailable](https://crawlrelay.com/docs/errors/503-service-unavailable/)
- [504 Gateway Timeout](https://crawlrelay.com/docs/errors/504-gateway-timeout/)
- [Cannot connect to proxy (ProxyError)](https://crawlrelay.com/docs/errors/cannot-connect-to-proxy/)
- [Connection reset by peer (ECONNRESET)](https://crawlrelay.com/docs/errors/connection-reset/)
- [ERR_PROXY_CONNECTION_FAILED](https://crawlrelay.com/docs/errors/err-proxy-connection-failed/)
- [ERR_TUNNEL_CONNECTION_FAILED](https://crawlrelay.com/docs/errors/err-tunnel-connection-failed/)
- [Read timed out (ReadTimeout)](https://crawlrelay.com/docs/errors/read-timeout/)
- [SOCKS5 proxy errors](https://crawlrelay.com/docs/errors/socks5-proxy-errors/)
- [SSL: CERTIFICATE_VERIFY_FAILED](https://crawlrelay.com/docs/errors/ssl-certificate-verify-failed/)
- [SSL: WRONG_VERSION_NUMBER](https://crawlrelay.com/docs/errors/ssl-wrong-version-number/)

**Residential proxies with docs that cover the failure cases** CrawlRelay residential proxies from $0.50/GB, traffic that never expires. [See pricing](https://crawlrelay.com/pricing/) · [Quick start](https://crawlrelay.com/docs/)

---

CrawlRelay: CrawlRelay is a developer-first rotating residential and 4G/5G mobile proxy service for scrapers and AI data pipelines: non-expiring pay-as-you-go traffic from $0.50/GB, paid in crypto, with no ID upload. No KYC — no ID, no selfie, no documents. Pay in crypto.