# Scrapy proxy middleware for rotating residential proxies

> Scrapy rotating proxy middleware: rotating and sticky residential IPs per request, AutoThrottle and retry settings, scrapy-playwright. Syntax-checked code.

- URL: https://crawlrelay.com/docs/scrapy/
- Last updated: 2026-10-13
- Publisher: CrawlRelay (https://crawlrelay.com)

A 30-line downloader middleware that picks the country and the session per request, plus the settings that keep a crawl polite: robots.txt, per-domain concurrency, AutoThrottle and retries.

**Quick answer — How do I use rotating proxies with Scrapy?**

Set `request.meta["proxy"]` to `http://USERNAME:PASSWORD@gw.crawlrelay.com:7777` in a small downloader middleware that runs before Scrapy's `HttpProxyMiddleware`; Scrapy then sends the credentials for you. With **CrawlRelay** residential proxies, the country and the sticky session go in the username, so one middleware handles both. Traffic costs $0.50/GB from 50 GB. Paid in crypto, with no KYC.

## Scrapy setup at a glance

| Attribute | Value |
| --- | --- |
| Where the proxy goes | `request.meta["proxy"]`, set by your middleware |
| Middleware order | Your middleware below 750, the priority of the built-in `HttpProxyMiddleware` |
| Credentials | In the proxy URL; Scrapy turns them into a `Proxy-Authorization` header |
| SOCKS5 | Not with the default HTTP/1.1 handler; Scrapy 2.17+ supports SOCKS through the optional HttpxDownloadHandler. The HTTP port 7777 is simplest |
| Rotation | New exit per new connection; a unique session ID per request normally gives a different IP |
| Politeness | `ROBOTSTXT_OBEY`, `CONCURRENT_REQUESTS_PER_DOMAIN`, `AUTOTHROTTLE_ENABLED` |
| Retries | 429 and 5xx are in the default `RETRY_HTTP_CODES`; Retry-After is not read by default |

## How do I write the proxy middleware?

Put the class in `middlewares.py`. It builds the username from settings and from three optional `meta` keys: `proxy_country`, `proxy_session` and `proxy_ttl`. A request without `proxy_session` rotates; a request with `proxy_session="new"` gets a new session, so normally a different exit IP; a request with a fixed session ID shares one exit with the other requests of the same flow. The middleware recomputes its proxy on every pass, retries included, and leaves alone a proxy set by other code. If `CRAWLRELAY_USER` is missing, the crawl stops at start-up instead of running from your own IP.

*middlewares.py*

```python
import uuid
from urllib.parse import quote


class CrawlRelayProxyMiddleware:
    """Sets request.meta["proxy"]; Scrapy's HttpProxyMiddleware then sends the credentials."""

    def __init__(self, user, password, country):
        self.user, self.password, self.country = user, password, country

    @classmethod
    def from_crawler(cls, crawler):
        s = crawler.settings
        if not s.get("CRAWLRELAY_USER"):
            # Fail loudly: NotConfigured would only disable the middleware and crawl from your own IP
            raise ValueError("CRAWLRELAY_USER is not set: refusing to crawl without the proxy")
        return cls(s.get("CRAWLRELAY_USER"), s.get("CRAWLRELAY_PASSWORD", ""), s.get("CRAWLRELAY_COUNTRY", "us"))

    def process_request(self, request, spider=None):
        if "proxy" in request.meta and not request.meta.get("_crawlrelay"):
            return None  # a proxy set by other code: leave it alone
        parts = [self.user, "country", request.meta.get("proxy_country", self.country)]
        session = request.meta.get("proxy_session")
        if session == "new":  # a fresh session for this request (and for each of its retries)
            session = uuid.uuid4().hex[:12]
        if session:
            parts += ["session", session, "ttl", str(request.meta.get("proxy_ttl", 10))]
        username = "-".join(parts)
        request.meta["proxy"] = "http://%s:%s@gw.crawlrelay.com:7777" % (
            quote(username, safe=""), quote(self.password, safe=""))
        request.meta["_crawlrelay"] = True
        return None
```

## Which Scrapy settings keep a proxied crawl polite and fast?

Proxies spread your requests over many IPs, but the target still sees your total rate. These settings keep each domain at a sustainable pace and let AutoThrottle slow down on its own when responses get slower. Keep `ROBOTSTXT_OBEY = True`, which new Scrapy projects already set.

*settings.py (excerpt)*

```python
import os

BOT_NAME = "myproject"
ROBOTSTXT_OBEY = True
USER_AGENT = "ExampleCrawler/1.0 (+https://example.com/crawler-info)"

CRAWLRELAY_USER = os.environ.get("CRAWLRELAY_USER")  # e.g. the username from your dashboard
CRAWLRELAY_PASSWORD = os.environ.get("CRAWLRELAY_PASSWORD")
CRAWLRELAY_COUNTRY = "us"

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.CrawlRelayProxyMiddleware": 350,  # runs before HttpProxyMiddleware (750)
}

CONCURRENT_REQUESTS = 32
CONCURRENT_REQUESTS_PER_DOMAIN = 4
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1.0
AUTOTHROTTLE_MAX_DELAY = 30.0
AUTOTHROTTLE_TARGET_CONCURRENCY = 2.0

RETRY_ENABLED = True
RETRY_TIMES = 3
RETRY_HTTP_CODES = [429, 500, 502, 503, 504, 522, 524, 408]
DOWNLOAD_TIMEOUT = 45
```

## How do I keep one IP for pagination and rotate for item pages?

Pass a fixed `proxy_session` on the requests of a flow that depends on the previous page, and none on independent pages. The spider below runs on books.toscrape.com, a sandbox site published for scraping practice.

*spiders/catalog.py*

```python
import scrapy


class CatalogSpider(scrapy.Spider):
    name = "catalog"
    start_urls = ["https://books.toscrape.com/"]

    def parse(self, response):
        for href in response.css("article.product_pod h3 a::attr(href)").getall():
            # Product pages are independent: rotating exits
            yield response.follow(href, callback=self.parse_item)
        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            # The listing flow keeps one sticky exit for 30 minutes
            yield response.follow(next_page, callback=self.parse,
                                  meta={"proxy_session": "listing1", "proxy_ttl": 30})

    def parse_item(self, response):
        yield {
            "title": response.css("div.product_main h1::text").get(),
            "price": response.css("p.price_color::text").get(),
            "url": response.url,
        }
```

## Does Scrapy reuse connections, and does that stop rotation?

Yes, Scrapy keeps connections alive and reuses them per proxy and target host, so consecutive requests on one connection share an exit. For most crawls that is fine: Scrapy opens up to `CONCURRENT_REQUESTS_PER_DOMAIN` connections per site, and each new connection usually gets a different residential IP.

When a request must leave through a new IP, for example a retry after a [502](https://crawlrelay.com/docs/errors/502-bad-gateway/) from a failing exit, re-issue it with `response.request.replace(meta={**response.meta, "proxy_session": "new"}, dont_filter=True)`. The middleware builds a new session username, a different username means a different connection pool, so Scrapy opens a new connection and the gateway normally assigns a different exit.

## How do I honour Retry-After on 429 responses in Scrapy?

Scrapy's built-in `RetryMiddleware` retries 429 but does not wait for the `Retry-After` delay. AutoThrottle slows the domain down over time, which is usually enough. For APIs that send long `Retry-After` values, add a small middleware that reads the header and schedules the retry with a delay, or lower `CONCURRENT_REQUESTS_PER_DOMAIN` for that domain. The [429 page](https://crawlrelay.com/docs/errors/429-too-many-requests/) explains what the header means.

## How do I use the proxy with scrapy-playwright?

For pages that need JavaScript, scrapy-playwright passes Playwright's launch options through a setting. Give the credentials as separate fields: Chromium ignores credentials written inside the server URL. Use one context per sticky session if you render logged-out flows that span several pages.

*settings.py for scrapy-playwright*

```python
import os

DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

PLAYWRIGHT_LAUNCH_OPTIONS = {
    "proxy": {
        "server": "http://gw.crawlrelay.com:7777",
        "username": os.environ.get("CRAWLRELAY_USER", "USERNAME") + "-country-us",
        "password": os.environ.get("CRAWLRELAY_PASSWORD", "PASSWORD"),
    },
}
```

## Scrapy and proxies: questions

### Do I need scrapy-rotating-proxies or a proxy list?

No. Those packages rotate a list of individual proxy addresses and ban the ones that fail. With a residential gateway, rotation happens behind one address: each new connection usually gets a different residential IP. A single middleware that sets `request.meta["proxy"]` is all you need.

### Why does Scrapy show TunnelError with my proxy?

`TunnelError: Could not open CONNECT tunnel` means the proxy refused or failed the CONNECT request for an HTTPS site. The status in the message tells you why: 407 is credentials, 502 or 504 is the exit or the target. See the [407](https://crawlrelay.com/docs/errors/407-proxy-authentication-required/) and [502](https://crawlrelay.com/docs/errors/502-bad-gateway/) pages.

### How many concurrent requests per domain should I use?

Start with 2 to 4 per domain and let AutoThrottle adjust. Raise the number only while the share of 429, 403 and 5xx responses stays flat, and watch the site's response times. Large sites tolerate more than small ones; a small site may need a delay between requests and a single connection.

### Should I disable ROBOTSTXT_OBEY to get more data?

No. robots.txt is how a site states which paths automated clients may fetch, and RFC 9309 standardised the format in 2022. Ignoring it raises your legal and ethical risk and breaks our [Acceptable Use Policy](https://crawlrelay.com/acceptable-use/) when it is used to reach content the site has excluded.

### How much traffic does a Scrapy crawl use?

Scrapy downloads HTML only, so pages usually cost tens of kilobytes each, compressed. Check the `downloader/response_bytes` value in the stats Scrapy prints at the end of a run and divide it by the number of responses. The [bandwidth guide](https://crawlrelay.com/guides/bandwidth-calculator/) turns that into a cost.

**Plug residential IPs into your Scrapy project** $5.94 for 10 GB to test, $0.50/GB from 50 GB. [See pricing](https://crawlrelay.com/pricing/) · [Concurrency guide](https://crawlrelay.com/docs/concurrency/)

---

CrawlRelay: CrawlRelay is a developer-first rotating residential and 4G/5G mobile proxy service for scrapers and AI data pipelines: non-expiring pay-as-you-go traffic from $0.50/GB, paid in crypto, with no ID upload. No KYC — no ID, no selfie, no documents. Pay in crypto.