403, 429 and CAPTCHAs: respond, don't evade

What is each signal telling my crawler?

• 429 is the politest: the content is available, but not at your pace. RFC 6585 defines it, often with a Retry-After header that says when to return.
• 403 is a refusal of this request. It can be fixable (wrong country, missing headers, too fast) or final (the site does not allow automated access to that content).
• A CAPTCHA or a challenge page, often served with a 403 or 503 status, means the site's bot management doubts your traffic. It is an escalation: the step after ignored 429s.

All three are rising signals. Sites that see their 429s ignored escalate to 403s and challenges, then to long blocks on whole IP ranges, which hurts every user of those ranges.

How do I build a crawler that responds correctly?

Put the rules in the fetcher, not in each spider. This one sends one request at a time per host, honours robots.txt and its Crawl-delay, waits for Retry-After, and stops asking a host after three refusals in a row. It treats a 4xx robots.txt as permission and a 5xx as a full disallow, as RFC 9309 specifies.

import time
import urllib.robotparser
from urllib.parse import urlsplit

import requests

PROXY = "http://USERNAME-country-us:[email protected]:7777"
USER_AGENT = "ExampleCrawler/1.0 (+https://example.com/crawler-info)"


class PoliteFetcher:
    """One request at a time per host; robots.txt, Crawl-delay and Retry-After honoured."""

    def __init__(self, min_interval=1.0, max_refusals=3):
        self.min_interval, self.max_refusals = min_interval, max_refusals
        self.http = requests.Session()
        self.http.proxies = {"http": PROXY, "https": PROXY}
        self.http.headers["User-Agent"] = USER_AGENT
        self.robots, self.next_time, self.refusals = {}, {}, {}

    def _robots(self, root):
        if root not in self.robots:
            rp = urllib.robotparser.RobotFileParser(root + "/robots.txt")
            try:
                r = self.http.get(root + "/robots.txt", timeout=(10, 30))
                if r.status_code >= 500:
                    rp.parse(["User-agent: *", "Disallow: /"])  # unreachable: assume disallow
                else:
                    rp.parse(r.text.splitlines() if r.status_code == 200 else [])  # 4xx: allowed
            except requests.RequestException:
                rp.parse(["User-agent: *", "Disallow: /"])
            self.robots[root] = rp
        return self.robots[root]

    def get(self, url):
        scheme, host = urlsplit(url)[:2]
        if self.refusals.get(host, 0) >= self.max_refusals:
            return None  # this host said no: stop asking
        rp = self._robots("%s://%s" % (scheme, host))
        if not rp.can_fetch(USER_AGENT, url):
            return None
        delay = max(self.min_interval, rp.crawl_delay(USER_AGENT) or 0)
        wait = self.next_time.get(host, 0) - time.monotonic()
        if wait > 0:
            time.sleep(wait)
        r = self.http.get(url, timeout=(10, 30))
        pause = delay
        if r.status_code in (429, 503):
            retry_after = r.headers.get("Retry-After", "")
            pause = max(delay, int(retry_after) if retry_after.isdigit() else 60)
        refused = r.status_code in (403, 429) or "captcha" in r.text[:5000].lower()
        self.refusals[host] = self.refusals.get(host, 0) + 1 if refused else 0
        self.next_time[host] = time.monotonic() + pause
        return r


fetcher = PoliteFetcher()
resp = fetcher.get("https://example.com/")
print(resp.status_code if resp is not None else "skipped")

What fixes a 403 legitimately?

Several 403 causes are on your side, and fixing them is fair:

• The wrong country. Many sites serve one region. Target the country with -country-<code>.
• Incomplete headers. Send Accept, Accept-Language and an honest user agent with a contact URL. A browser-like user agent on a plain HTTP client is inconsistent and often refused.
• Your own pace. A 403 that starts after N requests is a rate problem; slow down.

What is not a fix: rotating user agents to look like many different people, spoofing browser fingerprints, or cycling IPs to get around a block the site set on purpose. If a site filters hosting ranges to keep automated traffic out, a residential IP does not make that access permitted.

Why not just solve the CAPTCHAs?

Because a CAPTCHA is the site asking a question you would answer falsely. Automated solving turns a polite crawler into a tool built to defeat an access control, which raises your legal risk: the 2026 US cases over search scraping, Google v. SerpApi and Reddit v. SerpApi and others, turn on claims that defendants circumvented an anti-bot system under the DMCA's anti-circumvention rule. Courts have not settled the question, but the risk is real.

CrawlRelay does not support CAPTCHA solving or bot-detection evasion, and its Acceptable Use Policy forbids them through the gateway. If you need data behind a challenge, ask the site for access, use its API, or license the data.

When should I contact the site instead?

When you need more than its defaults allow. Many sites offer an API, a data feed, a partner programme or simply a higher limit for a crawler that identifies itself and explains its purpose. Put a contact URL in your user agent, keep a page that describes your crawler and how to opt out, and answer abuse reports quickly. A crawler that can be reached gets warnings before blocks.