Proxies for AI training data and LLM web corpora

How much proxy traffic does an AI training corpus need?

Fetch HTML and text, not rendered pages, and the volume stays manageable. With 50 KB per compressed HTML page:

Do I need proxies at all for AI training data?

Often not for the bulk of it. Before you crawl, check three cheaper sources:

• Open crawls. Common Crawl publishes petabytes of web archives collected by its CCBot crawler.
• Official dumps and APIs. Wikipedia, many governments, open-data portals and research repositories publish downloads and APIs with clear licences.
• Licensed data. Publishers and platforms increasingly sell or license access, and Cloudflare's pay-per-crawl uses HTTP 402 to put a price on crawling.

Proxies earn their cost on the rest: fresh public pages that are not in open crawls, content that changes by country or city, and sites that serve different editions per region. Residential exits let you collect what a local visitor sees.

Which rules apply to AI data crawlers in 2026?

Four layers matter, from technical to legal:

1. robots.txt (RFC 9309). The standard way a site says which paths automated clients may fetch. Parse it per site, apply the group for your crawler's user agent or for *, and do not cache it for more than 24 hours, as the RFC recommends.
2. AI-specific signals. Cloudflare's Content Signals Policy adds ai-train, ai-input and search values in robots.txt; more than 3.8 million domains use Cloudflare's managed robots.txt. The IETF aipref working group is drafting a standard vocabulary, still without consensus in October 2026.
3. EU text-and-data-mining opt-outs. Under the EU copyright rules for text and data mining and Article 53 of the AI Act, providers of general-purpose AI models must respect machine-readable rights reservations. In Kneschke v. LAION (Hamburg Higher Regional Court, 10 December 2025), LAION, a non-commercial research association, relied on the scientific-research exception, and a plain-language reservation was judged not machine-readable with 2021 technology; the court said this could change, and an appeal to the Federal Court of Justice was allowed. For a commercial crawler, treat plain-language reservations in terms of use as binding today.
4. Personal data. The EDPB's Opinion 28/2024 says legitimate interest can be a legal basis for AI training, case by case, and the CNIL published recommendations on scraping for AI in June 2025.

This is not legal advice. Read our 2026 overview of web scraping law and ask counsel for your project.

How do I check robots.txt and AI training signals before I crawl?

Check both before you queue any URL of a site. The function below fetches robots.txt through the proxy, applies RFC 9309's rules for missing (4xx) and unreachable (5xx) files, and reads a Cloudflare-style Content-Signal line. It is deliberately simple: it applies any ai-train=no in the file to your crawler.

import urllib.robotparser
from urllib.parse import urlsplit

import requests

PROXY = "http://USERNAME:[email protected]:7777"
PROXIES = {"http": PROXY, "https": PROXY}
USER_AGENT = "ExampleDataBot/1.0 (+https://example.com/bot)"
_rules = {}


def site_rules(url):
    """Robots rules and Content-Signal values for the site of url (cache them for 24 h at most)."""
    scheme, host = urlsplit(url)[:2]
    root = "%s://%s" % (scheme, host)
    if root not in _rules:
        try:
            r = requests.get(root + "/robots.txt", proxies=PROXIES, timeout=(10, 30),
                             headers={"User-Agent": USER_AGENT})
            status, text = r.status_code, r.text
        except requests.RequestException:
            status, text = 599, ""
        if 400 <= status < 500:
            lines = []  # RFC 9309: unavailable, crawling allowed
        elif status >= 500:
            lines = ["User-agent: *", "Disallow: /"]  # unreachable: assume complete disallow
        else:
            lines = text.splitlines()
        parser = urllib.robotparser.RobotFileParser()
        parser.parse(lines)
        signals = {}
        for line in lines:
            if line.lower().startswith("content-signal:"):
                for part in line.split(":", 1)[1].split(","):
                    key, _, value = part.strip().partition("=")
                    signals[key.strip().lower()] = value.strip().lower()
        _rules[root] = (parser, signals)
    return _rules[root]


def may_collect_for_training(url):
    parser, signals = site_rules(url)
    return parser.can_fetch(USER_AGENT, url) and signals.get("ai-train") != "no"


print(may_collect_for_training("https://example.com/some/page"))

How do I handle personal data in a web corpus?

Collect less, then remove more. Under the GDPR and similar laws, public does not mean free to use: names, emails, phone numbers and profile pages are personal data wherever they appear.

• Exclude by design. Skip profile, people-search and login-only areas, and sites whose main content is personal data.
• Filter after collection. Detect and remove emails, phone numbers and identifiers before data reaches training.
• Document your legal basis. The EDPB's Opinion 28/2024 and the CNIL's 2025 recommendations describe the balancing test for legitimate interest.
• Honour objections. Keep a way to remove a site or a person from future crawls and from your stored data.

CrawlRelay's Acceptable Use Policy forbids collecting personal data in breach of the law that applies to it.

How should an AI data pipeline use the proxy?

• One sub-user per crawl with a traffic cap, so a runaway job cannot drain the plan and each dataset has its own usage record.
• Rotate for breadth. Independent pages leave through different exits; use sticky sessions only for paginated flows.
• Cap per host. Two to four requests in flight per site, more only for large sites that answer fast. See the concurrency guide.
• Fetch once. Deduplicate URLs, use If-Modified-Since on re-crawls, and store raw responses (for example as WARC) so you never pay twice for the same page.
• Identify your crawler. A clear user agent with a contact URL lets site owners reach you instead of blocking you.