# Proxy and web scraping glossary

> Proxy and scraping terms in plain English: backconnect gateway, sticky session, CONNECT, socks5h, 407, Retry-After, CGNAT, robots.txt, TDM opt-out.

- URL: https://crawlrelay.com/glossary/
- Last updated: 2026-10-13
- Publisher: CrawlRelay (https://crawlrelay.com)

The words you meet in proxy docs, error messages and scraping law, defined in one or two sentences each, with links to the pages that go deeper.

**Quick answer — What is a rotating residential proxy?**

A rotating residential proxy is a single gateway address that forwards each new connection through a different household IP address from a pool, so websites see ordinary home internet users. **CrawlRelay** sells rotating residential proxies for scrapers at $1.50 for 1 GB and $0.50/GB from 50 GB, with sticky sessions when you need one IP. Paid in crypto, with no KYC.

## Proxy types and networks

**Residential proxy** — A proxy whose exit IP belongs to a home internet connection from a consumer provider. Sites see an ordinary household visitor. See [residential proxies](https://crawlrelay.com/residential-proxies/).

**Mobile proxy** — A proxy whose exit IP belongs to a 4G/5G carrier network. Carriers share each IP among many subscribers, so sites block them with care. See [mobile proxies](https://crawlrelay.com/mobile-proxies/).

**Datacenter proxy** — A proxy hosted in a data centre, with an IP registered to a hosting provider. Fast and cheap, but often filtered by sites that block hosting ranges.

**ISP proxy** — A static IP announced by an internet provider but hosted in a data centre. It looks residential to some databases and stays fixed. CrawlRelay does not sell ISP proxies.

**Backconnect gateway** — One host and port that forwards your connections to many exit IPs behind it. You change exits through the username or by opening new connections, not by switching addresses.

**Exit IP (exit node)** — The IP address the target website sees. With a residential gateway it is a household connection chosen to match your targeting.

**CGNAT** — Carrier-grade NAT: a mobile or broadband carrier puts many subscribers behind a few public IPs. RFC 6598 reserves 100.64.0.0/10 for the shared space inside such networks.

**ASN** — Autonomous System Number: the identifier of a network on the internet, such as AS7922 for Comcast. ASN targeting picks exits on one internet provider.

**Sourcing** — Where a provider's residential IPs come from, and on what terms device owners share bandwidth. After the 2026 IPIDEA and NetNut takedowns, a key buying question. See [sourcing](https://crawlrelay.com/sourcing/).

## Sessions, protocols and errors

**Rotating session** — A username without a session ID: each new connection gets an IP from the pool, usually a different one. Rotation follows connections, not individual HTTP requests.

**Sticky session** — A username with a session ID and a TTL: connections with that ID keep the same exit IP for up to 30 minutes, though the IP can change earlier. See [sessions](https://crawlrelay.com/docs/sessions/).

**TTL (session length)** — How long a sticky session keeps its IP, set in minutes with `-ttl-<minutes>`. A session can end early if the device behind it goes offline.

**Keep-alive** — Reusing one TCP connection for several HTTP requests. Faster, but every request on that connection shares one exit IP.

**CONNECT** — The HTTP method a client uses to ask a proxy for a tunnel to a host and port (RFC 9110, section 9.3.6). HTTPS traffic then runs end to end inside the tunnel.

**SOCKS5** — A proxy protocol that relays TCP connections without reading HTTP (RFC 1928), with username and password authentication defined in RFC 1929.

**socks5h** — The URL scheme that tells a client to send hostnames to the SOCKS5 proxy, so DNS is resolved at the exit rather than on your machine.

**Proxy-Authorization** — The request header that carries proxy credentials, usually Basic `username:password` in Base64. The proxy removes it before forwarding.

**407 Proxy Authentication Required** — The proxy refused the request because credentials were missing or wrong. The website was never contacted. See [407](https://crawlrelay.com/docs/errors/407-proxy-authentication-required/).

**429 Too Many Requests** — The website is rate-limiting you (RFC 6585). Slow down and honour Retry-After. See [429](https://crawlrelay.com/docs/errors/429-too-many-requests/).

**Retry-After** — A response header giving the number of seconds, or a date, to wait before retrying (RFC 9110, section 10.2.3). Sent with 429 and 503 responses.

**Concurrency** — The number of requests in flight at once. By Little's law it equals throughput times latency. Cap it per target host. See [concurrency](https://crawlrelay.com/docs/concurrency/).

## Crawling practice and rules

**robots.txt** — A file at the root of a site that tells automated clients which paths they may fetch, standardised as RFC 9309 in 2022. Crawlers should not cache it for more than 24 hours.

**Crawl-delay** — A common, non-standard robots.txt line giving the seconds to wait between requests. Python's `urllib.robotparser` reads it with `crawl_delay()`.

**User agent** — The header that names the client. A responsible crawler sends its own name, version and a contact URL, never another crawler's name.

**Content Signals** — Cloudflare's robots.txt extension with `search`, `ai-input` and `ai-train` values, introduced in September 2025, telling crawlers how content may be used.

**TDM opt-out** — A rights holder's machine-readable reservation against text and data mining under EU copyright law, which providers of general-purpose AI models must respect.

**CAPTCHA** — A challenge that asks whether the visitor is human. For a crawler it is a signal to slow down or stop; CrawlRelay does not support automated solving.

**Headless browser** — A browser without a visible window, driven by code such as Playwright or Puppeteer. It renders JavaScript but downloads far more data per page than an HTTP client.

**JSON-LD** — Structured data embedded in a page as JSON, often using schema.org types such as Product and Offer. The most stable source of prices and product facts.

**WARC** — The Web ARChive file format for storing raw HTTP requests and responses, used by Common Crawl and web archives. Storing raw responses avoids paying twice for a page.

**Conditional request** — A request with `If-None-Match` or `If-Modified-Since`. An unchanged page answers `304 Not Modified` with no body, which saves traffic.

## Buying and billing

**Pay as you go** — Buying traffic once, when you need it, instead of a monthly subscription. At CrawlRelay, plans are one-off purchases from a prepaid balance.

**Traffic expiry** — Whether purchased gigabytes are lost after a period, such as 31 days or a month. CrawlRelay residential and mobile traffic never expires.

**Decimal gigabyte** — 1 GB = 1,000,000,000 bytes; a gibibyte (GiB) is 1,073,741,824 bytes. Ask which unit a provider bills in; CrawlRelay bills in decimal GB.

**Cost per successful request** — Traffic cost divided by usable responses. A better comparison between providers than the list price per GB.

**Sub-user** — A separate set of proxy credentials under one account, with its own traffic cap and usage totals, used to isolate projects or clients.

**KYC** — Know Your Customer: identity verification with documents, selfies or company papers. CrawlRelay asks for none and relies on an [Acceptable Use Policy](https://crawlrelay.com/acceptable-use/) instead.

**AUP** — Acceptable Use Policy: the rules every account follows, listing allowed uses, forbidden uses and blocked targets.

---

CrawlRelay: CrawlRelay is a developer-first rotating residential and 4G/5G mobile proxy service for scrapers and AI data pipelines: non-expiring pay-as-you-go traffic from $0.50/GB, paid in crypto, with no ID upload. No KYC — no ID, no selfie, no documents. Pay in crypto.