Proxy and web scraping glossary

A proxy whose exit IP belongs to a home internet connection from a consumer provider. Sites see an ordinary household visitor. See residential proxies.

A proxy whose exit IP belongs to a 4G/5G carrier network. Carriers share each IP among many subscribers, so sites block them with care. See mobile proxies.

A proxy hosted in a data centre, with an IP registered to a hosting provider. Fast and cheap, but often filtered by sites that block hosting ranges.

A static IP announced by an internet provider but hosted in a data centre. It looks residential to some databases and stays fixed. CrawlRelay does not sell ISP proxies.

One host and port that forwards your connections to many exit IPs behind it. You change exits through the username or by opening new connections, not by switching addresses.

The IP address the target website sees. With a residential gateway it is a household connection chosen to match your targeting.

Carrier-grade NAT: a mobile or broadband carrier puts many subscribers behind a few public IPs. RFC 6598 reserves 100.64.0.0/10 for the shared space inside such networks.

Autonomous System Number: the identifier of a network on the internet, such as AS7922 for Comcast. ASN targeting picks exits on one internet provider.

Where a provider's residential IPs come from, and on what terms device owners share bandwidth. After the 2026 IPIDEA and NetNut takedowns, a key buying question. See sourcing.

A username without a session ID: each new connection gets an IP from the pool, usually a different one. Rotation follows connections, not individual HTTP requests.

A username with a session ID and a TTL: connections with that ID keep the same exit IP for up to 30 minutes, though the IP can change earlier. See sessions.

How long a sticky session keeps its IP, set in minutes with -ttl-<minutes>. A session can end early if the device behind it goes offline.

Reusing one TCP connection for several HTTP requests. Faster, but every request on that connection shares one exit IP.

The HTTP method a client uses to ask a proxy for a tunnel to a host and port (RFC 9110, section 9.3.6). HTTPS traffic then runs end to end inside the tunnel.

A proxy protocol that relays TCP connections without reading HTTP (RFC 1928), with username and password authentication defined in RFC 1929.

The URL scheme that tells a client to send hostnames to the SOCKS5 proxy, so DNS is resolved at the exit rather than on your machine.

The request header that carries proxy credentials, usually Basic username:password in Base64. The proxy removes it before forwarding.

The proxy refused the request because credentials were missing or wrong. The website was never contacted. See 407.

The website is rate-limiting you (RFC 6585). Slow down and honour Retry-After. See 429.

A response header giving the number of seconds, or a date, to wait before retrying (RFC 9110, section 10.2.3). Sent with 429 and 503 responses.

The number of requests in flight at once. By Little's law it equals throughput times latency. Cap it per target host. See concurrency.

A file at the root of a site that tells automated clients which paths they may fetch, standardised as RFC 9309 in 2022. Crawlers should not cache it for more than 24 hours.

A common, non-standard robots.txt line giving the seconds to wait between requests. Python's urllib.robotparser reads it with crawl_delay().

The header that names the client. A responsible crawler sends its own name, version and a contact URL, never another crawler's name.

Cloudflare's robots.txt extension with search, ai-input and ai-train values, introduced in September 2025, telling crawlers how content may be used.

A rights holder's machine-readable reservation against text and data mining under EU copyright law, which providers of general-purpose AI models must respect.

A challenge that asks whether the visitor is human. For a crawler it is a signal to slow down or stop; CrawlRelay does not support automated solving.

A browser without a visible window, driven by code such as Playwright or Puppeteer. It renders JavaScript but downloads far more data per page than an HTTP client.

Structured data embedded in a page as JSON, often using schema.org types such as Product and Offer. The most stable source of prices and product facts.

The Web ARChive file format for storing raw HTTP requests and responses, used by Common Crawl and web archives. Storing raw responses avoids paying twice for a page.

A request with If-None-Match or If-Modified-Since. An unchanged page answers 304 Not Modified with no body, which saves traffic.

Buying traffic once, when you need it, instead of a monthly subscription. At CrawlRelay, plans are one-off purchases from a prepaid balance.

Whether purchased gigabytes are lost after a period, such as 31 days or a month. CrawlRelay residential and mobile traffic never expires.

1 GB = 1,000,000,000 bytes; a gibibyte (GiB) is 1,073,741,824 bytes. Ask which unit a provider bills in; CrawlRelay bills in decimal GB.

Traffic cost divided by usable responses. A better comparison between providers than the list price per GB.

A separate set of proxy credentials under one account, with its own traffic cap and usage totals, used to isolate projects or clients.

Know Your Customer: identity verification with documents, selfies or company papers. CrawlRelay asks for none and relies on an Acceptable Use Policy instead.

Acceptable Use Policy: the rules every account follows, listing allowed uses, forbidden uses and blocked targets.