Quick answer
How do I use a rotating proxy in Node.js?
With fetch, create an undici ProxyAgent for http://gw.crawlrelay.com:7777 and pass it as dispatcher. With axios or got, pass a proxy agent such as https-proxy-agent or hpagent, and set proxy: false in axios. CrawlRelay residential proxies read the country and session from the username, from $0.50/GB. Paid in crypto, with no KYC.
Node.js setup at a glance
| fetch / undici | new ProxyAgent({ uri, token }), then fetch(url, { dispatcher }) |
|---|---|
| axios | httpsAgent: new HttpsProxyAgent(url) and proxy: false |
| got | agent: { https: new HttpsProxyAgent({ proxy }) } from hpagent |
| SOCKS5 | socks-proxy-agent with a socks5h:// URL on port 7778 |
| New IP per request | A new agent or a new session ID per request; pooled connections keep their exit |
| Module format | Examples are ES modules (.mjs or "type": "module") with top-level await |
How do I use a proxy with Node's fetch?
Node's global fetch is built on undici but does not read HTTP_PROXY or HTTPS_PROXY by default. Install undici and pass a ProxyAgent as the dispatcher. The token field sends the credentials as a Proxy-Authorization header, so no password ever appears in a URL that might be logged.
import { ProxyAgent, fetch } from "undici";
const credentials = Buffer.from("USERNAME-country-us:PASSWORD").toString("base64");
const dispatcher = new ProxyAgent({
uri: "http://gw.crawlrelay.com:7777",
token: `Basic ${credentials}`,
});
const res = await fetch("https://api.ipify.org?format=json", {
dispatcher,
signal: AbortSignal.timeout(30000),
});
console.log(res.status, await res.json());
await dispatcher.close();How do I use a proxy with axios?
Do not rely on axios's own proxy option for HTTPS sites: it sends the request to the proxy without a CONNECT tunnel, which many proxies reject. Disable it with proxy: false and pass an agent that opens a proper tunnel.
import axios from "axios";
import { HttpsProxyAgent } from "https-proxy-agent";
const agent = new HttpsProxyAgent("http://USERNAME-country-de:[email protected]:7777");
const client = axios.create({ httpsAgent: agent, httpAgent: agent, proxy: false, timeout: 30000 });
const { data } = await client.get("https://api.ipify.org?format=json");
console.log(data.ip);How do I use a proxy with got?
got takes one agent per protocol. hpagent follows Node's http.Agent: keep-alive is off unless you pass keepAlive: true. Leave it off, or set keepAlive: false as below, when you want each request on a new connection, and so usually on a new exit.
import got from "got";
import { HttpsProxyAgent } from "hpagent";
const proxy = "http://USERNAME-country-fr:[email protected]:7777";
const body = await got("https://api.ipify.org", {
agent: { https: new HttpsProxyAgent({ proxy, keepAlive: false }) },
timeout: { request: 30000 },
retry: { limit: 2, statusCodes: [429, 502, 503, 504] },
}).text();
console.log(body);How do I use SOCKS5 in Node.js?
Use socks-proxy-agent with the socks5h:// scheme so the hostname is resolved at the exit. It works with Node's http and https modules and with any library that accepts an agent.
import https from "node:https";
import { SocksProxyAgent } from "socks-proxy-agent";
const agent = new SocksProxyAgent("socks5h://USERNAME-country-gb:[email protected]:7778");
const req = https.get("https://api.ipify.org", { agent, timeout: 30000 }, (res) => {
let body = "";
res.on("data", (chunk) => (body += chunk));
res.on("end", () => console.log(res.statusCode, body));
});
req.on("timeout", () => req.destroy(new Error("timeout after 30 s")));
req.on("error", (err) => console.error(err.code, err.message));How do I limit concurrency per host without a library?
Promise.all over 10,000 URLs opens 10,000 requests at once. A small limiter per hostname keeps each site at a polite level while the crawl as a whole runs in parallel. The version below has no dependency and works with any client above.
import { ProxyAgent, fetch } from "undici";
const PER_HOST = 4;
const active = new Map();
const waiting = new Map();
const dispatcher = new ProxyAgent("http://USERNAME-country-us:[email protected]:7777");
async function withHostSlot(url, task) {
const host = new URL(url).hostname;
while ((active.get(host) ?? 0) >= PER_HOST) {
await new Promise((resolve) => {
if (!waiting.has(host)) waiting.set(host, []);
waiting.get(host).push(resolve);
});
}
active.set(host, (active.get(host) ?? 0) + 1);
try {
return await task();
} finally {
active.set(host, active.get(host) - 1);
waiting.get(host)?.shift()?.();
}
}
const urls = ["https://example.com/", "https://example.org/", "https://example.net/"];
const results = await Promise.all(
urls.map((url) =>
withHostSlot(url, async () => {
const res = await fetch(url, { dispatcher, signal: AbortSignal.timeout(30000) });
return [url, res.status, (await res.arrayBuffer()).byteLength];
}).catch((err) => [url, err.name, 0])
)
);
console.table(results);
await dispatcher.close();Why do I get the same IP twice, or ECONNRESET?
Agents pool connections, and a pooled connection keeps the exit it was opened with. For a new IP on each request, create a new agent per request and close it afterwards, turn keep-alive off in the agent, or put a unique -session-<id> in each request's username.
ECONNRESET and socket hang up usually come from reusing a pooled socket that the other side has already closed, or from an exit that dropped. Retry GET requests once on a new connection. The connection reset page covers the details.
Node.js and proxies: questions
Does Node.js fetch support HTTP_PROXY?
Not by default in the versions most projects run today: the global fetch ignores proxy environment variables unless you pass a dispatcher. Use an undici ProxyAgent as shown above, or set a global dispatcher with undici's setGlobalDispatcher() if every request in the process should use the proxy.
Why does axios fail with HTTPS sites behind a proxy?
axios's built-in proxy option sends HTTPS requests to the proxy in a form many proxies refuse, instead of opening a CONNECT tunnel. Set proxy: false and pass an HttpsProxyAgent as httpsAgent. The request then travels through a proper tunnel, and TLS stays between your code and the site.
How do I send different countries from one Node.js process?
Create one agent per country, each with its own username such as USERNAME-country-de or USERNAME-country-jp, and pick the agent per request. Agents are cheap, and keeping them separate also keeps their connection pools, and so their exits, separate.
Which timeout should I set in Node.js?
Always set one: AbortSignal.timeout(30000) for fetch, timeout: 30000 for axios, timeout: { request: 30000 } for got. Thirty seconds suits HTML pages; give slow targets and mobile exits up to 60 seconds, and retry once on a new connection rather than waiting longer.
Connect your Node.js crawler
$5.94 for 10 GB, $0.50/GB from 50 GB.