Quick answer
How do I use a proxy with curl?
Use curl -x http://USERNAME:[email protected]:7777 https://api.ipify.org for the HTTP proxy, or --socks5-hostname gw.crawlrelay.com:7778 with -U USERNAME:PASSWORD for SOCKS5. CrawlRelay residential proxies take the country and a sticky session in the username, such as USERNAME-country-de-session-t1-ttl-10. From $0.50/GB. Paid in crypto, with no KYC.
curl flags you need
-x / --proxy | Proxy URL: http://host:port, credentials optional inside it |
|---|---|
-U / --proxy-user | Proxy credentials as USERNAME:PASSWORD, kept out of the URL |
--socks5-hostname | SOCKS5 proxy with DNS resolved at the exit |
-v | Shows the CONNECT exchange and the proxy's status line |
-w | Prints timing, size and %{http_connect}, the proxy's CONNECT status |
--max-time, --connect-timeout | Total and connection time limits, in seconds |
What are the basic proxy commands?
Each of these prints the exit IP the website sees.
# Rotating exit, any country
curl -x http://gw.crawlrelay.com:7777 -U USERNAME:PASSWORD https://api.ipify.org
# Country and city (city availability varies)
curl -x http://gw.crawlrelay.com:7777 -U USERNAME-country-us-city-chicago:PASSWORD https://api.ipify.org
# Sticky: run it twice, the IP stays the same for up to 10 minutes (it can change earlier)
curl -x http://gw.crawlrelay.com:7777 -U USERNAME-country-de-session-t1-ttl-10:PASSWORD https://api.ipify.org
# SOCKS5, DNS resolved at the exit
curl --socks5-hostname gw.crawlrelay.com:7778 -U USERNAME-country-gb:PASSWORD https://api.ipify.orgHow do I debug a proxy connection with curl?
Run with -v and read the lines around CONNECT. A working tunnel shows HTTP/1.1 200 Connection established, then the TLS handshake with the website. Any other status on that line came from the proxy: 407 means credentials, 502 or 504 means the exit or the target. The -w format below prints the same information in one line, which is handy in loops and CI checks.
# Full exchange, including the proxy's answer to CONNECT
curl -v -x http://gw.crawlrelay.com:7777 -U USERNAME:PASSWORD https://example.com/ -o /dev/null
# One line: proxy CONNECT status, HTTP status, compressed bytes and timings
curl -s --compressed -o /dev/null -x http://gw.crawlrelay.com:7777 -U USERNAME-country-us:PASSWORD \
-w 'connect=%{http_connect} http=%{http_code} bytes=%{size_download} tcp=%{time_connect}s tls=%{time_appconnect}s ttfb=%{time_starttransfer}s total=%{time_total}s\n' \
https://example.com/How do I check that rotation and sessions work?
Request the IP several times. Without a session ID, every run is a new connection and usually shows a different IP. With a session ID, the IP should stay the same until the TTL ends, though it can change earlier.
echo "rotating:"
for i in 1 2 3 4 5; do
curl -s -x http://gw.crawlrelay.com:7777 -U USERNAME-country-us:PASSWORD https://api.ipify.org; echo
done
echo "sticky:"
for i in 1 2 3 4 5; do
curl -s -x http://gw.crawlrelay.com:7777 -U USERNAME-country-us-session-loop1-ttl-5:PASSWORD https://api.ipify.org; echo
doneHow do I use curl with a proxy on Windows?
Call curl.exe, not curl. In Windows PowerShell 5.1, curl is an alias for Invoke-WebRequest, which does not understand -x or -U. Windows 10 and 11 ship the real curl as curl.exe. In PowerShell, quote arguments that contain special characters, and write a literal $ in a password inside single quotes.
curl.exe -x http://gw.crawlrelay.com:7777 -U 'USERNAME-country-fr:PASSWORD' https://api.ipify.orgHow do I set the proxy once for every command?
curl reads the https_proxy and http_proxy environment variables, and many other tools read them too. Set them for a shell session, and use NO_PROXY for hosts that must not go through the proxy, such as your own API. Avoid putting the password in a variable that is exported to every child process on a shared machine.
export https_proxy="http://USERNAME-country-us:[email protected]:7777"
export http_proxy="$https_proxy"
export NO_PROXY="localhost,127.0.0.1,.internal.example"
curl https://api.ipify.org # now uses the proxy
unset https_proxy http_proxy # back to directcurl and proxies: questions
What is the difference between --socks5 and --socks5-hostname?
With --socks5, curl resolves the hostname on your machine and sends the IP address to the proxy. With --socks5-hostname, it sends the hostname and the proxy side resolves it. Use --socks5-hostname for scraping so DNS answers match the exit country, and so your local resolver does not see the domains you fetch.
Why does curl say 'CONNECT tunnel failed, response 407'?
The proxy rejected the credentials, or received none. Check the username and password, and URL-encode special characters if they are inside the -x URL, or pass them with -U instead. The 407 page lists the other causes.
How do I see how many bytes a page costs through the proxy?
Add --compressed -o /dev/null -w '%{size_download}\n'. --compressed makes curl ask for gzip or brotli like your scraper does, and %{size_download} then counts the compressed bytes that crossed the wire. Without it, the server sends the page uncompressed and the figure can be several times higher. Run it on a handful of real pages to estimate your traffic per 1,000 pages.
Can curl use HTTP/2 or HTTP/3 through the proxy?
HTTP/2 works inside the CONNECT tunnel when the website supports it, because the tunnel carries TLS end to end. HTTP/3 runs over UDP, which a CONNECT tunnel through an HTTP proxy does not carry, so use HTTP/2 or HTTP/1.1 through the gateway.
Test your targets from the command line
$1.50 for 1 GB, enough for thousands of curl tests.