Sticky sessions and IP rotation: session IDs, TTL and connection reuse

What happens at the gateway when I send a session ID?

The gateway reads the username on each new connection. Without a session part, it picks a residential exit that matches your targeting for that connection only. With a session part, it keeps the same exit for that ID for up to the TTL you set.

A sticky IP is a real household connection, so it can disappear before the TTL ends: the device is switched off, loses Wi-Fi or changes network. Long flows must survive that, by checking the exit IP at checkpoints and restarting from a checkpoint instead of failing the whole job.

How do I manage a pool of sticky sessions in Python?

A small pool spreads load over a few stable exits and retires each one after a set number of requests or after a sign that its exit is failing, such as 502 or 503. Note what it does not do: it does not swap IPs to get past a 403 or a 429. Those mean the site wants you to slow down, so the pool raises them to your code.

import random
import uuid

import requests

GATEWAY = "gw.crawlrelay.com:7777"
BASE_USER, PASSWORD = "USERNAME-country-us", "PASSWORD"


class SlowDown(Exception):
    """The target asked us to slow down (403 or 429): back off, do not rotate."""


class StickySession:
    def __init__(self, ttl=15, max_requests=50):
        self.id = uuid.uuid4().hex[:10]
        self.max_requests, self.used = max_requests, 0
        user = "%s-session-%s-ttl-%d" % (BASE_USER, self.id, ttl)
        proxy = "http://%s:%s@%s" % (user, PASSWORD, GATEWAY)
        self.http = requests.Session()
        self.http.proxies = {"http": proxy, "https": proxy}


class SessionPool:
    EXIT_PROBLEMS = {502, 503, 504}

    def __init__(self, size=5, **options):
        self.options = options
        self.sessions = [StickySession(**options) for _ in range(size)]

    def _replace(self, i):
        self.sessions[i].http.close()
        self.sessions[i] = StickySession(**self.options)

    def get(self, url, **kwargs):
        i = random.randrange(len(self.sessions))
        s = self.sessions[i]
        s.used += 1
        try:
            r = s.http.get(url, timeout=(10, 30), **kwargs)
        except requests.exceptions.RequestException:
            self._replace(i)  # dropped exit or timeout: start a fresh session
            raise
        if r.status_code in (403, 429):
            raise SlowDown("%s returned %d" % (url, r.status_code))
        if r.status_code in self.EXIT_PROBLEMS or s.used >= s.max_requests:
            self._replace(i)
        return r


pool = SessionPool(size=5, ttl=15, max_requests=50)
print(pool.get("https://api.ipify.org").text)

How do I know that a sticky IP has changed?

Ask. Request an IP echo service such as https://api.ipify.org through the same session at the start of a flow and at each checkpoint, and compare the answers. A change means the exit behind the session changed: restart the flow from its last checkpoint, or open a new session. Log it with the session ID so you can tell IP changes apart from site behaviour when you debug.

Which session mistakes cost the most?

• Rotating on every request of a multi-step flow. The site sees a cart, a cursor or a search token jump between countries and resets it.
• One session ID for every worker. A session is meant for one flow; give each worker and each flow its own ID.
• Sessions far longer than the job. A 30-minute TTL for a 2-minute flow keeps an IP busy for nothing and makes logs harder to read.
• Treating a 429 as an IP problem. A new IP does not change the site's rule; slowing down does. See 429 Too Many Requests.
• Assuming rotation per request. Keep-alive keeps the exit; check the table above for your library.