Proxy & IP Rotation for Web Scraping: Checklist, Configuration Templates, and Provider Comparison

Overview

Proxies place an intermediary between your scraper and a destination website. Depending on the provider and configuration, the destination may see the proxy address instead of the address of your application. IP rotation means changing that intermediary address according to a defined policy, such as once per request, once per session, or only after a failure.

Proxy rotation is not a substitute for responsible crawling. Before collecting data, check the site’s terms, applicable law, robots guidance where relevant, authentication requirements, and the sensitivity of the data involved. Use an identifiable client where appropriate, keep request rates conservative, cache responses, and do not use proxies to defeat access controls, CAPTCHAs, account restrictions, or other security measures. If a site does not permit your intended activity, seek permission or use an official API.

A sound design usually has five parts:

• Scope: define the permitted domains, paths, data fields, and crawl volume.
• Proxy pool: select addresses with the required protocol, region, stability, and authentication method.
• Rotation policy: decide when an address changes and when a session must remain sticky.
• Politeness controls: add delays, concurrency limits, retries, caching, and backoff.
• Observability: record status codes, latency, proxy identity, response size, and block indicators without storing unnecessary personal data.

For broader architecture decisions, see the web scraping stack comparison and decision checklist and the guide to retries, rate limits, caching, and data validation.

Checklist by scenario

Choose the proxy type

Do not select a proxy category solely because it is marketed as harder to detect. First ask whether a proxy is necessary at all, whether the target offers an API, and whether reducing concurrency or using a permitted access method solves the problem.

Requests and BeautifulSoup

For a straightforward Python scraper, keep the proxy configuration in one place and use a session so headers, cookies, and connection behavior are consistent. The exact environment-variable names depend on your provider.

import os
import requests

proxy_url = os.environ["SCRAPER_PROXY_URL"]
proxies = {
    "http": proxy_url,
    "https": proxy_url,
}

session = requests.Session()
session.headers.update({"User-Agent": "ExampleResearchBot/1.0"})

response = session.get(
    "https://example.com/catalog",
    proxies=proxies,
    timeout=20,
)
response.raise_for_status()

Use a bounded retry policy for temporary network failures, not an endless loop for access denials. Cache pages that do not change frequently, and parse only the fields you need. The Python web scraping guide covers pagination and exporting data.

Playwright

Configure a proxy when creating the browser or browser context, according to the Playwright version and provider format. Keep credentials in environment variables rather than source code.

import os
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(
        proxy={
            "server": os.environ["SCRAPER_PROXY_SERVER"],
            "username": os.environ.get("SCRAPER_PROXY_USER"),
            "password": os.environ.get("SCRAPER_PROXY_PASSWORD"),
        }
    )
    page = browser.new_page()
    page.goto("https://example.com/catalog", wait_until="domcontentloaded")
    html = page.content()
    browser.close()

Use one browser context per logical session when cookies or location need to remain consistent. Avoid creating a new browser for every URL unless testing requires it; that can add overhead and create an unnatural request pattern. Compare browser approaches in Playwright versus Puppeteer for web scraping.

Playwright with a proxy pool

For a permitted workload that needs multiple proxy endpoints, keep pool selection separate from browser logic. A simple selector can rotate after a session rather than after every request:

from itertools import cycle

proxy_pool = cycle([
    {"server": "http://proxy-a.example:8000"},
    {"server": "http://proxy-b.example:8000"},
])

# Select one endpoint for a logical session.
proxy = next(proxy_pool)
# Pass proxy to browser.launch() or the supported context configuration.
# Reuse the session for related pages, then select the next endpoint.

In production, add health checks, maximum session age, provider failure handling, and a per-domain concurrency limit. Never rotate simply to continue a request stream after a site has clearly denied it; pause, investigate, and confirm that your access is allowed.

Scrapy

Scrapy commonly accepts an HTTP proxy through request metadata. A downloader middleware component can select an endpoint from an approved pool, while settings control concurrency and delays.

def start_requests(self):
    yield scrapy.Request(
        "https://example.com/catalog",
        meta={"proxy": self.settings["SCRAPER_PROXY_URL"]},
    )

Set conservative values for CONCURRENT_REQUESTS, DOWNLOAD_DELAY, and retry behavior. Keep proxy selection, retry classification, and domain policy in separate components so each can be tested independently.

Python and aiohttp

For asynchronous scraping, the main risk is accidental concurrency. A semaphore and explicit timeout are as important as the proxy parameter:

import aiohttp
import asyncio

semaphore = asyncio.Semaphore(5)

async def fetch(session, url, proxy):
    async with semaphore:
        async with session.get(url, proxy=proxy, timeout=20) as response:
            response.raise_for_status()
            return await response.text()

Measure the real request rate across all workers, not just within one coroutine. Add jittered delays and stop conditions for repeated denial responses.

What to double-check

• Protocol support: confirm whether the endpoint supports HTTP, HTTPS tunneling, or SOCKS, and configure the client accordingly.
• Authentication: test credentials in a secret manager or environment variable; never commit proxy usernames, passwords, or tokens.
• Geography: verify the effective country or region from the response and document why location is needed.
• Session consistency: keep cookies, headers, browser state, and proxy location coherent for workflows that span several pages.
• Rotation semantics: distinguish per-request rotation from sticky sessions. The latter is usually easier to reason about for login-free multi-page flows.
• Error handling: separate DNS failures, connection timeouts, rate limits, authentication errors, server errors, and explicit denial pages.
• Data quality: compare response status, content type, body length, and expected selectors. A successful HTTP response can still contain a challenge or an error page.
• Testing: use fixtures and mocks before sending traffic to a live site. The scraper testing checklist is useful for regression coverage.

When a response changes, inspect headers and a small, redacted body sample. Selector and parsing issues can look like proxy failures. Guides to CSS selectors and XPath and structured data extraction can help isolate the application layer from the network layer.

Common mistakes

1. Rotating too aggressively: changing IPs on every request can break sessions, increase latency, and make failures harder to diagnose. Start with sticky sessions and change only when the workflow requires it.
2. Ignoring rate limits: a large proxy pool does not make unlimited traffic appropriate. Set a global budget per domain and stop when the budget or error threshold is reached.
3. Retrying denials: repeating a blocked request with new addresses can escalate the problem and may violate site rules. Treat repeated denials as a signal to pause and review permission.
4. Mixing identities: switching between regions while reusing cookies, authentication state, or browser fingerprints can produce inconsistent results. Keep session state deliberately scoped.
5. Using untrusted free proxies: unknown intermediaries can inspect or alter traffic. For sensitive workflows, use an approved provider, HTTPS, and test credentials only in a controlled environment.
6. Skipping observability: without proxy ID, timestamp, URL pattern, status, latency, and retry count, you cannot tell whether the issue is provider quality, application logic, or destination behavior.
7. Overlooking the source: proxies do not fix brittle selectors, JavaScript rendering problems, malformed URLs, or incorrect pagination. Validate the scraper itself before changing infrastructure.

When to revisit

Review the setup before a seasonal planning cycle, a new geographic launch, a major increase in volume, or a change in the scraper framework or browser version. Also revisit it when block rates, latency, empty responses, CAPTCHA pages, or proxy failure rates rise above the baseline you have documented.

Use this practical sequence:

1. Pause or reduce the affected workload rather than adding more proxies immediately.
2. Compare recent metrics with a known-good period: status codes, latency, response size, error categories, and data completeness.
3. Test one domain, one region, and one controlled session with a low request rate.
4. Check provider health, credentials, protocol support, and geographic availability.
5. Review terms, permissions, robots guidance, and any official access option.
6. Update the pool, rotation interval, concurrency, or parser only after identifying the likely cause.
7. Record the change, expected result, and rollback condition.

Finally, keep a short provider comparison record with the fields that matter to your project: proxy type, supported protocols, geographic coverage, session persistence, authentication method, observed reliability, and operational limits. Recheck those fields when providers change their products or when your workflow expands. A measured, permission-aware configuration is more durable than a large pool used without clear controls.