HTTP errors in a web scraper are signals, not instructions to retry blindly. This guide explains how to diagnose 403, 404, 408, 429, 500, 502, 503, connection failures, and timeouts; what to record; and how to build retry logic that is measured, observable, and respectful of a website's limits.
Overview
A scraper can fail at several different layers: DNS resolution, the network connection, TLS negotiation, the HTTP request, the server application, or the parsing step after a response arrives. Treating every failure as the same problem makes debugging slower and can increase load on the target site.
Start by separating three questions:
- Did the request reach a server? A connection or DNS error may occur before an HTTP status exists.
- What did the server communicate? The status code, headers, and response body provide clues.
- Can the operation be repeated safely? A transient server failure may justify a delayed retry, while a permission response or malformed request usually requires a code or configuration change.
Before changing headers, proxies, or browser settings, confirm that your collection is permitted. Review the site's terms, robots.txt guidance, authentication requirements, personal-data obligations, and published rate limits in the web scraping legal and ethical checklist. A technically successful request is not automatically an appropriate one.
What to track
Keep structured logs for every request, not just exceptions. A useful record lets you compare a successful run with a failing run and identify whether the problem affects one URL, one host, one worker, or the entire job.
| Signal | What it helps reveal |
|---|---|
| Timestamp and timezone | Whether failures cluster around a schedule or deployment. |
| URL, host, and HTTP method | Whether one endpoint or request type is affected. |
| Status code | Whether the response suggests permission, throttling, a missing resource, or a server fault. |
| Elapsed time and phase | Whether delay occurred during connection, response waiting, or content processing. |
| Attempt number and sleep duration | Whether retry logic is working as intended. |
| Response headers such as Retry-After | Whether the server supplied timing guidance for a later request. |
| Response size and content type | Whether an error page, login page, or unexpected format replaced the expected content. |
| Scraper version, runtime, and environment | Whether a deployment, library update, or network change introduced the failure. |
Do not log credentials, session cookies, authorization tokens, or unnecessary personal data. Store a short response preview only when appropriate, and redact sensitive values before sending logs to a shared system. If you use browser automation, also record the browser version, navigation URL, and whether the failure occurred during navigation, waiting, or extraction.
Status-code comparison
| Response | Likely meaning | First action | Retry? |
|---|---|---|---|
| 403 Forbidden | The server will not authorize the request as received. | Check permission, authentication, request policy, and whether the target is intended for your client. | Not automatically. Repeated retries can worsen the problem. |
| 404 Not Found | The requested resource is unavailable at that URL. | Check URL construction, redirects, identifiers, and whether the page moved or was removed. | Usually no, unless the resource is expected to appear later. |
| 408 Request Timeout | The server did not receive the complete request in time. | Check request size, connection quality, and client timeout settings. | Sometimes, with a bounded delay. |
| 429 Too Many Requests | The client has sent requests too quickly or exceeded a stated limit. | Reduce concurrency, slow the schedule, and honor Retry-After when present. | Yes, only after waiting and within a limit. |
| 500 Internal Server Error | The server encountered an unexpected condition. | Capture request details and test whether the issue is temporary or endpoint-specific. | Often, cautiously and with backoff. |
| 502 Bad Gateway | A gateway or proxy received an invalid upstream response. | Check whether the failure is intermittent and whether an upstream dependency is involved. | Often, with backoff. |
| 503 Service Unavailable | The server is temporarily unable to handle the request. | Look for maintenance or capacity signals and reduce request pressure. | Often, after a delay; honor Retry-After. |
| Connection or DNS error | The request may not have reached the HTTP server. | Check DNS, TLS, firewall rules, network access, and the target host. | Yes for transient network faults, but cap attempts. |
Cadence and checkpoints
Use a small diagnostic sequence before changing the scraper. First, reproduce one failing URL with the same method, parameters, and environment. Then compare it with a known-good URL on the same host. Confirm whether the response is consistent across runs, because an intermittent 502 is a different investigation from a repeatable 403.
For a scheduled scraper, review these checkpoints on every run:
- Record total requests, successful responses, each status-code count, and connection failures.
- Measure response-time percentiles or at least minimum, typical, and slowest requests.
- Track retry count, abandoned URLs, and the age of the oldest unprocessed item.
- Validate a sample of extracted records so an HTTP 200 response does not hide a login page, block page, or changed layout.
- Compare the current run with recent runs instead of judging it from one failed request.
A weekly or monthly review is useful for recurring jobs. Inspect changes in error mix, latency, request volume, concurrency, and deployment version. If a job normally completes with a small number of transient failures but suddenly produces many 429 or 403 responses, pause automatic escalation and investigate request behavior first. For browser-based projects, review browser and automation-library updates separately from site changes. The web scraper testing guide can help turn representative pages into regression fixtures.
How to interpret changes
Patterns are more informative than isolated codes. A rise in 429 responses after increasing worker count points toward concurrency or rate control. A 403 that begins after an authentication change suggests a session, permission, or request-policy issue. A sudden 404 increase may indicate a URL-generation bug or a site structure change. A mixture of 500, 502, and 503 responses with increased latency can indicate temporary instability, but it may also reflect excessive request pressure from the scraper.
Use retries only for failures that are plausibly transient. A practical policy is to allow a small, fixed maximum number of attempts and use exponential backoff with jitter. For example, the delay can increase after each failed attempt while a random small offset prevents multiple workers from retrying at exactly the same time. Respect a server-provided Retry-After value when it is available, and apply an overall deadline so one URL cannot hold the entire job indefinitely.
for attempt in range(max_attempts):
try:
response = client.get(url, timeout=(connect_timeout, read_timeout))
if response.status_code in {429, 500, 502, 503}:
delay = retry_after(response) or backoff_with_jitter(attempt)
sleep(delay)
continue
response.raise_for_status()
return response
except (ConnectError, ReadTimeout):
if attempt + 1 == max_attempts:
raise
sleep(backoff_with_jitter(attempt))
raise RuntimeError("request failed after bounded retries")The exact exception names and timeout behavior depend on your HTTP library, so check the version-specific documentation before copying this pattern into production. Also distinguish connection timeout from read timeout: increasing both without a limit can make a scheduled job appear stuck. For asynchronous clients, bound concurrency with a semaphore or queue rather than launching an unrestrained task for every URL.
Retries are not a substitute for a correct request. Fix malformed URLs, missing authentication, invalid parameters, broken selectors, and policy violations at their source. If the site requires JavaScript to render data, assess whether a documented endpoint or structured representation is available before moving to browser automation. Related guidance on extracting structured data and choosing a scraping stack can help reduce unnecessary complexity.
When to revisit
Revisit this troubleshooting checklist whenever a scheduled run shows a meaningful change in status-code distribution, latency, completion time, or extracted-record quality. Also review it after a target site redesign, an authentication change, a network or proxy change, a Python or HTTP-library upgrade, a browser-automation upgrade, or a change to worker count and request pacing.
On a monthly or quarterly cadence, update your runbook with the current timeout values, retry limits, concurrency settings, known status patterns, and escalation owner. Test one controlled failure path so you know that alerts fire and failed URLs are recorded without exposing secrets. If proxy or IP rotation is part of the approved architecture, review its configuration and necessity rather than treating rotation as an automatic solution; the proxy and IP rotation checklist provides a separate review point.
For the next incident, take these actions in order: pause aggressive retries, save the request metadata, compare one failing URL with one successful URL, classify the failure by layer and status code, apply the smallest safe configuration change, and rerun a limited sample. Then record the cause and prevention step in the scraper's runbook. This turns recurring HTTP errors into measurable maintenance work instead of a cycle of guesswork.
