Web Scraping Legal and Ethical Checklist: Robots.txt, Terms, Personal Data, and Rate Limits

Overview

Responsible web scraping starts before the first HTTP request. A technically successful scraper can still create avoidable problems if it ignores a website’s published instructions, collects more personal data than necessary, overloads a service, or crosses an authentication boundary.

Use the following questions during project planning:

• Purpose: Can you explain why the data is needed and what decision or workflow it supports?
• Scope: Which domains, paths, page types, fields, and request methods are actually required?
• Permission and restrictions: Have you reviewed robots.txt, terms of service, access controls, API documentation, and any relevant platform rules?
• Data protection: Will the project collect personal data, sensitive attributes, account content, or information that could identify a person?
• Operational impact: Can the scraper work slowly, cache responses, and stop automatically when a site signals a problem?
• Accountability: Can you show what was collected, when it was collected, why it was collected, and how long it will be retained?

For implementation details, pair this checklist with a reliable web scraper workflow covering retries, caching, rate limits, and validation. Legal and ethical review should be treated as an ongoing part of the system, not a one-time approval.

Checklist by scenario

Public pages without an account

• Identify the exact pages and fields required; avoid broad crawling by default.
• Read and record the site’s current robots.txt instructions for the relevant user agent.
• Review the website’s terms of service and any data-use, copyright, or content-licensing notices.
• Check whether a documented API or data feed is available and better suited to the use case.
• Use a descriptive user agent where appropriate and provide a contact route for operational issues.
• Set conservative concurrency, request delays, timeouts, and stop conditions.

Pages behind a login

• Confirm that you have authorization from the account holder and the organization responsible for the account.
• Do not bypass authentication, paywalls, CAPTCHAs, access controls, or technical restrictions.
• Limit collection to the account areas and records necessary for the stated purpose.
• Protect credentials and session tokens; do not place them in source code, logs, screenshots, or exported datasets.
• Define who may access the collected data and how access will be removed when the project ends.

Personal or user-generated data

• List every personal field the scraper may encounter, including names, usernames, emails, locations, profile text, and identifiers.
• Collect the minimum necessary fields and exclude sensitive or irrelevant attributes by design.
• Decide whether the data should be redacted, hashed, aggregated, or discarded immediately.
• Document the intended purpose, retention period, access controls, deletion process, and handling of correction or removal requests where applicable.
• Obtain a jurisdiction-specific privacy review when the data, scale, audience, or use case creates material risk.

High-volume or recurring crawling

• Estimate requests per domain before scheduling the job.
• Prefer incremental updates, conditional requests, and cached responses over repeated full crawls.
• Use backoff after errors, throttling signals, or unusual response patterns.
• Separate domains and queues so a problem on one site does not create pressure on another.
• Provide a kill switch and alerting for spikes in traffic, repeated failures, or unexpected content changes.

Commercial, research, or redistribution use

• Clarify whether the output is for internal analysis, a customer-facing product, resale, publication, or model training.
• Review whether the intended use creates additional copyright, licensing, privacy, contractual, or platform-policy concerns.
• Preserve source URLs and collection timestamps for provenance, while avoiding unnecessary storage of full page copies.
• Have an approval owner sign off on the scope before production crawling begins.

What to double-check

Robots.txt is guidance, not a complete legal decision

Parse robots.txt accurately and apply the rules to the user agent and paths you intend to crawl. Record the file version or retrieval time because instructions can change. Do not assume that permission in robots.txt resolves questions about terms, copyright, privacy, contracts, or access controls. Conversely, do not treat a missing or permissive robots.txt file as blanket permission for every use.

Test your parser with disallowed paths, wildcard rules, query strings, and redirects. If your tooling supports multiple user agents, verify which identity is actually being used. Keep robots handling separate from selector and extraction code so it can be reviewed and updated independently.

Rate limits should protect the site and your workflow

A rate limit is not merely a way to avoid a block. It should reflect the target site’s capacity, the size of the job, and the value of timely results. Start with low concurrency, introduce jitter rather than synchronized bursts, cache successful responses, and back off after server errors. A proxy or IP rotation setup should not be used to conceal excessive traffic or defeat a site’s restrictions. See the proxy and IP rotation checklist for infrastructure considerations.

Data handling needs the same attention as extraction

Define a data map before coding: source field, purpose, sensitivity, destination, retention period, and deletion method. Keep raw HTML separate from normalized records, restrict access to both, and avoid logging response bodies or personal values during debugging. Add validation that prevents unexpected fields from silently entering production exports.

Keep an audit record

Store the project owner, approved domains and paths, purpose, robots.txt review date, terms review date, collection schedule, user agent, rate settings, data fields, retention rule, and incident contact. This record makes future reviews faster and helps explain decisions if the scope changes.

Common mistakes

• Starting with a broad crawl: Begin with a small fixture set and expand only when each field is justified.
• Assuming public means unrestricted: Public visibility does not answer every contractual, privacy, copyright, or use question.
• Ignoring authentication boundaries: Never treat exposed endpoints, guessed URLs, or weak controls as permission to access more data.
• Using retries without limits: Unbounded retries can multiply traffic during an outage. Set caps, exponential backoff, and a circuit breaker.
• Rotating identities to bypass blocks: A block or explicit restriction is a signal to pause and investigate, not simply a prompt to add more proxies.
• Collecting first and reviewing later: Define fields, retention, and access controls before production collection.
• Failing silently: Monitor status codes, response volume, extraction counts, and schema changes so the scraper stops when assumptions break.
• Skipping change management: A new domain, country, data type, customer, or publication channel can change the risk profile and requires a fresh review.

For a technical implementation review, use a web scraper testing workflow with fixtures, mocks, selector checks, and regression tests. Clear tests support responsible operation because they reduce unnecessary requests made while debugging production pages.

When to revisit

Recheck this checklist before each new scraper, before expanding a crawl, and whenever the workflow or its inputs change. Put a recurring review on the project calendar before seasonal planning cycles, especially if traffic, page structure, or business requirements may change. Review immediately when:

• the site updates robots.txt, terms, authentication, or API documentation;
• the scraper receives blocks, complaints, unusual error rates, or a contact request;
• you add personal data, authenticated pages, a new country, or a new downstream user;
• the collection frequency, request volume, proxy configuration, or browser automation stack changes;
• the output moves from internal analysis to publication, resale, customer delivery, or automated decision-making;
• privacy, data protection, copyright, or platform rules relevant to the project change.

At each review, compare the approved scope with the live configuration, run a small test crawl, confirm rate limits and stop conditions, verify retention and deletion jobs, and record the reviewer and date. If the answer to a key permission, privacy, or authorization question is unclear, pause production collection and obtain qualified legal or compliance guidance. Responsible web scraping is not about extracting as much as possible; it is about collecting only what you can justify, at a pace the service can handle, with controls you can explain.