Before building a parser for rendered HTML, inspect how the page gets its data. This practical checklist shows how to find a website’s public JSON endpoints with browser DevTools, distinguish Fetch and XHR responses from unrelated requests, understand parameters and pagination, recognize GraphQL patterns, and document the boundaries that apply before you replay any request.
Overview
Many modern websites load the initial page shell first and request product lists, search results, prices, profiles, or other records afterward. The browser receives those records as JSON or another structured response, then uses frontend code to render them. If the data you need is already present in a permitted client-side request, inspecting that request may be more reliable than parsing deeply nested HTML.
This is not a shortcut around access controls. Treat an endpoint as usable only when you are authorized to access and automate it. Review the site’s terms, robots guidance, privacy considerations, and rate limits before collecting data. The web scraping legal and ethical checklist is a useful companion for that decision.
Your goal during discovery is to produce a small record of the request and response:
- The page or user action that triggered the request.
- The request URL, method, query parameters, and request body.
- The response format and the fields you actually need.
- Pagination, filtering, sorting, locale, or search parameters.
- Whether authentication, cookies, CSRF tokens, or other session state is required.
- Observed limits, failure responses, and a safe request rate.
Keep this record separate from production code. It becomes a useful reference when the frontend changes or when you need to explain how a data pipeline works.
Checklist by scenario
Scenario 1: Finding data loaded after a page opens
- Open the target page in a current desktop browser.
- Open DevTools and select the Network panel before refreshing.
- Enable the filter for Fetch/XHR, or use the equivalent request-type filter in your browser.
- Refresh the page and wait for the relevant content to finish loading.
- Inspect requests whose names, paths, or response previews relate to the page content. Look for terms such as
api,search,items,products,results, orgraphql, but do not assume a name proves anything. - Open each plausible request and compare its response with the visible page. Confirm that the response contains the records or metadata you need.
Use the response preview first, then inspect the raw response when you need to identify exact field names. A JSON response might contain a top-level array, a nested collection such as data.items, or a GraphQL envelope such as data plus errors.
Scenario 2: Finding requests triggered by search, filters, or scrolling
- Clear the Network log and perform one action at a time: submit a search, change a filter, select a sort option, or move to another page.
- Identify the new request created by that action rather than relying only on requests from the initial load.
- Compare the request before and after the action. Note changed query parameters, JSON body values, headers, and response fields.
- Repeat the test with a second search term or filter to separate fixed values from dynamic inputs.
Pagination may use a page number, offset, cursor, continuation token, or a link supplied in the response. Record where the next value comes from. A cursor often cannot be safely guessed or reused indefinitely; treat it as response data and test the permitted workflow carefully.
Scenario 3: Recognizing a GraphQL request
GraphQL commonly uses a request body containing an operation name, a query, and variables. The URL alone may not identify the resource because several page features can share one endpoint. Inspect the request payload and response together:
operationNamemay identify the frontend operation.variablesmay contain IDs, search terms, filters, or cursors.querydescribes the requested fields and nested relationships.- The response usually places returned records under
data, with possible messages undererrors.
Capture only the fields required for your use case. A smaller selection makes your extraction logic easier to validate and reduces unnecessary data transfer.
Scenario 4: Checking whether the request can be replayed
Use DevTools’ copy-as-cURL or equivalent feature only for an authorized diagnostic check. Review the generated command before running it. Remove secrets from notes, avoid sharing cookies or authorization headers, and confirm that the request does not depend on a short-lived token or a user-specific session. A request that works in a browser is not automatically a stable public API.
When an endpoint requires an account, permission, or session state, document that requirement instead of trying to bypass it. If direct requests are unsuitable, a browser automation workflow may be more appropriate, but it still must respect the same authorization and rate-limit boundaries. The web scraping stack comparison checklist can help with that choice.
What to double-check
Request details
- Method: GET and POST requests carry parameters in different places.
- Query and body: preserve encoding, arrays, nested JSON, and empty values accurately.
- Headers: distinguish required content negotiation or language headers from browser noise.
- Cookies and tokens: identify whether the response is public, session-based, or account-specific.
- Locale: language, currency, region, or timezone parameters may change the returned records.
Response details
- Confirm the response status and content type instead of trusting a successful-looking preview.
- Check whether missing values are represented by null, an empty array, an omitted field, or an error object.
- Look for duplicate records across pages and verify the endpoint’s sort order.
- Compare a few response values with the rendered page to detect formatting or transformation performed by frontend code.
- Save a small, sanitized fixture for testing. The web scraper testing guide covers fixtures, mocks, and regression checks.
Before coding, define a schema for the fields you intend to keep. Validation makes it easier to detect a changed endpoint, an incomplete page, or a response that contains an error message with an unexpected status. See the data validation guide for a reusable approach.
Common mistakes
- Filtering too early: Looking only at Fetch/XHR can hide useful document, script, or WebSocket activity. Use the filter to narrow the list, then broaden it if the data is absent.
- Confusing analytics with application data: Tracking, telemetry, and advertising requests may contain familiar words but not the records you need.
- Copying every header: Browser-generated headers can become stale or unnecessary. Start with the smallest request that works in an authorized environment.
- Ignoring the request body: POST-based searches and GraphQL operations often put the important parameters in JSON rather than the URL.
- Assuming one response equals one page: Infinite scroll, lazy loading, and client-side merging can require several requests.
- Hard-coding a cursor: Cursors can depend on a previous response, sort order, or session. Store and process them according to the observed workflow.
- Starting with proxies or rotation: Infrastructure does not solve a poorly understood endpoint or permission problem. First establish a compliant, low-volume request pattern. For later operational questions, consult the proxy and IP rotation checklist.
- Skipping alternate structured data: JSON may not be necessary when the page already exposes JSON-LD, Open Graph, or microdata. Compare those options using the structured data extraction guide.
When to revisit
Re-run this discovery checklist whenever a page redesign changes its frontend, a parser begins returning empty fields, a search or filter stops working, or the site adds a new locale or content type. Also revisit it before seasonal planning cycles if your extraction depends on changing catalogs, rankings, inventories, or editorial collections.
Keep a dated note of the last successful request, a sanitized response fixture, the fields selected, and the assumptions behind pagination. When a workflow changes, compare a new Network capture with the saved record rather than rewriting the scraper blindly. Check for changed status codes, response shapes, field names, authentication requirements, and rate-limit behavior. The HTTP errors troubleshooting guide is useful when a previously valid request starts returning 403, 429, 500, or timeout responses.
Finally, test one complete path from page action to stored record: open the permitted page, trigger the request, collect the response, validate the schema, and check for duplicates or missing fields. If that path is documented and tested, you have more than a discovered URL—you have a maintainable frontend workflow reference that can guide future extraction work.