Technical SEO Crawling with Python: A Repeatable Workflow for Titles, Canonicals, Status Codes, and Indexability
technical SEOsite auditsPythonindexabilitycrawling

Technical SEO Crawling with Python: A Repeatable Workflow for Titles, Canonicals, Status Codes, and Indexability

WWeb Tools Lab
2026-11-16
7 min read

A repeatable Python workflow for crawling titles, canonicals, status codes, redirects, robots directives, hreflang, links, and indexability.

A technical SEO crawl is most useful when it produces consistent, comparable evidence rather than a one-time list of warnings. This workflow shows how to crawl a site with Python, collect status codes, titles, canonicals, robots directives, hreflang, links, and indexability signals, then export the results into a report that can be prioritized and checked again after changes.

Overview

A Python technical SEO crawler does not need to replace a full enterprise crawler to be valuable. A focused script can answer practical questions such as:

  • Which discovered URLs return errors, redirect, or resolve successfully?
  • Which pages lack a title, meta description, canonical link, or indexability signal?
  • Do canonical URLs point to the expected host and scheme?
  • Are important pages linked internally, or are they only present in an XML sitemap?
  • Are duplicate URLs being created by query parameters, fragments, trailing slashes, or inconsistent casing?
  • Which issues are widespread template problems and which affect only individual pages?

Begin with a defined scope. Record the starting URL, allowed domains, maximum pages, request delay, timeout, user agent, and crawl date. The goal is not to request every possible URL. It is to create a controlled sample that can be repeated safely before and after a release, migration, template change, or seasonal planning cycle.

Before crawling, review the site’s published rules, terms, and access expectations. The web scraping legal and ethical checklist is a useful companion for thinking through robots.txt, personal data, rate limits, and permitted use. Treat a technical SEO audit as an authorized diagnostic activity, not as permission to bypass controls.

A minimal crawl record

Store one structured record per requested URL. A useful schema includes:

  • requested_url: the URL taken from the queue
  • final_url: the URL after redirects
  • status_code: the final HTTP response status
  • content_type: whether the response appears to be HTML
  • title: normalized title text and character count
  • meta_description: normalized description text and character count
  • canonical: the resolved canonical URL, if present
  • robots_directives: values such as noindex or nofollow found in the page
  • hreflang: alternate language or regional references
  • internal_links: normalized links discovered in the document
  • depth: click distance from the starting URL
  • issues: consistent, export-ready issue labels

Checklist by scenario

1. Starting a new crawl

  1. Choose a seed URL, such as the homepage, an approved section page, or a sitemap URL.
  2. Normalize the seed URL by adding a scheme, removing fragments, and applying the intended host rules.
  3. Set a page limit, timeout, request delay, and maximum crawl depth.
  4. Restrict requests to the approved domain and decide how subdomains will be handled.
  5. Use an HTTP client and HTML parser that can preserve response headers and redirect history.
  6. Save raw crawl output as well as the issue summary so individual findings can be reproduced.

A simple queue-based crawler should mark URLs as seen before adding them to the queue. This prevents repeated requests when several pages link to the same destination. Resolve relative links against the current page, remove fragments, and decide consistently whether query parameters are retained, removed, or allowed only from an approved list.

2. Checking status codes and redirects

For each request, record both the requested URL and the final URL. A successful final response does not erase the fact that the original URL redirected. Report at least:

  • successful HTML responses that can be parsed;
  • client and server errors;
  • redirect chains and loops;
  • timeouts, connection errors, and blocked requests;
  • non-HTML resources encountered during discovery.

Separate a crawler failure from a site failure. A timeout may reflect network conditions, a rate limit, or an application response rather than a permanent page problem. For troubleshooting response issues, compare your findings with the HTTP errors guide, and retain the response timestamp and request context in your export.

3. Checking titles and meta descriptions

Extract the first meaningful title element and normalize whitespace. Flag missing titles, empty titles, duplicate titles, and unusually long or short values for review. Do not treat a character count as an automatic ranking judgment; use it as a triage signal for snippets and template consistency.

Apply the same process to meta descriptions. A missing or duplicated description may be a useful template finding, but the report should distinguish between a missing field, a duplicate value, and a field that merely exceeds your chosen review threshold. Store the exact text so an editor or developer can inspect the source page.

4. Checking canonicals and indexability

Parse canonical link elements and resolve relative values against the page URL. Flag missing canonicals, multiple conflicting canonical elements, invalid URLs, cross-host canonicals, and canonicals that point to an error or redirecting destination. These are review categories, not automatic proof that every canonical is wrong.

Collect robots directives from both the HTML meta element and relevant HTTP headers where available. Record directives such as noindex and nofollow separately. A page can be crawlable but intended not to appear in search results, so report crawling and indexability as related but distinct dimensions.

For international pages, collect each alternate URL and its language or region value. Flag missing return references, malformed values, duplicate declarations, and links that do not resolve. Validate these findings with the site’s intended localization model rather than assuming every language version should link to every other version.

For internal links, record source, destination, anchor text, and crawl depth. Group links by normalized destination to identify pages with no discovered internal links, excessive parameter variants, or links that repeatedly lead through redirects. If structured data is part of the audit, supplement the crawl with the guide to JSON-LD, Open Graph, and Microdata extraction.

6. Exporting an actionable report

Use one row per URL for page-level data and a second table for link relationships. Add an issue category, severity, evidence, first-seen crawl date, and recommended owner. Useful categories include status, redirect, title, description, canonical, robots, hreflang, duplicate, orphan-candidate, and crawl-error.

Prioritize findings by impact and confidence. A broad template issue affecting indexable pages usually deserves attention before an isolated formatting concern. Keep the raw values beside the recommendation so the report remains useful during implementation and verification.

What to double-check

  • URL normalization: Decide how to treat trailing slashes, uppercase paths, default ports, fragments, encoded characters, and tracking parameters. Inconsistent normalization can make one page appear to be several.
  • Rendered content: A basic HTTP request may not contain content created by client-side JavaScript. If important metadata or links are rendered in the browser, use an approved browser automation approach and compare the rendered DOM with the raw response.
  • Robots handling: Do not confuse a crawler’s local decision to avoid a URL with the page’s own robots meta directive. Record both the request decision and the page signal.
  • Canonical interpretation: A canonical is a declared preference, not necessarily the URL a search engine will select. Check whether it matches the site’s content, protocol, host, and redirect behavior.
  • Duplicate detection: Compare normalized URL forms and, where appropriate, meaningful page content. Identical titles alone are not enough to establish duplicate content.
  • Site boundaries: A link to an external domain should be recorded as external rather than silently added to the crawl queue. Subdomains, staging hosts, asset hosts, and authentication areas need explicit rules.
  • Data quality: Run checks for missing fields, duplicates, unexpected types, and changed values before analyzing results. The data validation guide can help turn crawl output into a dependable dataset.

Common mistakes

Crawling without a limit can generate an uncontrolled request pattern through calendars, filters, search pages, and session parameters. Start small, inspect the queue, and expand only when the URL rules are understood.

Using one label for every problem makes prioritization difficult. “Indexability issue” could mean noindex, a blocked request, a canonical conflict, or a server error. Preserve the specific signal and use broader categories only for reporting.

Relying on status code alone misses empty titles, incorrect canonicals, orphan candidates, and pages that return a successful response but contain an unexpected template. Combine transport data with parsed HTML and link-graph data.

Ignoring redirects during comparison makes migrations appear cleaner than they are. Keep requested and final URLs, redirect count, and redirect targets in every crawl export.

Changing several crawl rules at once prevents meaningful comparison. Version the configuration, record the crawl date, and note changes to URL normalization, rendering, request headers, or allowed domains.

When to revisit

Run the crawl before and after major releases, domain or protocol migrations, navigation changes, CMS template updates, and large content reorganizations. A smaller recurring crawl can monitor important page types between larger audits. Revisit the workflow before seasonal planning cycles if landing pages, filters, or campaign parameters are introduced.

Update the crawler when the site’s rendering model, URL structure, localization setup, or access policy changes. Also review parser behavior and chosen thresholds when your reporting requirements change. Keep a dated baseline so you can compare status codes, canonical targets, title coverage, indexability signals, and internal-link depth over time.

Your next practical step is to define the crawl scope in writing, create the output schema, run a deliberately small sample, and inspect ten to twenty records manually. Fix normalization and parsing problems before increasing the limit. After each implementation cycle, rerun the same configuration, compare the issue categories, and verify that fixes changed the underlying evidence rather than only the report wording.

Related Topics

#technical SEO#site audits#Python#indexability#crawling
W

Web Tools Lab

Editorial Team

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.