Social previews often fail for small, invisible reasons: a missing image URL, a relative path, a blocked page, or a template that emits conflicting metadata. This checklist explains how to crawl Open Graph and Twitter Card tags across a site, normalize the results, identify meaningful issues, and create an exportable repair queue for technical SEO reviews, launches, migrations, and recurring audits.
Overview
Open Graph and Twitter Card metadata gives social platforms and link-preview systems structured instructions for representing a page. The page title, description, image, canonical URL, and content type are usually the most important fields to inspect. A useful audit does more than search for a few tags in the HTML: it connects each metadata result to the requested URL, final response status, redirect chain, indexability signals, and page template.
Start by defining the crawl scope. Decide whether the audit covers the whole domain, selected folders, an XML sitemap, or a list of campaign and landing pages. Record the source of every URL so that an issue can be traced back to a sitemap, internal link, redirect, or manual test list. Respect the site’s crawl rules, use a sensible request rate, and avoid treating a failed request as a metadata failure until you know whether the server returned an error, a challenge page, or a timeout.
For each URL, collect at least these fields:
- Requested URL and final URL after redirects.
- HTTP status, content type, response time, and crawl timestamp.
- Open Graph title, description, URL, type, image URL, image dimensions when available, and image alternative text.
- Twitter Card type, title, description, image, and any site or creator attribution fields used by the implementation.
- Page title, meta description, canonical URL, robots directives, and a short error message when parsing fails.
Keep raw values as well as normalized values. Raw HTML helps investigate a template problem; normalized values make it possible to group missing, duplicated, invalid, or inconsistent fields in a spreadsheet or dashboard.
Checklist by scenario
Before a launch or redesign
- Crawl representative templates, including articles, products, categories, home pages, search results, and campaign pages.
- Confirm that production URLs, not staging URLs, appear in canonical and social URL fields.
- Check that every important page has one clear title, description, image, and page URL.
- Verify that images resolve over HTTPS and are not accidentally served from a private, temporary, or development host.
- Compare server-rendered HTML with what a browser displays. If metadata is added only after client-side JavaScript runs, test whether the target preview system can access it reliably.
After a migration or URL change
- Crawl both old and new URL lists and record the final destination for each old URL.
- Look for social URL values that still reference the previous domain, path, protocol, or language version.
- Check redirected pages separately from final pages. A redirect can be technically correct while the destination still has incomplete metadata.
- Compare duplicate groups before and after the migration. A sudden increase in identical titles, descriptions, or images may indicate a fallback template.
For a recurring technical SEO audit
- Use a stable URL set so that changes can be compared over time.
- Prioritize indexable, internally linked, high-value URLs before low-value parameter variations.
- Export first-seen and last-seen values for important fields to detect regressions.
- Review a sample of successful pages manually instead of assuming that a complete tag set guarantees a correct preview.
For debugging one broken preview
- Fetch the exact public URL, including its protocol, hostname, path, and query string.
- Inspect the initial HTML response and the final redirected document.
- Check for duplicate tags, malformed HTML, unexpected character encoding, and tags placed outside the document head.
- Open the image URL directly and verify its status, content type, dimensions, and accessibility from outside the local network.
- Compare the result with a known-good page using the same template.
What to double-check
URL normalization: Normalize hostnames, trailing slashes, default ports, fragments, and obvious tracking parameters before grouping results. Do not discard meaningful query parameters without documenting the rule. Store both the original value and the normalized value so that a cleanup decision remains reversible.
Required versus recommended fields: Define a severity model before reporting issues. A missing social title or image on an important shareable page is usually more actionable than a missing optional attribution field. Mark fields as required, conditionally required, or informational according to the site’s templates and publishing goals.
Length checks: Flag unusually short or long titles and descriptions for review, but do not treat a character threshold as an absolute rendering rule. Display behavior varies by platform and context. The audit should identify candidates for editing, not promise an exact preview.
Image checks: Validate that image URLs are absolute, return an image content type, and do not lead to a login page, HTML error document, or redirect loop. Record dimensions where possible and establish image guidance for the site’s design system. A technically reachable image may still be unsuitable if its subject is cropped, text is unreadable, or its aspect ratio does not fit the intended card.
Duplicates and conflicts: Find pages with repeated values across a large set, but interpret duplication in context. A shared image may be intentional for a category, while identical titles and descriptions across unrelated articles often suggest a template or CMS fallback. Also check conflicts between Open Graph and Twitter Card values. A platform-specific value may override a general one, so document which field the implementation intends to control.
Status and indexability: Separate successful HTML pages from redirects, client errors, server errors, blocked responses, and non-HTML resources. A missing tag on a PDF is not the same issue as a missing tag on an indexable article. For a broader crawl model, the technical SEO crawling workflow provides a useful way to combine status codes, canonicals, titles, and indexability signals.
Validation and reproducibility: Save the crawl date, user agent, request method, timeout, and parser version. Use fixtures or captured HTML for important templates so that fixes can be tested without repeatedly crawling production. The guide to testing web scrapers with Python covers fixtures, golden files, and CI checks that can support this process.
Common mistakes
- Checking only the home page: Social metadata is often generated by page type, so one successful page does not validate the whole site.
- Reading only visible page content: The browser view may look correct while the initial HTML contains missing or stale tags.
- Counting tags without checking values: An empty content attribute, relative URL, placeholder image, or escaped template variable should be reported as invalid, not complete.
- Ignoring redirects: Auditing only the requested URL can hide the metadata that users and crawlers receive at the final destination.
- Treating every duplicate as an error: Group duplicates by template, page type, and business purpose before assigning priority.
- Using a single pass as proof of failure: Temporary timeouts, rate limits, and server errors can create false positives. Retry carefully and retain the original response evidence.
- Assuming a validator replaces a real crawl: A browser-based checker is useful for investigation, but a site-wide audit needs URL discovery, normalization, exports, and change tracking.
If your audit also extracts JSON-LD or other structured data, keep those checks separate from social metadata while sharing the same URL, status, and template dimensions. The guide to extracting structured data from web pages explains how these formats can be collected together without treating them as interchangeable.
When to revisit
Run a focused check before a major launch, seasonal campaign, URL migration, CMS release, or template change. Revisit the full crawl when the site’s navigation, rendering method, image pipeline, or publishing workflow changes. A recurring audit is also useful after a period of content production, because new pages may inherit incomplete defaults even when older templates remain correct.
Refresh the audit rules when social-platform documentation, image guidance, parser behavior, or the site’s own metadata conventions change. Keep a small regression set of representative URLs and compare it after every relevant deployment. For each issue, export the URL, field, observed value, expected condition, severity, owner, and verification date. After fixes are released, recrawl the affected URLs and confirm both the metadata and the final response behavior.
The practical next step is to create a crawl table with one row per URL and one column per metadata field, then sort by status errors, missing required fields, invalid images, conflicting URLs, and high-value pages. This turns an Open Graph audit tool or Twitter Card tag checker from a one-time diagnostic into a repeatable technical SEO workflow.