Website Crawl Comparison Tool helps you compare two website crawl snapshots and identify meaningful changes between them. Instead of reviewing two crawl datasets manually, you can compare URL-level results for added and removed URLs, HTTP status, canonical URLs, page titles, HTML size, response time, fetch errors, and unchanged pages. This makes before-and-after technical SEO analysis easier after migrations, redesigns, URL changes, content updates, or other site-wide changes.
Website Crawl Comparison Tool
Compare two crawl snapshots to find added, removed and changed URLs plus status, canonical, title, size and response-time changes.
Website crawl changes
URL-level differences and selected source-level changes between Crawl A and Crawl B.
| URL | Change | Before | After |
|---|
This comparison is a diagnostic crawl snapshot, not a full browser crawl. It checks up to 50 public URLs per side, does not execute target JavaScript, and does not claim that every URL on a site was discovered.
What Is a Website Crawl Comparison Tool?
A website crawl comparison tool compares two snapshots of a website’s URLs and highlights what changed between them.
The important distinction is that the tool does not simply tell you whether a URL exists. It compares the information collected for the same URL across two crawl runs so you can investigate changes that may otherwise be difficult to spot.
For example, a comparison can help identify:
- URLs that were added between Crawl A and Crawl B
- URLs that disappeared between the two crawls
- HTTP status changes
- Canonical URL changes
- Page title changes
- HTML-size changes
- Response-time changes
- Fetch errors
- URLs whose tracked values did not materially change
This is particularly useful when you need to understand what happened after a technical SEO change rather than only inspecting the current state of a website.
What Does PKCapra’s Website Crawl Comparison Tool Check?
The current PKCapra implementation compares two crawl snapshots and reports several URL-level signals.
Added URLs
An added URL is present in Crawl B but not Crawl A.
For example:
Crawl A:https://example.com/about/
Crawl B:https://example.com/about/https://example.com/services/
The /services/ URL would appear as an added URL.
Added URLs can be useful when validating:
- New pages
- New landing pages
- Newly published content
- Newly discovered sitemap URLs
- Site restructures
- New sections introduced during a redesign
An added URL is not automatically a problem. It is a change that should be understood in the context of the work performed between the two crawls.
Removed URLs
A removed URL existed in Crawl A but is absent from Crawl B.
For example:
Crawl A:https://example.com/contact/
Crawl B:https://example.com/about/
The /contact/ URL would be reported as removed.
Removed URLs deserve attention when they were expected to remain available. A missing URL could represent an intentional deletion, migration, consolidation, redirect strategy, sitemap change, or an accidental loss of a page.
For deeper URL-status investigation, use the HTTP Status Checker or Redirect Checker.
HTTP Status Changes
The comparison can identify changes in the recorded HTTP status between crawls.
Examples include:
200→404200→301301→200200→500
Status changes can indicate important changes in availability, redirects, migrations, or server behavior.
A status change should always be interpreted in context. A 200 to 301 change may be intentional during a URL migration, while a 200 to 404 change may require investigation if the page was expected to remain available.
You can investigate individual responses further with the HTTP Status Checker and review broader response-code patterns with the Status Code Distribution Analyzer.
Canonical URL Changes
The comparison can identify changes in the canonical URL recorded for a page.
Canonical changes may occur after:
- URL migrations
- Template changes
- SEO plugin configuration changes
- Duplicate-content consolidation
- Domain or protocol changes
- Changes to canonical tags
Canonicalization is an important part of technical SEO because search engines use multiple signals when selecting representative URLs. A declared canonical is a signal rather than an absolute guarantee that a search engine will select that URL.
If a canonical value changes unexpectedly, investigate the page and compare it with your intended URL structure. You can use the Canonical URL Checker for a more focused canonical inspection.
Title Changes
The tool can identify changes in page titles between the two crawl snapshots.
Title changes can occur after:
- SEO template changes
- CMS updates
- Bulk metadata edits
- Content migrations
- Category or product template changes
- Manual optimization
A title change is not inherently an error. The purpose of the comparison is to make the change visible so that you can determine whether it was intentional.
For individual title analysis, use the Title Tag Analyzer.
HTML Size Changes
The comparison also tracks changes in the recorded HTML size.
A significant change can occur because of:
- Additional page content
- Removed content
- Template modifications
- New inline markup
- Larger embedded data
- Technical implementation changes
HTML-size changes should be treated as diagnostic signals rather than automatic SEO errors. A larger page is not necessarily worse, and a smaller page is not necessarily better.
If a large change needs further investigation, the Large Page Analyzer can provide a more detailed page-level analysis.
Response-Time Changes
The comparison records response-time changes between the two crawl snapshots.
For example:
| URL | Crawl A | Crawl B | Change |
|---|---|---|---|
/ | 145.5 ms | 4.2 ms | Changed |
/about/ | 135.5 ms | 4.7 ms | Changed |
Response time can vary because of server load, caching, network conditions, infrastructure changes, and other environmental factors. Therefore, a single change should not automatically be treated as proof of a persistent performance problem.
For broader page-performance investigation, use the Page Speed Analyzer.
Fetch Errors
Fetch errors are separated from normal URL comparison results.
This helps distinguish a URL that was successfully analyzed from a URL that could not be retrieved during the crawl.
A fetch error can result from temporary availability, networking, server behavior, or other retrieval problems. Investigate the affected URL before treating it as a permanent website change.
Unchanged URLs
URLs that remain present and do not show tracked changes can be identified as unchanged.
This is useful because a large crawl comparison can contain many URLs that require no immediate investigation. Separating unchanged URLs from changed URLs helps focus attention on the parts of the site that actually require review.
How Does the Website Crawl Comparison Tool Work?
The current PKCapra tool accepts two crawl snapshots.
Each crawl can be supplied using:
- A sitemap URL
- A sitemap index URL
- A newline-separated list of public URLs
The tool processes the two datasets and compares their URL-level results.
For the current implementation, each crawl is limited to a maximum of 50 URLs.
The comparison then categorizes the results into meaningful change groups such as:
- Added
- Removed
- Status changes
- Canonical changes
- Title changes
- HTML-size changes
- Response-time changes
- Fetch errors
- Unchanged URLs
The purpose is not to reproduce the internal crawl system of Google or another search engine. It is a practical before-and-after comparison of the data collected by the PKCapra tool.
How to Use the Website Crawl Comparison Tool
Step 1: Prepare Crawl A
Enter the first crawl source.
You can provide a sitemap URL, sitemap index URL, or a list of public URLs separated by new lines.
Crawl A normally represents the earlier state of the website.
For example:
Step 2: Prepare Crawl B
Enter the second crawl source.
Crawl B normally represents the newer state of the website.
For example:
Step 3: Compare the Crawls
Run the comparison.
PKCapra processes the two crawl sources and presents a summary of detected changes.
Step 4: Review the Summary
Start with the high-level totals.
The summary helps you see whether the crawl contains:
- Added URLs
- Removed URLs
- Status changes
- Canonical changes
- Title changes
- HTML-size changes
- Response-time changes
- Fetch errors
- Unchanged URLs
Step 5: Review Individual URLs
After reviewing the summary, inspect the detailed URL-level results.
Look for unexpected changes first.
For example, if a migration was supposed to preserve 100 existing URLs but several URLs appear as removed, those URLs deserve investigation.
How to Interpret Added and Removed URLs
Added and removed URLs are especially useful during migrations.
Consider this example:
| URL | Crawl A | Crawl B | Interpretation |
|---|---|---|---|
/old-service/ | 200 | — | Removed |
/new-service/ | — | 200 | Added |
This could represent an intentional replacement.
However, if /old-service/ was supposed to redirect to /new-service/, the next step should be to verify the redirect behavior rather than assuming the migration is complete.
The Redirect Checker can help investigate the redirect itself.
Using Crawl Comparison for a Website Migration
A crawl comparison can be useful before and after:
- Domain migrations
- HTTP-to-HTTPS migrations
- URL restructuring
- Website redesigns
- CMS migrations
- Major template changes
- Content consolidation
- Taxonomy restructuring
A practical migration workflow is:
- Capture a baseline crawl before the migration.
- Complete the migration.
- Crawl the new site.
- Compare the two snapshots.
- Review removed URLs.
- Review HTTP status changes.
- Review canonical changes.
- Review title changes.
- Review significant HTML-size changes.
- Investigate response-time changes and fetch errors.
- Validate important URLs individually.
The HTTP/HTTPS Redirect Checker can be useful when the migration involves protocol changes.
Using Crawl Comparison After an SEO Change
Crawl comparison is also useful when making bulk SEO changes.
For example, if page titles are updated across a large section of a website, compare a crawl taken before the update with one taken afterward.
You can then identify which URLs changed their titles and investigate unexpected changes.
The same approach can be used for canonical changes or changes caused by templates and SEO configuration.
Using Crawl Comparison With Sitemaps
Sitemaps can provide a practical source of URLs for crawl comparisons.
Before comparing two sitemap-based crawls, make sure you understand what each sitemap represents. A sitemap can change because URLs were added, removed, redirected, consolidated, or excluded from the sitemap generation process.
The XML Sitemap Analyzer can help inspect sitemap structure and URL information before you use a sitemap as a crawl source.
For sitemap index structures, the Sitemap Index Checker can be used for a more focused check.
Crawl Comparison vs. a Single Crawl
A single crawl answers a question such as:
What does the website look like right now?
A crawl comparison answers a different question:
What changed between these two website states?
Both perspectives are useful.
A current crawl is useful for discovering present technical issues. A comparison is useful when the important question is whether something changed after a release, migration, optimization, or configuration update.
For current-state investigation, the Indexability & Crawlability Analyzer can provide a more focused URL analysis.
Common Problems a Crawl Comparison Can Reveal
Unexpected URL Removal
A page disappears from the newer crawl even though it was expected to remain available.
Investigate:
- Whether the page was intentionally deleted
- Whether the URL changed
- Whether a redirect was added
- Whether the sitemap changed
- Whether the page is still linked internally
- Whether the page now returns an error
Unexpected Status Change
A URL that previously returned 200 now returns another status.
Investigate the current response and redirect behavior.
Unexpected Canonical Change
A canonical URL changes after a template or SEO configuration update.
Compare the new canonical with the intended canonical structure.
Unexpected Title Change
A page title changes after a bulk template or metadata update.
Review whether the new title was intentional and whether the affected page type was supposed to change.
Unexpected HTML-Size Change
A page becomes substantially larger or smaller.
Investigate template changes, content changes, embedded data, and other implementation differences.
Response-Time Variation
The same URL reports a different response time between crawls.
Do not treat a single measurement as conclusive evidence of a persistent performance problem. Compare patterns and investigate infrastructure or caching changes when the difference is meaningful.
A Practical Before-and-After SEO Audit Workflow
For a technical SEO change, use this sequence:
Before the Change
Create a baseline crawl and preserve its results.
Record what you are changing and why.
After the Change
Run another crawl using the same general URL source where practical.
Consistency makes the comparison easier to interpret.
Compare the Two Snapshots
Review the summary first.
Then investigate:
- Removed URLs
- Added URLs
- HTTP status changes
- Canonical changes
- Title changes
- HTML-size changes
- Response-time changes
- Fetch errors
Validate Unexpected Findings
Do not assume every difference is an error.
Check whether the change matches the intended implementation.
For example, a URL changing from 200 to 301 may be expected during a migration, while a previously valid page changing to 404 may require additional investigation.
How to Investigate Unexpected Changes
Use the comparison report to identify the affected URLs, then move to a more focused diagnostic tool.
For HTTP behavior, use the HTTP Status Checker.
For redirects, use the Redirect Checker.
For canonical problems, use the Canonical URL Checker.
For indexability and crawlability questions, use the Indexability & Crawlability Analyzer.
For JavaScript-related rendering investigation, use the JavaScript Rendering SEO Checker.
For unusually large HTML pages, use the Large Page Analyzer.
For crawl-related performance diagnostics, use the Crawl Budget Analyzer.
Important Limitations
The Website Crawl Comparison Tool is a comparison and diagnostic tool, not a complete search-engine simulation.
Important limitations include:
- Each crawl is limited to 50 URLs in the current implementation.
- The tool compares the data collected in its two crawl snapshots.
- Response-time measurements can vary because of network, server, caching, and environmental conditions.
- A detected change does not automatically mean that the change is an SEO error.
- Added or removed URLs must be interpreted according to the intended website changes.
- The tool does not reproduce Google’s private crawling, indexing, ranking, or canonical-selection systems.
- A canonical value reported by a page should not be interpreted as a guarantee of the canonical URL selected by a search engine.
- Technical differences should be validated against the actual website before corrective changes are made.
Privacy and Processing
The Website Crawl Comparison Tool processes the crawl sources and public URLs required to perform the comparison.
When using public URL sources, the URLs need to be accessible to the tool’s server-side fetching process. Do not submit private, authenticated, internal-only, or confidential URLs unless you understand the processing implications and have permission to expose them to the requested processing environment.
For general information about how PKCapra handles data, review the Privacy Policy.
Frequently Asked Questions
What is a website crawl comparison?
It is a before-and-after comparison of two website crawl snapshots. It helps identify URLs and technical signals that changed between the two crawls.
What can PKCapra compare between two crawls?
The current tool compares URL presence and several recorded signals, including HTTP status, canonical URL, page title, HTML size, response time, and fetch errors. It also identifies unchanged URLs.
Can I compare two sitemaps?
Yes. The current implementation accepts a sitemap URL or sitemap index URL as a crawl source, subject to the tool’s current crawl limit.
Can I paste a list of URLs?
Yes. You can provide newline-separated public URLs.
How many URLs can each crawl contain?
The current implementation supports up to 50 URLs per crawl.
Does a removed URL always mean an SEO problem?
No. A removed URL may be intentional, particularly during migrations, content consolidation, or URL restructuring. The important question is whether its removal matches the intended website change.
Is a response-time change automatically a performance problem?
No. Response time can vary between requests because of server load, caching, network conditions, and other factors. Treat the result as a diagnostic signal and investigate meaningful or repeated changes.
Does the tool tell me what Google will index?
No. It compares crawl data collected by PKCapra. It does not reproduce Google’s private indexing, ranking, or crawling systems.
Should I run a crawl comparison after a website migration?
A before-and-after comparison can be useful for identifying unexpected URL, status, canonical, title, size, response-time, and fetch changes after a migration.
What should I do when I find an unexpected change?
First determine whether the change was intentional. Then inspect the affected URL with an appropriate diagnostic tool such as the HTTP Status Checker, Redirect Checker, or Canonical URL Checker.
What other PKCapra tools can I use with crawl comparison?
For a broader technical SEO workflow, useful related tools include the XML Sitemap Analyzer, Indexability & Crawlability Analyzer, Status Code Distribution Analyzer, Large Page Analyzer, JavaScript Rendering SEO Checker, and Page Speed Analyzer.
When Should You Use a Website Crawl Comparison Tool?
Use crawl comparison when the key question is not simply whether a website has technical issues, but whether something changed.
It is especially useful after:
- Website migrations
- Redesigns
- CMS changes
- URL restructuring
- Large-scale metadata updates
- Canonical configuration changes
- Content consolidation
- Sitemap changes
- Template updates
- Technical SEO releases
A consistent before-and-after crawl gives you a practical way to separate expected changes from unexpected ones and provides a focused starting point for deeper investigation.