Methodology

What we fetch, and what we refuse to guess

A tool that reports a number it cannot justify is worse than no tool. Here is every input, every source, and every deliberate omission.

What we fetch

  • The exact URL you paste — unmodified. We do not silently upgrade http to https or strip www, because those are differences you may be asking us about.
  • /robots.txt on that same host.
  • Nothing else. No crawl of your site, no third-party API, no lookup of your domain anywhere.

Requests identify themselves as AdvancePreviewTools/0.1. Every fetch is capped in size and time, redirects are followed at most three times, and each hop is re-checked so a public URL cannot redirect us somewhere private.

What we store

Nothing about the sites you check. Results are computed per request and returned; there is no database of audited URLs. Result pages carry X-Robots-Tag: noindex so they never enter a search index.

Where the rules come from

The robots.txt matcher implements RFC 9309 and follows Google's documented behaviour where the two differ — longest-match precedence, Allow winning an exact tie, * and $ wildcards, and the 500 KiB parse limit.

The crawler registry lists 87 tokens. Every row cites the operator's own documentation and carries the date it was last verified. Download it — CC BY 4.0.

What we deliberately do not measure

  • Domain authority, backlink counts, traffic estimates, keyword volume. These require a proprietary crawl index we do not have. A heuristic guess would be falsifiable by anyone with a free Ahrefs account, so we publish nothing rather than publish a wrong number.
  • A composite score out of 100. Scores hide which specific thing is broken, which is the only part that helps.
  • Whether an AI model has "learned" your site. Not observable from outside. We report crawler access, which is observable, and stop there.

Corrections

If a row in the registry is wrong, it is a bug and we want to know. The dataset is versioned and every entry links to its source.