Methodology

What we fetch, and what we refuse to guess

A tool that reports a number it cannot justify is worse than no tool. Here is every input, every source, and every deliberate omission.

What we fetch

The index checker makes at most four kinds of request, all to the host you name or to its www/apex twin:

  • The page you paste. If you leave the scheme off, we try https:// and say so in the report. We never change http to https on a URL that has a scheme, and we never strip www — those are differences you may be asking us about.
  • /robots.txt on that host.
  • /sitemap.xml on that host, as a HEAD request, retried once as a GET if the server refuses HEAD.
  • The www or apex twin of that host, as a HEAD request, retried once as a GET if the server refuses HEAD. In neither case is its redirect followed — the redirect itself is the answer.
  • Nothing else. No crawl of your site, no third-party API, no lookup of your domain anywhere.

The robots.txt tester fetches only /robots.txt, or nothing at all when you paste the file.

Requests identify themselves as AdvancePreviewTools/0.1. Every fetch is capped in size and time, redirects are followed at most three times, and each hop is re-checked so a public URL cannot redirect us somewhere private. A single index check is capped at 13 outbound requests in total, redirect hops included; past that, the remaining probes are reported as not run rather than guessed.

What we store

Nothing about the sites you check. Results are computed per request and returned; there is no database of audited URLs. Result pages carry X-Robots-Tag: noindex so they never enter a search index.

Where the rules come from

The robots.txt matcher implements RFC 9309 and follows Google's documented behaviour where the two differ — longest-match precedence, Allow winning an exact tie, * and $ wildcards, and the 500 KiB parse limit.

The crawler registry lists 87 tokens. 69 are robots.txt tokens the operator publishes on its own page; they are marked official, and the row's respects_robots value records what that page says about robots.txt — undocumented when it says nothing. The other 18 are marked unofficial because the operator publishes no robots.txt token for them: the string is the crawler's observed user-agent, and what we say about its behaviour rests on the operator's general statements (Google, for instance, says its user-triggered fetchers ignore robots.txt) or on third-party observation — the registry notes say which, so you can weigh each row accordingly. Where an operator documents only partial compliance, or nothing, the tester reports Uncertain rather than Blocked. The registry as a whole was last updated 2026-09-02. Download it — CC BY 4.0.

What we deliberately do not measure

  • Domain authority, backlink counts, traffic estimates, keyword volume. These require a proprietary crawl index we do not have. A heuristic guess would be falsifiable by anyone with a free Ahrefs account, so we publish nothing rather than publish a wrong number.
  • A composite score out of 100. Scores hide which specific thing is broken, which is the only part that helps.
  • Whether an AI model has "learned" your site. Not observable from outside. We report crawler access, which is observable, and stop there.

Corrections

If a row in the registry is wrong, it is a bug and we want to know. The dataset is versioned and every entry links to its source.