robots.txt tester

Why this is worth checking before you launch

Two failures account for almost every "why is my site invisible" question from a pre-launch founder, and both live in this one file:

• A leftover Disallow: / from a staging template. It blocks everything, and it is invisible unless you look.
• A robots.txt that returns a server error. Google treats a 5xx here as "do not crawl this site at all" — not as "no rules". A flaky server on this one path quietly suppresses the whole domain.

A 404 is fine. You only need a robots.txt if you want to restrict something or point at a sitemap.

Search, AI answers, and AI training are three different decisions

Blocking a training crawler does not remove you from AI answers, and blocking an answer crawler does not remove you from search. They are separate tokens with separate consequences, which is why this tool groups them rather than giving you one number.

The crawlers we check

Registry last updated 2026-09-02. Obeys robots.txt is what the operator documents; official means the operator publishes this robots.txt token at the linked page — what that page says about robots.txt, if anything, is the Obeys robots.txt value beside it; unofficial means the operator publishes no robots.txt token for it — the string is the observed user-agent, and the value rests on what the operator says in general terms or on third-party observation (the notes in the download say which). Download as JSON (CC BY 4.0).

Search index

AI answers

AI training

AI agent

Link preview

SEO tool

Ads

Archive

Other