AI SEOAI OS
Free · No sign-up · Instant results

Robots.txt Tester

Check a site’s robots.txt for mistakes and test whether any URL is allowed for Googlebot, Bingbot or AI crawlers like GPTBot.

robots.txt tells crawlers which parts of a site they may visit. One wrong line can hide a whole section from Google, or let AI crawlers in when you meant to keep them out. Enter any page URL to fetch its site’s robots.txt, see which rule applies, and check every common crawler at once.

No need to type https:// — the domain alone works. Nothing is stored. Public sites only.

What this tool checks

File
Whether robots.txt exists, returns 200 as plain text, and stays under Google’s 500 KiB limit.
Syntax
Unknown directives, rules before any User-agent, and paths that don’t start with “/”.
Sitemap
Whether a Sitemap: line points crawlers to your XML sitemap.
URL test
Which group and rule decide this URL, using Google’s matching rules — wildcards, longest match wins, Allow wins a tie.
AI crawlers
The same URL tested for GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Extended.

Frequently asked questions

No. It stops crawling, not indexing — a blocked URL can still appear in results if other sites link to it. Use a noindex tag on a crawlable page to keep it out of search.
Google uses the most specific (longest) matching rule. If an Allow and a Disallow rule are the same length, Allow wins.
Add a group per crawler, e.g. “User-agent: GPTBot” followed by “Disallow: /”. Google-Extended controls use for Google’s AI models without affecting Google Search.
A 404 means “everything allowed”. A 5xx server error makes Google treat the whole site as blocked until it recovers.

Want every page of your site checked automatically?

Start a Free Audit