Free Googlebot, Bingbot & 15 AI crawlers · Exact matching line
Robots.txt tester: see exactly which rule blocks which bot
Test any URL against live or edited robots.txt for Googlebot, Bingbot and AI crawlers. See allowed or blocked, the exact deciding line, warnings and a clean-file generator.
WordPress: /wp-sitemap.xml (core) or /sitemap_index.xml (Yoast, Rank Math).
Upload it as robots.txt in your site root (WordPress: an SEO plugin's file editor, or your host's file manager). Blocking training crawlers does not remove you from AI search answers.
Robots.txt test results
Shared report. These results were saved when the check ran. Run it again below for live results, or test your own site.
· · lines · groups
Test results
| Path | Crawler | Result | Matching rule |
|---|---|---|---|
| |
Warnings & syntax found
Search & AI crawler overview for the first path
| Crawler | Type | Status | Deciding rule |
|---|---|---|---|
Sitemaps declared
- Line
View the file with matching lines highlighted
Migrating, or worried a rule is costing you traffic? I'll check your robots.txt, sitemaps and crawl stats together on a free call.
Book a strategy callA robots.txt tester that shows its working
This tool tells you whether a crawler may fetch a URL under your robots.txt and which line decided it. It follows the robots exclusion standard (RFC 9309) the way Google applies it: the crawler uses the group that names it (or the * group), the longest matching rule wins, Allow wins a tie, and * and $ wildcards work as Google documents. Google retired the robots.txt tester in Search Console in 2023, and this is a free replacement that also covers AI crawlers.
How to use it
- Test live robots.txt: enter your domain. We fetch
/robots.txtlike a crawler, following up to five redirects. - Paste & edit: paste a draft file, change rules and re-test before you publish. Pasted files are never saved.
- Add the URLs or paths you care about, one per line, such as a product page,
/wp-admin/or a search URL with parameters. - Pick the crawlers to test. The overview table always shows all 17 for the first path.
- Generator: tick the policy you want (allow AI search, block AI training, WordPress defaults) and copy the result.
How to read the results
Each test shows Allowed or Blocked, the rule that matched (for example Disallow: /search, line 17) and the group it came from. “No matching rule” means the crawler’s group exists but nothing matches the path, so it is allowed. If no group applies and there is no * group, everything is allowed.
The file’s HTTP status matters too. A 404 (or any 4xx) means “no restrictions”. A 5xx server error or an unreachable file is far worse: Google stops crawling the site for the first 12 hours, then relies on the last good copy of the file for up to 30 days, so an outage can quietly stall crawling. The tester flags both, along with robots.txt served as HTML, files over Google’s 500 KiB limit, a UTF-8 byte-order mark, misspelled directives, rules placed before any User-agent line, Crawl-delay (Google ignores it) and Noindex in robots.txt (unsupported since 2019).
Search crawlers vs AI crawlers
Blocking a crawler in robots.txt stops it fetching the page; it doesn’t remove a URL that is already indexed (use noindex for that). For AI, separate the jobs: OAI-SearchBot, PerplexityBot and Claude-SearchBot power AI search answers and citations; GPTBot, ClaudeBot, CCBot and the Google-Extended and Applebot-Extended tokens relate to model training. You can block training and still be cited. Google-Extended doesn’t affect Google Search or AI Overviews, which use Googlebot.
Some crawlers inherit rules: Googlebot-Image falls back to the Googlebot group when it has no group of its own. Tokens are matched case-insensitively, and a version suffix such as Googlebot/2.1 counts as googlebot.
Frequently asked questions
Does Google still have a robots.txt tester?
Search Console now has a robots.txt report that shows the files Google fetched and any parse errors, but you can no longer test a URL against the rules there. This tool fills that gap and adds AI crawlers.
Why is my page blocked when I only disallowed one folder?
Rules are prefix matches. Disallow: /blog also blocks /blog-news/ and /blogging.html. Add a trailing slash (/blog/) or use $ to anchor the end. The tester shows the exact line that matched.
If a crawler has its own group, does it still follow the * rules?
No. A crawler follows only the most specific group that names it. If you create a User-agent: GPTBot group, copy any private-area rules into it too; the generator does this for you.
Can robots.txt remove a page from Google?
No. It stops crawling, not indexing. A blocked URL can still appear in results if other pages link to it. To remove a page, allow crawling and add a noindex meta tag or X-Robots-Tag header.
Where do I edit robots.txt in WordPress?
WordPress serves a virtual robots.txt. SEO plugins such as Yoast and Rank Math include a file editor, or you can upload a physical robots.txt to the site root. Check the result with the redirect and header checker to confirm it returns 200 as text/plain.
Next, run the AI search readiness checker to see crawler access alongside content and entity signals, or the free SEO audit for a full technical check. For migrations or crawl-budget problems, see my technical SEO services or get in touch.
Let’s talk
Find your biggest growth lever
Tell me where you are and where you want to be. I’ll reply with an honest view of what it takes, usually within one business day.
- Free 30-minute strategy call
- Your site reviewed before we speak
- No obligation, no hard sell
Thank you, message received.