It's free, continue with GoogleFree tools · Instant check
Which AI crawlers can read my site?
20 AI bots, one robots.txt.
Type your website. We read its robots.txt and tell you, for each of 20 AI crawlers, whether it may read your site, which rule decides it, and whether your firewall is turning bots away before robots.txt even matters.
Reads your robots.txt and loads your homepage once. Nothing is stored.
The free plan
One check is a clue.
The free plan is the whole picture.
- A full audit of up to 50 pages, every week
- The three fixes that win the most enquiries, with the evidence
- Speed, security, AI visibility, tracking and the enquiry path in one report
- Connect Google Analytics, Search Console and Clarity to see what visitors really do
- One website free, or five client sites for an agency. No card.
Questions people ask
Should I block AI crawlers?
Block the training crawlers if you do not want your content used to train models. Think twice before blocking the search and user agents (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot): they are how AI answers find, cite and link to you.
robots.txt allows them. Why might they still be blocked?
Firewalls and bot protection (Cloudflare, Kinsta, Sucuri and others) often challenge automated visitors before robots.txt is read. We flag it when your site challenged our own honest crawler.
What is a Content-Signal line?
A newer robots.txt line, popularised by Cloudflare, that says whether content may be used for search, AI answers or AI training. We show it when your file has one.
More instant checks
Stop guessing.
See what stops visitors enquiring.
Create your free PostCliq account and get your full website audit in minutes.