Search bots vs training bots
It separates the two: keep AI search/answer bots (so you can be cited) while optionally opting out of model-training crawlers.
Free tool · 100% in your browser
Robots.txt Builder for AI
Build a robots.txt with explicit, purpose-aware rules for AI crawlers — welcome the search bots, optionally block the training bots, add your sitemap.
What this tool checks
Crawl control, not deindexing
robots.txt manages crawling. To remove a page from search you need a noindex tag the crawler can actually reach.
This tool sees your page from the outside
The free Aetos plugin sees inside your WordPress.
This tool checks what a bot sees from a public URL. The free Aetos edition runs a real audit from inside WordPress — across your pages and WordPress environment — so it catches signals an outside view cannot see.
Free edition, no card. Pro adds additional analysis modules, exports, and the AI fix assistant. Want us to do it for you? Talk to the team → Talk to the team
FAQ
Common questions
Where does robots.txt go? +
At your domain root, so it is reachable at https://yourdomain.com/robots.txt. On WordPress you can edit it via your SEO plugin or upload it by FTP.
Should I block GPTBot? +
Only if you want to opt out of OpenAI model training. It does not affect being cited in ChatGPT search — that is OAI-SearchBot, which you can keep allowed.
Is blocking a bot guaranteed to stop it? +
robots.txt is a directive that well-behaved crawlers honor. It is not an enforcement mechanism — for hard blocks use server rules.