AI crawlers and robots.txt: which to allow?
GetSeoHit team · Updated:
SHORT ANSWERSplit AI crawlers in two: search crawlers that fetch pages to cite in answers, and crawlers used for model training. Block search crawlers and you can’t be a source in those answers. Blocking training crawlers can be a deliberate choice.
Search versus training crawlers
OAI-SearchBot and PerplexityBot serve search answers. GPTBot and Google-Extended relate to training preferences. Company rules can change, so check their current docs before deciding.
A robots.txt pattern
To stay open to search crawlers and closed to a training crawler, write a separate User-agent group for each bot with an Allow or Disallow rule. Once a bot has its own group, it ignores the general * group.
Common mistakes
Watch out for these:
- Closing the whole site by accident with Disallow: /
- A CDN or firewall blocking the bot regardless of robots.txt
- Staging rules carried over to production
Check yours
Our free AI crawler check reads your robots.txt and shows each bot’s status. It can’t see firewall blocks, which are verified with server logs.