The situation
Your robots.txt has been the same file since a redesign years ago, and nobody currently on the team wrote it, remembers why a rule is there, or knows whether it touches any of today's AI crawlers at all.
Why it's a real problem
You are silently allowing or blocking today's AI bots based on a decision nobody remembers making, and nobody can currently explain. A blanket bot-blocking rule written for a completely different reason - stopping scrapers, fighting a bandwidth spike - can quietly wall off every AI engine's ability to cite you, with the effect only visible once someone finally checks the logs.
What we implement
We audit every bot rule - robots.txt, WAF, and CDN-level blocks - against the current AI crawler list (GPTBot, PerplexityBot, ClaudeBot, Google-Extended and peers), and rebuild it as a deliberate, documented crawler policy: what is allowed, what is blocked, what is shaped, and why.
What you get
- A written record of every allow/block/shape decision, with its reasoning, that outlives any one person's memory or tenure.
- Elimination of the accidental blanket-block pattern that quietly costs citation access without anyone deciding it should.
- A policy your legal and security teams can actually review, because it says what it does and why, instead of being an opaque legacy file.
Illustrative scenario
A company running a five-year-old WAF rule aimed at a long-forgotten scraping incident might discover it has been blocking every current AI crawler on the list, right alongside the traffic it was actually written to stop. (Illustrative scenario - not a measured result.) Whether that specific failure mode exists on your estate is exactly what the audit checks.
Not every block is a mistake: there is a real, defensible case for blocking some AI crawlers on some content - the trade-off between citation access and training-data control is covered honestly in the AI crawlers guide. The goal here is a decision made on purpose, not a rule inherited by accident.