Does robots.txt keep my site out of Google's AI Overviews?

Not necessarily. Reporting from Search Engine Journal, citing data from John Shehata, indicates that a commonly used robots.txt setting doesn't function the way many publishers assume when it comes to AI Overviews. The exact mechanics aren't fully detailed in initial coverage, so it's worth reviewing the full report before changing anything.

Who is John Shehata and why does his data matter here?

Shehata's analysis is cited as the basis for this report, and it's described as showing a real cost to newsrooms tied to the robots.txt misunderstanding. The available summary doesn't provide his full methodology or figures.

What should publishers do right now?

Check your current robots.txt configuration against how it actually interacts with Google's AI Overviews rather than assuming it works the same as it does for classic organic search, and read the full source report for the specific technical details before making changes.

Robots.txt Won't Keep You Out Of AI Overviews. Here's What Actually Will.

Why this matters

If you manage a newsroom, content team, or brand publishing site, there's a decent chance someone on your team assumes a standard robots.txt configuration is enough to keep your content out of Google's AI Overviews. According to reporting from Search Engine Journal, that assumption is wrong, and getting it wrong has a real cost.

The reporting, based on analysis from John Shehata, indicates that a widely used robots.txt setting doesn't do what many publishers think it does when it comes to AI Overviews. That gap between assumption and reality is the whole story here, and it's worth a few minutes of your time to check your own setup rather than take your CMS defaults on faith.

What we know so far

Search Engine Journal's report, credited to Shehata's data, centers on a specific misunderstanding: publishers commonly believe a certain robots.txt directive excludes their content from Google's AI Overviews, but the actual mechanics don't line up with that belief. The report frames this as a costly mistake for newsrooms specifically, though the source material available to us doesn't spell out the exact technical setting, the precise mechanism, or the dollar or traffic figures involved.

What is clear from the framing is that this isn't a hypothetical concern. Shehata's data is described as showing what the wrong setting "actually costs newsrooms," which points to measurable impact rather than a theoretical risk. Beyond that, the publicly available excerpt doesn't provide the granular breakdown, so treat any more specific claims about mechanics or numbers with appropriate caution until the full report is reviewed directly.

Why this matters for marketers and publishers

The practical takeaway is straightforward even without every technical detail: don't assume your existing robots.txt file is doing the job you think it's doing with respect to AI Overviews. Crawler directives and AI-driven surfaces in Google's search results don't necessarily follow the same rules that governed classic organic indexing, and conflating the two is apparently common enough that it warranted its own report.

For teams responsible for technical SEO, content distribution, or brand visibility in AI-generated search summaries, this is a prompt to go verify your own configuration rather than rely on inherited defaults or assumptions carried over from traditional search optimization.

What to do next

Given the limited detail available in this initial coverage, the most responsible move for marketers and publishers is to read the full Search Engine Journal report directly before making changes to crawler settings, since the specifics of which directive is involved and how it affects AI Overviews inclusion matter enormously for getting a fix right. Acting on assumptions here could just as easily cause a new problem as solve the current one.