Encypher Research · June 2026
Websites are locking the wrong door.
We audited 457 news websites. Most of them block the wrong AI robots.
457
sites audited
4 in 5
let the answer robot in
50.8%
graded F on AI access
Free download
Get the full report
8 pages. Grades for 457 sites, block rates for all 13 crawlers, and what to do this week.
What we found
The doors are half open.
457
big news websites audited: live robots.txt, 12 groups
5 of 13
AI crawlers blocked by the average site
1 in 3
sites block no AI robots at all
4 in 5
let the answer robot walk right in, the one that replaces the click
The big mistake
Guarding the archive. Giving away the audience.
Sites block the robots that learn from old stories. They leave the door open for the robots that write instant answers and keep the reader.
Robots that LEARN from stories
blocked
Robots that ANSWER readers
blocked
94 websites block OpenAI's training bot, then wave its answer bot straight through. That is backwards.
The report card
Only 3 sites earned an A. Over half got an F.
A
B
C
D
F
The A's: MSNBC, The Seattle Times, Publishers Weekly. They block the most robots and post clear rules. Even they cannot make rules follow a story once it is copied. The full report has the whole table: block rates for all 13 crawlers and which kinds of sites protect themselves best.
Why blocking alone fails
robots.txt is a sign on your door.
- 1
Robots can ignore it
Only crawlers that choose to obey the sign will obey it.
- 2
A blocked robot can rename itself
New user agent, same robot. Your sign will not know.
- 3
Stories leak out other ways
Syndication, scrapers, copies. AI reads your story somewhere else.
- 4
It stops at your own domain
Once a story is shared, the sign stays home. And it cannot see how your work gets used.
What actually works
Put a stamp on the story itself.
- 1
Hard to forge
A cryptographic stamp shows who signed the story, and when.
- 2
Travels with the story
It rides along when the next site keeps it. Stripped? We can still find close copies of the text.
- 3
Carries your terms
A machine-readable license tells AI companies what you allow, and where to pay you.
- 4
Gives you evidence
Find marked stories and close copies across the open web. Evidence you can take into a licensing deal.
See how the bots crawl your site.
Run the free AI Cost and Exposure Report. It reads your site, shows which of the 13 doors you left open, and gives a clearly labeled estimate of the money at risk. One minute. No signup.
Encypher in the media
The conversation is already live.
Beeler.Tech · May 2026
Content provenance for premium publishers in the AI era
Why provenance, not access control alone, is how premium publishers stay discoverable and get credited as AI reads the web.
Watch the talk
Cannes Film Festival · May 2026
Future of IP and Media
Main-stage talk on how intellectual property, licensing, and provenance reshape the media business as AI becomes the primary reader of the web.
Watch the talk
Method: we read the public robots.txt file of 457 big news websites in 12 groups on June 26, 2026. 419 of the 457 websites let us read their file. A robot counts as blocked when the file says Disallow at the front page. Full details, the domain list, and the scoring approach are described in the report.