Websites are locking the wrong door.
We read the robots.txt file of 457 news websites. Most block the AI robots that learn from old stories. 4 in 5 let in the robot that writes quick answers and keeps the reader.
Encypher Research, June 2026. The findings are free to read; the 8-page PDF is free with an email.
What we found.
- 457
- Big news websites checked, in 12 groups.
- 5 of 13
- AI robots blocked by the average site.
- 1 in 3
- Sites block no AI robots at all.
- 4 in 5
- Let the answer robot in, the one that replaces the click.
Guarding the archive. Giving away the audience.
Sites block the robots that learn from old stories. They leave the door open for the robots that write instant answers and keep the reader.
94 websites block OpenAI's training robot, then let its answer robot straight through.
Only 3 sites earned an A. Over half got an F.
The A's: MSNBC, The Seattle Times and Publishers Weekly. They block the most robots and post clear rules. Even they cannot make the rules follow a story once it is copied.
robots.txt is a sign on your door.
robots.txtrobots.txt: The file on a website that tells crawlers which pages they may fetch. asks robots to stay out. It cannot make them.
- Robots can ignore it. Only robots that choose to obey the sign will obey it.
- A blocked robot can rename itself. New name, same robot, and your sign will not know.
- Stories leak out other ways: syndication, scrapers and copies. AI reads your story somewhere else.
- It stops at your own site. Once a story is shared, the sign stays home.
Put a label on the story itself.
Hard to forge
A signed label shows who published the story, and when.
Travels with the story
It goes along when the next site keeps it. If it is stripped, close copies of the text can still be found.
Carries your terms
Licensing terms machines can read tell AI companies what you allow and where to pay you.
Get the full report.
8 pages. Grades for 457 sites, block rates for all 13 robots, and what to do this week.
How we measured
Show the technical detailMethod and scope
We read the public robots.txt file of 457 big news websites in 12 groups on June 26, 2026. 419 of the 457 websites let us read their file. A robot counts as blocked when the file says Disallow at the front page. The report describes the full domain list and the scoring approach.
Create. Mark. Endure.
See which AI robots your own site lets in. It takes one minute and needs no signup.