The New York Times and 3 other US papers are suing Brave over a search index of more than 30 billion pages that it sells to AI companies. Brave's crawler has no name of its own, so robots.txt cannot target it.
Dominic Coutts
Most businesses have no idea their content feeds AI answers through indexes they have never heard of. There is no lever to pull on this one, so do not break your Google visibility trying to find one.
What happened
On 2 November the New York Times sued Brave, the browser and search company. The Daily News, Chicago Tribune and Orlando Sentinel filed a separate case the same day. The claim is that Brave copied more than 30 billion web pages without identifying its crawler or giving site owners any way to block it, then sold access to AI companies including Perplexity, Mistral and Cohere. Brave has not responded, so these are allegations rather than findings.
What it means for you
Brave says its crawler follows the same rules as Googlebot and does not use a name of its own. So there is no line you can put in robots.txt that blocks Brave and leaves Google alone. The only control Brave points to is the noindex tag, which also removes the page from Google. Your pages are probably in the index whether or not you have ever used Brave Search, and the stored copies can be out of date. AI tools can then answer using your content without sending anyone to your site.
What to do
- Run a site search for your own domain on Brave Search and see what it holds, and how old the descriptions are.
- List which AI crawlers your robots.txt blocks, so you know what you have actually decided. Brave will not be one of them.
- Do not reach for noindex to keep Brave out. It would take the page out of Google as well, which costs far more than it saves.
Search Engine Journal
Real and worth watching, but nothing to act on yet.
More signals
Get a focused review of how your business is found, where competitors win attention, and what to change to generate more of the right enquiries. You leave with a plan you can act on, whether we do the work that follows or not.