Almost everyone in the "AI" business other than Google, Microsoft, and Brave uses scrapers of Google. Looks like Google is finally fighting back.
Google just made a subtle but massive change Last month, Google quietly removed the num=100 search parameter. This means you can no longer view 100 results at once. The default max is now 10. Why does this matter? - Most LLMs (OpenAI, Perplexity, etc.) rely (directly or indirectly) on Google’s indexed results, alongside their own crawlers. - Overnight, their access to the “long tail” of the internet was cut by 90%. The fallout: - According to Search Engine Land, 88% of sites saw a drop in impressions. - Reddit, which often ranks in positions 11–100, saw its LLM citations plummet. Its stock dropped 15%. For startups, this is brutal. Visibility just got harder. Reddit as part of AEO just changed entirely. It’s no longer enough to build a great product you need to crack distribution first. Because if people can’t discover you, they’ll never get to evaluate you. Most engineers seem to always neglect this reality, but a mediocre product with great distribution will always beat a great product with mediocre distribution. As Peter Thiel says: “Most businesses get zero distribution channels to work: poor sales rather than bad product is the most common cause of failure. If you can get just one distribution channel to work, you have a great business. If you try for several but don’t nail one, you’re finished. Superior sales and distribution by itself can create a monopoly, even with no product differentiation. The converse is not true. No matter how strong your product — even if it easily fits into already established habits and anybody who tries it likes it immediately — you must still support it with a strong distribution plan." Distribution > Product (h/t Adarsh Appaiah on LinkedIn)

Oct 4, 2025 · 5:58 AM UTC

16
26
265
Replying to @BrendanEich
centralized data? agentfi world says no thanks
The Web is still decentralized forming a common good, overlayed on the Internet Protocols which support p2p at lower levels. Network effects make 1st and 2nd place winners, for a time. Crawling is a case in point: Googlebot gets a pass, others face robots.txt quasi-lawfare. 1/2
2
5
This tweet is unavailable
No disagreement, although I’m not going as far as “ethical”. Plus, they have a lot more anti-scraping work ahead.
1
1
Replying to @BrendanEich
Google has an underestimated moat in AI.
Replying to @BrendanEich
Blocking competitors was always inevitable given Google and Microsoft competing on the giga LLM scale Hopefully good news for Brave to grow revenues by filling the search API role for everyone
1
Replying to @BrendanEich
This also means that ~90% of traffic is bots.
1
Replying to @BrendanEich
When garbage in and garbage out are not acceptable it's time to turn to Brave 👍
1
Replying to @BrendanEich
Microsoft has its own search index?
1
1
Replying to @BrendanEich
They’re going to need to build their own purpose-built web indexes, and we’ll probably be better off for it.
Replying to @BrendanEich
And they’ll lose.