Cloudflare tracks Anthropic’s Claude crawl-to-refer ratio
A discussion in San Francisco about “ethical” AI providers has highlighted the growing tension between AI companies and website publishers.
The debate centers on how AI companies harvest web data to train their models and power their chatbots, often without sending traffic back to the original content sources.
Measuring the imbalance with a crawl-to-refer ratio
For years, the web has operated on an unwritten agreement: websites allow search engine bots to crawl their content in exchange for referral traffic, which drives users and revenue. Generative AI chatbots disrupt this model by providing direct answers, reducing the need for users to visit the source website.
To quantify this shift, Cloudflare, which supports about 20% of the world’s websites, has started tracking a “crawl-to-refer ratio.” This metric compares how many times a company’s bots access a website for data against the number of human users it refers back to that site. A high ratio indicates a company is taking far more data than the value it returns in traffic.
How different AI companies compare
Data from the first week of September revealed significant differences between companies. Anthropic, the maker of the Claude chatbot, showed a particularly high crawl-to-refer ratio. In response to the findings, Anthropic said it could not confirm Cloudflare’s figures and noted that a new web search feature launched earlier this year is generating a rapidly growing amount of referral traffic. OpenAI did not respond to requests for comment. Perplexity, another AI answer engine, provided a detailed statement on the matter.
In the case of public content, publishers can choose not to make their content public. In the case of facts, copyright law, as you know, has always drawn a line between facts and expression. That’s a foundation of human inquiry itself.
A methodological note states that these ratios only track web activity and exclude traffic from native apps, which could lower the overall numbers. However, the methodology is applied consistently to all companies.
The impact on website owners and Google’s changing role
This large-scale data collection has direct costs for website owners. A Business Insider report from about a year ago noted that crawling from Anthropic and OpenAI bots was causing significant increases in traffic costs for some sites, with one developer reporting their client’s cloud-computing bills had doubled.
Google’s crawl-to-refer ratio is currently lower than many AI-first companies, largely because its traditional search results still link out to websites. However, as Google integrates more direct AI answers through features like AI Overviews, its ratio is fluctuating. Cloudflare data showed Google’s ratio rose from 3.3:1 in January to 18:1 in April, before settling at 9:1 in July. Google has stated it remains committed to sending traffic to the web.
Featured image credit
FAQs
Frequently Asked Questions
What is a Premium Domain Name? A premium domain name is the digital equivalent of prime real estate. It’s a short, catchy, and highly desirable web address that can significantly boost your brand's impact. These exclusive domains are already owned but available for purchase, offering you a shortcut to a powerful online presence. Why Choose a Premium Domain? Instant Brand Boost: Premium domains are like instant credibility boosters. They command attention, inspire trust, and make your business look established from day one. Memorable and Magnetic: Short, sweet, and unforgettable - these domains stick in people's minds. This means more visitors, better recall, and ultimately, more business. Outshine the Competition: In a crowded digital world, a premium domain is your secret weapon. Stand out, get noticed, and leave a lasting impression. Smart Investment: Premium domains often appreciate in value, just like a well-chosen piece of property. Own a piece of the digital world that could pay dividends. What Sets Premium Domains Apart? Unlike ordinary domain names, premium domains are carefully crafted to be exceptional. They are shorter, more memorable, and often include valuable keywords. Plus, they often come with a built-in advantage: established online presence and search engine visibility. How Much Does a Premium Domain Cost? The price tag for a premium domain depends on its desirability. While they cost more than standard domains, the investment can be game-changing. Think of it as an upfront cost for a long-term return. BrandBucket offers transparent pricing, so you know exactly what you're getting. Premium Domains: Worth the Investment? Absolutely! A premium domain is more than just a website address; it's a strategic asset. By choosing the right premium domain, you're investing in your brand's future and setting yourself up for long-term success. What Are the Costs Associated with a Premium Domain? While the initial purchase price of a premium domain is typically higher than a standard domain, the annual renewal fees are usually the same. Additionally, you may incur transfer fees if you decide to sell or move the domain to a different registrar. Can I Negotiate the Price of a Premium Domain?
In some cases, it may be possible to negotiate the price of a premium domain. However, the success of negotiations depends on factors such as the domain's demand, the seller's willingness to negotiate, and the overall market conditions. At BrandBucket, we offer transparent, upfront pricing, but if you see a name that you like and wish to discuss price, please reach out to our sales team.
How Do I Transfer a Premium Domain?
Transferring a premium domain involves a few steps, including unlocking the domain, obtaining an authorization code from the current registrar, and initiating the transfer with the new registrar. Many domain name marketplaces, including BrandBucket, offer assistance with the transfer process.
