AI is killing the web economy and marketers are going to suffer
In Marketing We Trust CEO Paul Hewett takes a frank and detailed look at what AI bots are doing to web traffic. He concludes that unless the industry unites, AI will break the web economy.
“People create content for two reasons: to make money or to get famous. If we don’t fix the economics, the internet dies.”
When I stumbled upon Cloudflare CEO Matthew Prince making this stark statement on LinkedIn this week, it immediately stopped me scrolling. It wasn’t just dramatic; it succinctly captured a fundamental truth that every marketer and content creator instinctively knows but rarely states so bluntly.
Prince elaborated further, emphasising the seriousness of the situation: “If we don’t fix this, the internet is going to die. I’m not going to sugar coat it. If people do not have incentives to create content, they are not going to create content. It has to be that you make money or you get famous. Those are the two reasons people create content. That’s it. We have to get that back or the internet dies.”
I’ve spent my career in the trenches of digital marketing, helping global brands build visibility, drive traffic, and convert visitors into revenue. At its core, marketing is fundamentally an economic discipline. We create, measure, optimise, and repeat, all to generate revenue or elevate brand recognition. However, very recent and predictable shifts driven by AI and evolving search behaviours threaten the very foundations of this economic model.
The economics of crawling: unsustainable growth
Ten years ago, there were roughly 4 billion internet users, and Google crawled around two pages of a website for every visitor it sent to that site. Today, with around 6 billion internet users, Google’s crawl rate has ballooned to 18 pages per visitor. More alarming, OpenAI (ChatGPT) now crawls approximately 1,500 pages per visitor, and Anthropic’s crawler rate is an astonishing 60,000 pages per visitor (this because Anthropic is much smaller than OpenAI but still crawls at the same kind of frequency).
These statistics point to a critical imbalance. Publishers bear all the costs of hosting, bandwidth, and infrastructure, while AI platforms extract significant value from this content, often without sending any visitors to the original websites.
Real-world impact on marketing performance
In my work with global brands, I’m consistently seeing a troubling trend: impressions are rising, yet click-through rates are steadily declining. The rapid growth of AI-generated “zero-click” answers, where users find their information directly on search results pages, is a primary driver. Prince estimates that nearly 90% of Google queries today are resolved without a single external click, I actually do not believe that it is that high at this point in time.
Paul Hewett
We’ve already witnessed concerning examples where excessive AI crawling has caused real damage. Read the Docs, a platform supporting software documentation, experienced severe bandwidth overload due to aggressive AI crawlers. In May 2024, one crawler alone downloaded 73 terabytes of zipped HTML files, almost 10 terabytes in a single day, resulting in over $5,000 in bandwidth charges. Another crawler, attributed to Facebook, downloaded an additional 10 terabytes of data in June 2024. Another company, Triplegangers, recently highlighted infrastructure instability directly attributed to Anthropic’s intensive crawling practices. These cases illustrate how increased AI activity can threaten website uptime and reliability, both crucial for audience trust and advertising revenue. These are not isolated cases but indicative of a wider issue affecting publishers and website operators globally.
We are receiving tens of thousands of crawling requests from openai . com/gptbot, which is used to scrape data by OpenAI to train generative AI models. This has caused our website to go down for periods of time over the last few days. Which interrupted our normal operations.… pic.twitter.com/EyKHUi8tWs
— Triplegangers (@triplegangers) January 8, 2025
Traditional metrics and measurement strategies mask the full extent of this shift. Recent reports suggesting platforms like ChatGPT hold less market share in search than niche engines like DuckDuckGo are misleading. Such studies typically focus on explicit web searches, ignoring the vast number of implicit informational queries handled directly by large language models (LLMs).
In reality, tools like ChatGPT have significantly reshaped search behaviour. Users increasingly turn to conversational AI rather than traditional keyword searches. This invisible shift means businesses are missing opportunities, as these interactions are currently untracked by conventional analytics.
Marketing’s strategic blind spot
(Midjourney)
Why should marketers care about crawling and these behind-the-scenes shifts? Because the websites we manage and the content we create underpin brand awareness and revenue generation. When content isn’t adequately rewarded through clicks, visibility, or direct revenue, the motivation to create it weakens.
This is not about resisting AI, far from it. I’m an AI optimist and have never been more excited about the future of search. However, acknowledging and addressing current imbalances is critical. If content creators aren’t fairly rewarded, their incentive to produce valuable content diminishes.
This problem is too significant and systemic for marketers or publishers to address alone. The scale of disruption caused by AI-driven crawling and changing search behaviours requires an industry-wide response. Marketers and content creators need coordinated collective actions supported by market-driven solutions.
Immediate industry-level actions
- Industry unity and commercial standards
The industry must unite, establish fair commercial standards, and collectively advocate for equitable terms with AI platforms. Individual publishers or marketers cannot realistically negotiate with global AI giants alone. A unified stance, represented through trade associations, marketing bodies, and publisher alliances, will provide meaningful negotiation power and ensure fair compensation. - Support emerging market-driven solutions
Solutions like Cloudflare’s AI Labyrinth and it’s now much anticipated AI initiative, alongside those from their competitors, offer infrastructure to manage AI crawling more equitably. Matthew Prince has clearly indicated that significant solutions designed to restore balance are imminent. Publishers and marketers should closely monitor and embrace these market-driven tools as they become available, regaining control over their content’s value.
What marketers and publishers can do now
While individual actions won’t resolve this structural challenge entirely, marketers can proactively adapt to better understand these evolving market dynamics:
- Adopt new measurement strategies
Immediately evolve analytics and measurement frameworks to track user behaviours driven by AI accurately. Traditional metrics are inadequate and hide the real shift in user engagement. - Stay informed and engaged
Actively participate in industry conversations and monitor emerging solutions from infrastructure providers and AI platforms. Quick adoption of new tools will position organisations to benefit as the balance begins to shift.
Ultimately, addressing this issue demands a collective response. The market is already stepping in, exemplified by initiatives like Cloudflare’s forthcoming solution. Marketers and publishers should support these collective efforts and adjust their strategies accordingly.
Acting now for a sustainable future
The decisions marketers make in the coming months will shape the sustainability of content-driven businesses for years to come. I echo Prince’s urgency; without prompt action to rebalance the economics of the web, content creation, and the internet as we know it, face genuine risks.
Yet, despite these challenges, the opportunities ahead are enormous. AI will continue transforming marketing, making it more sophisticated and potentially more rewarding. The key is decisive action now, creating systems that fairly compensate content creators for their essential contributions.
After all, our content is the fuel for this AI-driven future, as acknowledged by OpenAI’s CEO Sam Altman. It’s time we ensured our industry continues to thrive, so creators can still make money, and yes, even get famous.
