Why we’re rethinking cache for the AI period

Cloudflare information exhibits that 32% of site visitors throughout our community originates from automated site visitors. This consists of search engine crawlers, uptime checkers, advert networks — and extra not too long ago, AI assistants seeking to the online so as to add related information to their information bases as they generate responses with retrieval-augmented technology (RAG). Not like typical human habits, AI brokers, crawlers, and scrapers’ automated habits could seem aggressive to the server responding to the requests. 

For example, AI bots regularly difficulty high-volume requests, usually in parallel. Slightly than specializing in standard pages, they could entry not often visited or loosely associated content material throughout a web site, usually in sequential, full scans of the web sites. For instance, an AI assistant producing a response could fetch photos, documentation, and information articles throughout dozens of unrelated sources.

Though Cloudflare already makes it simple to management and restrict automated entry to your content material, many websites could need to serve AI site visitors. For example, an utility developer could need to assure that their developer documentation is up-to-date in foundational AI fashions, an e-commerce web site could need to make sure that product descriptions are a part of LLM search outcomes, or publishers could need to receives a commission for his or her content material by way of mechanisms corresponding to pay per crawl.

Web site operators subsequently face a dichotomy: tune for AI crawlers, or for human site visitors. Given each exhibit extensively totally different site visitors patterns, present cache architectures power operators to decide on one method to avoid wasting assets.

On this submit, we’ll discover how AI site visitors impacts storage cache, describe some challenges related to mitigating this impression, and suggest instructions for the group to contemplate adapting CDN cache to the AI period.

This work is a collaborative effort with a staff of researchers at ETH Zurich. The complete model of this work was revealed on the 2025 Symposium on Cloud Computing as “Rethinking Web Cache Design for the AI Era” by Zhang et al.

Let’s begin with a fast refresher on caching. When a person initiates a request for content material on their system, it’s normally despatched to the Cloudflare information heart closest to them. When the request arrives, we test to see if we now have a sound cached copy. If we do, we will serve the content material instantly, leading to a quick response, and a cheerful person. If the content material is not obtainable to learn from our cache, (a “cache miss”), our information facilities attain out to the origin server to get a contemporary copy, which then stays in our cache till it expires or different information pushes it out. 

Holding the correct parts in our cache is essential for lowering our cache misses and offering an incredible person expertise — however what’s “right” for human site visitors could also be very totally different from what’s proper for AI crawlers!

Right here, we’ll deal with AI crawler site visitors, which has emerged as probably the most lively AI bot sort in current analyses, accounting for 80% of the self-identified AI bot site visitors we see. AI crawlers fetch content material to help real-time AI companies, corresponding to answering questions or summarizing pages, in addition to to reap information to construct massive coaching datasets for fashions like LLMs.

From Cloudflare Radar, we see that the overwhelming majority of single-purpose AI bot site visitors is for coaching, with search as a distant second. (See this weblog submit for a deep dialogue of the AI crawler site visitors we see at Cloudflare).

Whereas each search and coaching crawls impression cache by way of quite a few sequential, long-tail accesses, coaching site visitors has properties corresponding to excessive distinctive URL ratio, content material variety, and crawling inefficiency that make it much more impactful on cache.

AI crawler site visitors has three foremost differentiating traits: excessive distinctive URL ratio, content material variety, and crawling inefficiency.

Public crawl statistics from Frequent Crawl, which performs large-scale internet crawls on a month-to-month foundation, present that over 90% of pages are distinctive by content material. Completely different AI crawlers additionally goal distinct content material varieties: e.g., some focus on technical documentation, whereas others deal with supply code, media, or weblog posts. Lastly, AI crawlers don’t essentially comply with optimum crawling paths. A considerable fraction of fetches from standard AI crawlers lead to 404 errors or redirects, usually as a consequence of poor URL dealing with. The speed of those ineffective requests varies relying on how effectively the crawler is tuned to focus on reside, significant content material. AI crawlers additionally sometimes don’t make use of browser-side caching or session administration in the identical means human customers do. AI crawlers can launch a number of unbiased situations, and since they don’t share classes, every could seem as a brand new customer to the CDN, even when all situations request the identical content material.

Even a single AI crawler is more likely to dig deeper into web sites and discover a broader vary of content material than a typical human person. Utilization information from Wikipedia exhibits that pages as soon as thought-about “long-tail” or not often accessed are actually being regularly requested, shifting the distribution of content material reputation inside a CDN’s cache. In truth, AI brokers could iteratively loop to refine search outcomes, scraping the identical content material repeatedly. We mannequin this to point out that this iterative looping results in low content material reuse and broad protection.

Our modeling of AI agent habits exhibits that as they iteratively loop to refine search outcomes (a typical sample for retrieval-augmented technology), they preserve a constantly excessive distinctive entry ratio (the purple columns above) — sometimes between 70% and 100%. Which means every loop, whereas typically growing accuracy for the agent (represented right here by the blue line), is continually fetching new, distinctive content material quite than revisiting beforehand seen pages. 

This repeat entry to long-tail property churns the cache that the human site visitors depends on. That might make current pre-fetching and conventional cache invalidation methods much less efficient as the quantity of crawler site visitors will increase.  

For a CDN, a cache miss means having to go to the origin server to fetch the requested content material.  Consider a cache miss like your native library not having a guide in home, so you must wait to get the guide from inter-library mortgage. You’ll get your guide finally, however it would take longer than you needed. It is going to additionally inform your library that having that guide in inventory regionally may very well be a good suggestion.  

On account of their broad, unpredictable entry patterns with long-tail reuse, AI crawlers considerably elevate the cache miss price. And plenty of of our typical strategies to enhance our cache hit price, corresponding to cache hypothesis or prefetching, are considerably much less efficient.  

The primary chart under exhibits the distinction in cache hit charges for a single node in Cloudflare’s CDN with and with out our recognized AI crawlers. Whereas the impression of crawlers remains to be comparatively restricted, there’s a clear drop in hit price with the addition of AI crawler site visitors. We handle our cache with an algorithm known as “least recently used”, or LRU. Which means the least-requested content material might be evicted from cache first to create space for extra standard content material when cupboard space is full. The drop in hit price implies that LRU is struggling below the repeated scan habits of AI crawlers.

The underside determine exhibits Al cache misses throughout this time. Every of these cache misses represents a request to the origin, slowing response occasions in addition to growing egress prices and cargo on the origin.

This surge in AI bot site visitors has had real-world impression. The next desk from our paper exhibits the consequences on a number of massive web sites. Every instance hyperlinks to its supply report.

The impression is extreme: Wikimedia skilled a 50% surge in multimedia bandwidth utilization as a consequence of bulk picture scraping. Fedora, which hosts massive software program packages, and the Diaspora social community suffered from heavy load and poor efficiency for human customers. Many others have famous bandwidth will increase or slowdowns from AI bots repeatedly downloading massive recordsdata. Whereas blocking crawler site visitors mitigates among the impression, a wiser cache structure would let web site operators serve AI crawlers whereas sustaining response occasions for his or her human customers.

AI crawlers energy reside purposes corresponding to retrieval-augmented technology (RAG) or real-time summarization, so latency issues. That’s why these requests needs to be routed to caches that may steadiness bigger capability with average response occasions. These caches ought to nonetheless protect freshness, however can tolerate barely greater entry latency than human-facing caches. 

AI crawlers are additionally used for constructing coaching units and operating large-scale content material assortment jobs. These workloads can tolerate considerably greater latency and should not time-sensitive. As such, their requests might be served from deep cache tiers that take longer to succeed in (e.g., origin-side SSD caches), and even delayed utilizing queue-based admission or rate-limiters to stop backend overload. This additionally opens the chance to defer bulk scraping when infrastructure is below load, with out affecting interactive human or AI use circumstances.

Current initiatives like Cloudflare’s AI Index and Markdown for Brokers permit web site operators to current a simplified or decreased model of internet sites to identified AI brokers and bots. We’re planning to do far more to mitigate the impression of AI site visitors on CDN cache, main to raised cache efficiency for everybody. With our collaborators at ETH Zurich, we’re experimenting with two complementary approaches: first, site visitors filtering with AI-aware caching algorithms; and second, exploring the addition of a completely new cache layer to siphon AI crawler site visitors to a cache that may enhance efficiency for each AI crawlers and human site visitors. 

There are a number of several types of cache substitute algorithms, corresponding to LRU (“Least Recently Used”), LFU (“Least Frequently Used”), or FIFO (“First-In, First-Out”), that govern how a storage cache chooses to evict parts from the cache when a brand new factor must be added and the cache is full. LRU is usually the most effective steadiness of simplicity, low-overhead, and effectiveness for generic conditions, and is extensively used. For blended human and AI bot site visitors, nevertheless, our preliminary experiments point out {that a} totally different selection of cache substitute algorithm, notably utilizing SEIVE or S3FIFO, might permit human site visitors to attain the identical hit price with or with out AI interference. We’re additionally experimenting with creating extra straight workload-aware, machine learning-based caching algorithms to customise cache response in actual time for a sooner and cheaper cache.  

Long run, we count on {that a} separate cache layer for AI site visitors will probably be the easiest way ahead. Think about a cache structure that routes human and AI site visitors to distinct tiers deployed at totally different layers of the community. Human site visitors would proceed to be served from edge caches situated at CDN PoPs, which prioritize responsiveness and cache hit charges. For AI site visitors, cache dealing with might differ by process sort. 

The impression of AI bot site visitors on cloud infrastructure is just going to develop over the following few years. We want higher characterization of the consequences on CDNs throughout the globe, together with daring new cache insurance policies and architectures to deal with this novel workload and assist make a greater Web. 

Cloudflare is already fixing the issues we’ve laid out right here. Cloudflare reduces bandwidth prices for patrons who expertise excessive bot site visitors with our AI-aware caching, and with our AI Crawl Management and Pay Per Crawl instruments, we give clients higher management over who programmatically accesses their content material.

We’re simply getting began exploring this house. When you’re keen on constructing new ML-based caching algorithms or designing these new cache architectures, please apply for an internship! We’ve open internship positions in Summer time and Fall 2026 to work on this and different thrilling issues on the intersection of AI and Techniques.