compression that retains up with the agentic internet

Net pages have grown 6-9% heavier yearly for the previous decade, spurred by the net changing into extra framework-driven, interactive, and media-rich. Nothing about that trajectory is altering. What is altering is how typically these pages get rebuilt and what number of purchasers request them. Each are skyrocketing due to brokers. 

Shared dictionaries shrink asset transfers from servers to browsers so pages load sooner with much less bloat on the wire, particularly for returning customers or guests on a gradual connection. As a substitute of re-downloading total JavaScript bundles after each deploy, the browser tells the server what it already has cached, and the server solely sends the file diffs. 

At present, we’re excited to provide you a sneak peek of our assist for shared compression dictionaries, present you what we’ve seen in early testing, and reveal whenever you’ll have the ability to attempt the beta your self (trace: it’s April 30, 2026!). 

Agentic crawlers, browsers, and different instruments hit endpoints repeatedly, fetching full pages, typically to extract a fraction of knowledge. Agentic actors represented slightly below 10% of complete requests throughout Cloudflare’s community throughout March 2026, up ~60% year-over-year. 

Each web page shipped is heavier than final yr and browse extra typically by machines than ever earlier than. However brokers aren’t simply consuming the net, they’re serving to to construct it. AI-assisted improvement means groups ship sooner. Rising the frequency of deploys, experiments, and iterations is nice for product velocity, however horrible for caching.

As brokers push a one-line repair, the bundler re-chunks, filenames change, and each consumer on earth might re-download your complete software. Not as a result of the code is meaningfully any completely different, however as a result of the browser/consumer has no technique to know particularly what modified. It sees a brand new URL and begins from zero. Conventional compression helps with the scale of every obtain, however it might’t assist with the redundancy. It does not know the consumer already has 95% of the file cached. So each deploy, throughout each consumer, throughout each bot, sends redundant bytes time and again. Ship ten small modifications a day, and you have successfully opted out of caching. This wastes bandwidth and CPU in an internet the place {hardware} is shortly changing into the bottleneck.

So as to scale with extra requests hitting heavier pages which are re-deployed extra typically, compression has to get smarter. 

A compression dictionary is a shared reference between server and consumer that works like a cheat sheet. As a substitute of compressing a response from scratch, the server says “you already know this part of the file because you’ve cached it before” and solely sends what’s new. The consumer holds the identical reference and makes use of it to reconstruct the complete response throughout decompression. The extra the dictionary can reference content material within the file, the smaller the compressed output that’s transferred to the consumer.

This precept of compressing towards what’s already identified is how fashionable compression algorithms pull forward of their predecessors. Brotli ships with a built-in dictionary of widespread internet patterns like HTML attributes and customary phrases; Zstandard is purpose-built for customized dictionaries: you possibly can feed it consultant content material samples, and it generates an optimized dictionary for the type of content material you serve. Gzip has neither; it should construct dictionaries by discovering patterns in real-time because it’s compressing. These “traditional compression” algorithms are already out there on Cloudflare at present. 

Shared dictionaries take this precept a step additional: the beforehand cached model of the useful resource turns into the dictionary. Keep in mind the deploy downside the place a group ships a one-line repair and each consumer re-downloads the complete bundle? With shared dictionaries, the browser already has the previous model cached. The server compresses towards it, sending solely the diff. That 500KB bundle with a one-line change turns into just a few kilobytes on the wire. At 100K every day customers and 10 deploys a day, that is the distinction between 500GB of switch and some hundred megabytes.

Delta compression is what turns the model the browser already has into the dictionary. The protocol seems to when the server first serves a useful resource, it attaches a Use-As-Dictionary response header, telling the browser to basically maintain onto the file as a result of it’ll be helpful later. On the following request for that useful resource, the browser sends an Out there-Dictionary header again, telling the server, “here’s what I’ve got.” The server then proceeds to compress the brand new model towards the previous one and sends solely the diff. No separate dictionary file wanted. 

That is the place the payoff lands for actual purposes. Versioned JS bundles, CSS information, framework updates, and something that modifications incrementally between releases. The browser has app.bundle.v1.js cached already and the developer makes an replace and deploys app.bundle.v2.js. Delta compression solely sends the diff between these variations. Each subsequent model after can be only a diff. Model three compresses towards model two. Model 47 compresses towards model 46. The financial savings do not reset, they persist throughout your complete launch historical past.

There’s additionally lively dialogue locally about customized and dynamic dictionaries for non-static content material. That is future work, however the implications are vital. We’ll save that for an additional submit.

If shared dictionaries are so highly effective, why does not everybody use them already?

As a result of the final time they had been tried, the implementation could not survive contact with the open internet. 

Google shipped Shared Dictionary Compression for HTTP (SDCH) in Chrome in 2008. It labored effectively with some early adopters reporting double-digit enhancements in web page load instances. However SDCH amassed issues sooner than anybody was in a position to repair them.

Probably the most memorable was a category of compression side-channel assaults (CRIME, BREACH). Researchers confirmed that if an attacker might inject content material alongside one thing delicate that will get compressed (like a session cookie, token, and many others.) the scale of the compressed output might leak details about the key. The attacker might guess a byte at a time, watch whether or not the asset measurement shrank, and repeat till they extracted the entire secret. 

However safety wasn’t the one downside, and even the principle cause why adoption didn’t occur. SDCH surfaced a couple of architectural issues like violating the Identical-Origin Coverage (which paradoxically is partially why it carried out so effectively). Its cross-origin dictionary mannequin could not be reconciled with CORS, and it lacked some specification relating to interactions with issues just like the Cache API. After some time it grew to become clear that adoption wasn’t prepared, so in 2017 Chrome (the one browser supporting on the time) unshipped it. 

Getting the net neighborhood to choose up the baton took a decade, nevertheless it was price it.

The trendy normal, RFC 9842: Compression Dictionary Transport, closes key design gaps that made SDCH untenable. For instance, it enforces that an marketed dictionary is just usable on responses from the same-origin, stopping many situations that made side-channel compression assaults doable. 

Chrome and Edge have shipped assist with Firefox working to observe. The usual is transferring towards broad adoption, however full cross-browser assist remains to be catching up.

The RFC mitigates the safety issues however dictionary transport has at all times been complicated to implement. An origin might need to generate dictionaries, serve them with the fitting headers, test each request for an Out there-Dictionary match, delta-compress the response on the fly, and fall again gracefully when a consumer does not have a dictionary. Caching will get complicated too. Responses fluctuate on each encoding and dictionary hash, so each dictionary model creates a separate cache variant. Mid-deploy, you’ve purchasers with the previous dictionary, purchasers with the brand new one, and purchasers with none. Your cache is storing separate copies for every. Hit charges drop, storage climbs, and the dictionaries themselves have to remain contemporary underneath regular HTTP caching guidelines.

This complexity is a coordination downside. And precisely the type of factor that belongs on the edge. A CDN already sits in entrance of each request, already manages compression, and already handles cache variants (watch this house for a soon-to-come announcement weblog).

Shared dictionary compression touches each layer of the stack between the browser and the origin. We have seen robust buyer curiosity: some folks have already constructed their very own implementations like RFC writer Patrick Meenan‘s dictionary-worker, which runs the complete dictionary lifecycle inside a Cloudflare Employee utilizing WASM-compiled Zstandard (for instance).  We wish to make this accessible to everybody and as straightforward as doable to implement. So we’re rolling it out throughout the platform in three phases, beginning with the plumbing.

Section 1: Passthrough assist is presently in lively improvement. Cloudflare forwards the headers and encodings that shared dictionaries require like Use-As-Dictionary, Out there-Dictionary, and the dcb and dcz content material encodings, with out stripping, modifying, or recompressing them. The Cache keys are prolonged to fluctuate on Out there-Dictionary and Settle for-Encoding so dictionary-compressed responses are cached accurately. This part serves clients who handle their very own dictionaries on the origin.

We plan to have an open beta of Section 1 prepared by April 30, 2026. To use it, you may should be on a Cloudflare zone with the characteristic enabled, have an origin that serves dictionary-compressed responses with the proper headers (Use-As-Dictionary, Content material-Encoding: dcb or dcz), and your guests should be on a browser that advertises dcb/dcz in Settle for-Encoding and sends Out there-Dictionary. At present, meaning Chrome 130+ and Edge 130+, with Firefox assist in progress.

Hold your eyes fastened on the changelog for when this turns into out there and extra documentation for easy methods to use it. 

We’ve already began testing passthrough internally. In a managed take a look at, we deployed two js bundles in sequence. They had been practically equivalent aside from a couple of localized modifications between the variations representing successive deploys of the identical internet software. Uncompressed, the asset is 272KB. Gzip introduced that right down to 92.1KB, a stable 66% discount. With shared dictionary compression over DCZ, utilizing the earlier model because the dictionary, that very same asset dropped to 2.6KB. That is a 97% discount over the already compressed asset. 

In the identical lab take a look at, we measured two timing milestones from the consumer: time to first byte (TTFB) and full obtain completion. The TTFB outcomes are attention-grabbing for what they do not present. On a cache miss (the place DCZ has to compress towards the dictionary on the origin) TTFB is just about 20ms slower than gzip. The overhead is near-negligible for transmission.

The obtain instances are the place the distinction is. On a cache miss, DCZ accomplished in 31ms versus 166ms for gzip (an 81% enchancment). On a cache hit, 16ms versus 143ms (89% enchancment). The response is a lot smaller that even whenever you pay a slight penalty firstly, you end far forward.

Preliminary lab outcomes simulating minimal JS bundle diffs, outcomes will fluctuate based mostly on the precise delta between the dictionary and the asset.

Section 2: That is the place Cloudflare begins doing the give you the results you want. As a substitute of dealing with dictionary headers, compression, and fallback logic on the origin, on this part you inform Cloudflare which property must be used as dictionaries through a rule and we handle the remainder for you. We inject the Use-As-Dictionary headers, retailer the dictionary bytes, delta-compress new variations towards previous ones, and serve the fitting variant to every consumer. Your origin serves regular responses. Any dictionary complexity strikes off your infrastructure and onto ours.

To display this, we have constructed a stay demo to indicate what this seems like in apply. Attempt it right here: Can I Compress (with Dictionaries)? 

The demo deploys a brand new ~94KB JavaScript bundle each minute, meant to imitate a typical manufacturing single web page software bundle. The majority of the code is static between deploys; solely a small configuration block modifications every time, which additionally mirrors real-world deploys the place a lot of the bundle is unchanged framework and library code. When the primary model masses, Cloudflare’s edge shops it as a dictionary. When the following deploy arrives, the browser sends the hash of the model it already has, and the sting delta-compresses the brand new bundle towards it. The outcome: 94KB compresses to roughly 159 bytes. That is a 99.5% discount over gzip, as a result of the one factor on the wire is the precise diff.

The demo website contains walkthroughs so you possibly can confirm the compression ratios by yourself through curl or your browser.

Section 3: The dictionary is robotically generated on behalf of the web site. As a substitute of consumers specifying which property to make use of as dictionaries, Cloudflare identifies them robotically. Our community already sees each model of each useful resource that flows by it, which incorporates hundreds of thousands of websites, billions of requests, and each new deploy. The concept is that when the community observes a URL sample the place successive responses share most of their content material however differ by hash, it has a robust sign that the useful resource is versioned and a candidate for delta compression. It shops the earlier model as a dictionary and compresses subsequent variations towards it. No buyer configuration. No upkeep.

This can be a easy thought, however is genuinely arduous. Safely producing dictionaries that keep away from revealing non-public knowledge and figuring out site visitors for which dictionaries will supply essentially the most profit are actual engineering issues. However Cloudflare has the fitting items: we see the site visitors patterns throughout your complete community, we already handle the cache layer the place dictionaries have to stay, and our RUM beacon to purchasers may help give us a validation loop to verify {that a} dictionary really improves compression earlier than we decide to serving it. The mixture of site visitors visibility, edge storage, and artificial testing is what makes automated technology possible, although there are nonetheless many items to determine.

The efficiency and bandwidth advantages of part 3 are the crux of our motivation. That is what makes shared dictionaries accessible to everybody utilizing Cloudflare, together with the hundreds of thousands of zones that might by no means have had the engineering time to implement customized dictionaries manually. 

For a lot of the internet’s historical past, compression was stateless. Each response was compressed as if the consumer had by no means seen something earlier than. Shared dictionaries change that: they offer compression a reminiscence.

That issues extra now than it could have 5 years in the past. Agentic coding instruments are compressing the interval between deploys, whereas additionally driving a rising share of the site visitors that consumes them. Whereas at present AI instruments can produce huge diffs, brokers are gaining extra context and changing into surgical of their code modifications. This, coupled with extra frequent releases and extra automated purchasers means extra redundant bytes on each request. Delta compression helps each side of that equation by decreasing the variety of bytes per switch, and the variety of transfers that have to occur in any respect.

Shared Dictionaries took many years to standardize. Cloudflare helps to construct the infrastructure to make it work for each consumer that touches your website, human or not. Section 1 beta opens April 30, and we’re excited so that you can attempt it.

_____

 ^{1Bots = ~31.3% of all HTTP requests. AI = ~29-30% of all Bot site visitors (March 2026).}