This weblog publish you’re studying takes 16,180 tokens in HTML and three,150 tokens when transformed to markdown. That’s a 80% discount in token utilization.
Markdown has shortly turn out to be the lingua franca for brokers and AI techniques as a complete. The format’s specific construction makes it supreme for AI processing, in the end leading to higher outcomes whereas minimizing token waste.
The issue is that the Net is manufactured from HTML, not markdown, and web page weight has been steadily growing over time, making pages arduous to parse. For brokers, their purpose is to filter out all non-essential components and scan the related content material.
The conversion of HTML to markdown is now a typical step for any AI pipeline. Nonetheless, this course of is way from supreme: it wastes computation, provides prices and processing complexity, and above all, it will not be how the content material creator meant their content material for use within the first place.
What if AI brokers may bypass the complexities of intent evaluation and doc conversion, and as a substitute obtain structured markdown immediately from the supply?
Convert HTML to markdown, mechanically
Cloudflare’s community now helps real-time content material conversion on the supply, for enabled zones utilizing content material negotiation headers. Now when AI techniques request pages from any web site that makes use of Cloudflare and has Markdown for Brokers enabled, they will specific the desire for textual content/markdown within the request. Our community will mechanically and effectively convert the HTML to markdown, when potential, on the fly.
Right here’s the way it works. To fetch the markdown model of any web page from a zone with Markdown for Brokers enabled, the shopper wants so as to add the Settle for negotiation header with textual content/markdown as one of many choices. Cloudflare will detect this, fetch the unique HTML model from the origin, and convert it to markdown earlier than serving it to the shopper.
Here is a curl instance with the Settle for negotiation header requesting a web page from our developer documentation:
curl
-H "Accept: text/markdown"Or when you’re constructing an AI Agent utilizing Employees, you should use TypeScript:
const r = await fetch(
``,
{
headers: {
Settle for: "text/markdown, text/html",
},
},
);
const tokenCount = r.headers.get("x-markdown-tokens");
const markdown = await r.textual content();We already see a number of the hottest coding brokers immediately – like Claude Code and OpenCode – ship these settle for headers with their requests for content material. Now, the response to this request is formatted in markdown. It is that straightforward.
HTTP/2 200
date: Wed, 11 Feb 2026 11:44:48 GMT
content-type: textual content/markdown; charset=utf-8
content-length: 2899
differ: settle for
x-markdown-tokens: 725
content-signal: ai-train=sure, search=sure, ai-input=sure
---
title: Markdown for Brokers · Cloudflare Brokers docs
---
## What's Markdown for Brokers
The power to parse and convert HTML to Markdown has turn out to be foundational for AI.
...
Word that we embrace an x-markdown-tokens header with the transformed response that signifies the estimated variety of tokens within the markdown doc. You should utilize this worth in your move, for instance to calculate the dimensions of a context window or to determine in your chunking technique.
Right here’s a diagram of the way it works:
Throughout our final Birthday Week, Cloudflare introduced Content material Alerts — a framework that permits anybody to specific their preferences for the way their content material can be utilized after it has been accessed.
While you return markdown, you need to be sure that your content material is being utilized by the Agent or AI crawler. That’s why Markdown for Brokers transformed responses embrace the Content material-Sign: ai-train=sure, search=sure, ai-input=sure header signaling that signifies content material can be utilized for AI Coaching, Search outcomes and AI Enter, which incorporates agentic use. Markdown for Brokers will present choices to outline customized Content material Sign insurance policies sooner or later.
Verify our devoted Content material Alerts web page for extra info on this framework.
Strive it with the Cloudflare Weblog & Developer Documentation
We enabled this function in our Developer Documentation and our Weblog, inviting all AI crawlers and brokers to eat our content material utilizing markdown as a substitute of HTML.
Strive it out now by requesting this weblog with Settle for: textual content/markdown.
curl
-H "Accept: text/markdown"The result’s:
---
description: The way in which content material is found on-line is shifting, from conventional engines like google to AI brokers that want structured information from a Net constructed for people. It’s time to contemplate not simply human guests, however begin to deal with brokers as first-class residents. Markdown for Brokers mechanically converts any HTML web page requested from our community to markdown.
title: Introducing Markdown for Brokers
picture:
---
# Introducing Markdown for Brokers
The way in which content material and companies are found on-line is altering quickly. Prior to now, site visitors originated from conventional engines like google and search engine marketing decided who bought discovered first. Now the site visitors is more and more coming from AI crawlers and brokers that demand structured information inside the often-unstructured Net that was constructed for people.
...Different methods to transform to Markdown
In case you’re constructing AI techniques that require arbitrary doc conversion from exterior Cloudflare or Markdown for Brokers is just not obtainable from the content material supply, we offer different methods to transform paperwork to Markdown on your functions:
Employees AI AI.toMarkdown() helps a number of doc varieties, not simply HTML, and summarization.
Browser Rendering /markdown REST API helps markdown conversion if you could render a dynamic web page or utility in an actual browser earlier than changing it.
Anticipating a shift in how AI techniques browse the Net, Cloudflare Radar now contains content material kind insights for AI bot and crawler site visitors, each globally on the AI Insights web page and within the particular person bot info pages.
The brand new content_type dimension and filter exhibits the distribution of content material varieties returned to AI brokers and crawlers, grouped by MIME kind class.
It’s also possible to see the requests for markdown filtered by a selected agent or crawler. Listed here are the requests that return markdown to OAI-Searchbot, the crawler utilized by OpenAI to energy ChatGPT’s search:
This new information will enable us to trace the evolution of how AI bots, crawlers, and brokers are consuming Net content material over time. As all the time, every thing on Radar is freely accessible through the public APIs and the Information Explorer.
To allow Markdown for Brokers on your zone, log into the Cloudflare dashboard, choose your account, choose the zone, search for Fast Actions and toggle the Markdown for Brokers button to allow. This function is out there immediately in Beta for gratis for Professional, Enterprise and Enterprise plans, in addition to SSL for SaaS clients.
Yow will discover extra details about Markdown for Brokers on our Developer Docs. We welcome your suggestions as we proceed to refine and improve this function. We’re curious to see how AI crawlers and brokers navigate and adapt to the unstructured nature of the Net because it evolves.
