Introducing the Agent Readiness rating. Is your web site agent-ready?

The net has all the time needed to adapt to new requirements. It realized to talk to net browsers, after which it realized to talk to engines like google. Now, it wants to talk to AI brokers.

Immediately, we’re excited to introduce isitagentready.com — a brand new device to assist web site house owners perceive how they will make their websites optimized for brokers, from guiding brokers on the best way to authenticate, to controlling what content material brokers can see, the format they obtain it in, and the way they pay for it. We’re additionally introducing a brand new dataset to Cloudflare Radar that tracks the general adoption of every agent commonplace throughout the Web.

We wish to lead by instance. That’s the reason we’re additionally sharing how we just lately overhauled Cloudflare’s Developer Documentation to make it essentially the most agent-friendly documentation web site, permitting AI instruments to reply questions quicker and considerably cheaper.

The brief reply: not very. That is anticipated, but in addition reveals how way more efficient brokers might be than they’re at this time, if requirements are adopted.

To research this, Cloudflare Radar took the 200,000 most visited domains on the Web; filtered out classes the place agent readiness is not vital (like redirects, ad-servers, and tunneling providers) to concentrate on companies, publishers, and platforms that AI brokers may realistically must work together with; and scanned them utilizing our new device.

The result’s a brand new “Adoption of AI agent standards” chart that may now be discovered within the Cloudflare Radar AI Insights web page the place we will measure adoption of every commonplace throughout a number of area classes.

particular person checks, just a few issues stood out:

• robots.txt is almost common — 78% of websites have one — however the overwhelming majority are written for conventional search engine crawlers, not AI brokers.

• Content material Alerts: 4% of websites have declared their AI utilization preferences in robots.txt. It is a new commonplace that’s gaining momentum.

• Markdown content material negotiation (serving textual content/markdown on Settle for: textual content/markdown) passes on 3.9% of websites.

• New rising requirements like MCP Server Playing cards and API Catalogs (RFC 9727) collectively seem on fewer than 15 websites in all the dataset. It’s nonetheless early — there’s a number of alternative to face out by being one of many first websites to undertake new requirements and work effectively with brokers. 

This chart will probably be up to date weekly, and the info can be accessed by way of the Information Explorer or the Radar API.

You will get an agent readiness rating to your personal web site by going to isitagentready.com and getting into the positioning’s URL.

Scores and audits that present actionable suggestions have helped to drive adoption of latest requirements earlier than. For instance, Google Lighthouse scores web sites on efficiency and safety greatest practices, and guides web site house owners to undertake the newest net platform requirements. We expect one thing comparable ought to exist to assist web site house owners undertake greatest practices for brokers.

While you enter your web site, Cloudflare makes requests to it to test which requirements it helps, and supplies a rating based mostly on 4 dimensions:

^{Screenshot of outcomes from an agent-readiness test for an instance web site.}

Moreover, we test if the positioning helps agentic commerce requirements together with x402, Common Commerce Protocol, and Agentic Commerce Protocol, however these don’t presently rely in direction of the rating.

For every failing test, we offer a immediate you can give to your coding agent and have it implement assist in your behalf.

The positioning itself can be agent-ready, training what it preaches. It exposes a stateless MCP server ( with a scan_site device through Streamable HTTP, so any MCP-compatible agent can scan web sites programmatically with out utilizing the net interface. It additionally publishes an Agent Abilities index ( with ability paperwork for each commonplace it checks, so brokers not solely know what to repair, however the best way to repair it.

Let’s dig into the checks in every class, and why they matter for brokers.

robots.txt has been round since 1994, and most websites have one. It serves two functions for brokers: it defines crawl guidelines (who can entry what) and it factors to your sitemaps. A sitemap is an XML file that lists each path in your web site, primarily a map brokers can observe to find all of your content material with out having to crawl each hyperlink. The robots.txt is the place brokers look first.

Past sitemaps, brokers also can uncover vital sources straight from HTTP response headers, particularly, utilizing the Hyperlink response header (RFC 8288). Not like hyperlinks buried inside HTML, the Hyperlink header is a part of the HTTP response itself, which suggests an agent can discover hyperlinks to sources with out having to parse any markup:

HTTP/1.1 200 OK
Hyperlink: ; rel="api-catalog"

Getting an agent onto your web site is one factor. Ensuring it may really learn your content material is one other.

Again in September 2024, which appears like a lifetime in the past given how briskly AI is shifting, llms.txt was proposed as a manner to offer a LLM-friendly illustration of a web site, and match inside the mannequin’s context window. llms.txt is a plain textual content file on the root of your web site that provides brokers a structured studying record: what the positioning is, what’s on it, and the place the vital content material lives. Consider it as a sitemap written for an LLM to learn moderately than a crawler to index:

# My Website
> A developer platform for constructing on the sting.
## Documentation
- [Getting Started](
- [API Reference](
## Changelog
- [Release Notes](

Markdown content material negotiation goes even additional. When an agent fetches any web page and sends an Settle for: textual content/markdown header, the server responds with a clear markdown model as an alternative of HTML. The markdown model requires far fewer tokens — we measured as much as 80% token discount in some circumstances — which makes responses quicker, cheaper, and extra prone to be consumed in its entirety, given the boundaries on context home windows that almost all agent instruments have by default.

By default, we solely test whether or not the positioning accurately handles Markdown content material negotiation, and don’t test for llms.txt. You possibly can customise the scan to incorporate llms.txt should you select to.

Now that brokers can navigate your web site and devour your content material, the following query is: do you wish to let any bot do it?

robots.txt does greater than level to sitemaps. It is usually the place you outline your entry guidelines. You possibly can explicitly declare which crawlers are allowed and what they will entry, all the way down to particular paths. This conference is effectively established and continues to be the primary place any well-behaved bot appears to be like earlier than it begins crawling.

Content material Alerts allow you to be extra particular. Fairly than simply enable or block, you’ll be able to declare precisely what AI can do together with your content material. Utilizing a Content material-Sign directive in your robots.txt, you’ll be able to independently management three issues: whether or not your content material can be utilized for AI coaching (ai-train), whether or not it may be used as AI enter for inference and grounding (ai-input), and whether or not it ought to seem in search outcomes (search):

Consumer-agent: *
Content material-Sign: ai-train=no, search=sure, ai-input=sure

Inversely, the Internet Bot Auth IETF draft commonplace permits pleasant bots to authenticate themselves, and permits web sites receiving requests from bots to establish them. A bot indicators its HTTP requests, and the receiving web site verifies these signatures utilizing the bot’s printed public keys.

These public keys stay at a well known endpoint, /.well-known/http-message-signatures-directory, which we test as a part of the scan.

Not all websites must implement this. In case your web site simply serves content material, and doesn’t make requests to different websites, you don’t want it. However as extra websites on the Web run their very own brokers that make requests to different websites, we anticipate this to be more and more vital over time.

Past passive content material consumption, brokers also can work together together with your web site straight by calling APIs, invoking instruments, and finishing duties autonomously.

In case your service has a number of public APIs, the API Catalog (RFC 9727) provides brokers a single well-known location to find all of them. Hosted at /.well-known/api-catalog, it lists your APIs and hyperlinks to their specs, docs, and standing endpoints, with out requiring brokers to scrape your developer portal or learn your documentation.

We will not discuss brokers with out mentioning MCP. The Mannequin Context Protocol is an open commonplace that permits AI fashions to attach with exterior information sources and instruments. As a substitute of constructing a customized integration for each AI device, you construct one MCP server and any suitable agent can use it.

To assist brokers discover your MCP server, you’ll be able to publish an MCP Server Card (a proposal presently in draft). It is a JSON file at /.well-known/mcp/server-card.json that describes your server earlier than an agent even connects: what instruments it exposes, the best way to attain it, and the best way to authenticate. An agent reads this file and is aware of every thing it wants to start out utilizing your server:

{
  "$schema": "
  "version": "1.0",
  "protocolVersion": "2025-06-18",
  "serverInfo": {
    "name": "search-mcp-server",
    "title": "Search MCP Server",
    "version": "1.0.0"
  },
  "description": "Search across all documentation and knowledge base articles",
  "transport": {
    "type": "streamable-http",
    "endpoint": "/mcp"
  },
  "authentication": {
    "required": false
  },
  "tools": [
    {
      "name": "search",
      "title": "Search",
      "description": "Search documentation by keyword or question",
      "inputSchema": {
        "type": "object",
        "properties": {
          "query": { "type": "string" }
        },
        "required": ["query"]
      }
    }
  ]
}

Brokers work greatest once they have Agent Abilities that assist them carry out particular duties — however how can brokers uncover what expertise a web site supplies? We’ve proposed that websites could make this data out there at .well-known/agent-skills/index.json, an endpoint that tells the agent what expertise can be found and the place to search out them. You may discover that the .well-known commonplace (RFC 8615) is utilized by many different agent and authorization requirements — thanks to Cloudflare’s personal Mark Nottingham who authored the usual, and different IETF contributors!

Many websites require you to register first as a way to entry them. This makes it exhausting for people to offer brokers the power to entry these websites on their behalf, and is why some have taken the arguably unsafe workaround method of giving brokers entry to the person’s net browser, with their logged-in session.

There’s a greater manner that permits people to explicitly grant entry: websites that assist OAuth can inform brokers the place to search out the authorization server (RFC 9728), permitting brokers to ship people by way of an OAuth circulation, the place they will select to correctly grant entry to the agent. Introduced at Brokers Week 2026, Cloudflare Entry now totally helps this OAuth circulation, and we confirmed how brokers like OpenCode could make use of this commonplace to make issues simply work when customers give brokers protected URLs:

Brokers also can purchase issues in your behalf — however funds on the net had been designed for people. Add to cart, enter a bank card, click on pay. That circulation breaks down solely when the client is an AI agent.

x402 solves this on the protocol stage by reviving HTTP 402 Fee Required, a standing code that has existed within the spec since 1997 however was by no means broadly used. The circulation is easy: an agent requests a useful resource, the server responds with a 402 and a machine-readable payload describing the cost phrases, the agent pays and retries. Cloudflare partnered with Coinbase to launch the x402 Basis, whose mission is to drive adoption of x402 as an open commonplace for Web funds.

We additionally test for Common Commerce Protocol and Agentic Commerce Protocol — two rising agentic commerce requirements designed to permit brokers to find and buy merchandise that people would usually buy through ecommerce storefronts and checkout flows.

Cloudflare’s URL Scanner enables you to submit any URL and get an in depth report on it: HTTP headers, TLS certificates, DNS information, applied sciences used, efficiency information, and safety alerts. It’s a basic device for safety researchers and builders who wish to perceive what a URL is definitely doing underneath the hood.

We’ve taken the identical checks from isitagentready.com and added them to URL Scanner with a brand new Agent Readiness tab. While you scan any URL, you may now see its full agent readiness report alongside the present evaluation: which of the checks move, what stage the positioning is at, and actionable steering to enhance your rating.

The combination can be out there programmatically through the URL Scanner API. To incorporate agent readiness leads to a scan, move the agentReadiness choice in your scan request:

curl -X POST  
    -H 'Content material-Sort: software/json' 
    -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" 
    -d '{
          "url": "
          "options": {"agentReadiness": true}
        }'

As we constructed the instruments to measure the Internet’s readiness, we knew we had to make sure our personal home was so as. Our docs should be simply digestible by the brokers our prospects use.

We naturally adopted the related content material web site requirements talked about above, and you may test our rating right here. Nonetheless, we didn’t cease there. Right here is how we refined Cloudflare’s Developer Docs to be essentially the most agent-friendly useful resource on the net.

Sadly, as of February 2026, of seven brokers examined, solely Claude Code, OpenCode, and Cursor request content material with the Settle for: textual content/markdown header by default. For the remainder, we would have liked a seamless URL-based fallback.

To do that, we make each web page out there individually through Markdown at /index.md relative to the web page’s URL. We do that dynamically, with out duplicating static information, by combining two Cloudflare Guidelines: 

• A URL Rewrite Rule matches requests ending in /index.md and dynamically rewrites them to the bottom path utilizing regex_replace (stripping /index.md). 

• A Request Header Remodel Rule matches in opposition to the unique request’s path earlier than the rewrite (uncooked.http.request.uri.path) and mechanically units the Settle for: textual content/markdown header. 

With these two guidelines, any web page might be fetched as Markdown through appending the /index.md path to the URL:

We level to those /index.md URLs in our llms.txt information. Successfully, for these /index.md paths, we all the time return markdown, no matter what headers the consumer units. And we do that with none further construct step or content material duplication.

llms.txt serves as a “home base” for brokers, offering a listing of pages to assist LLMs discover content material. Nonetheless, 5,000+ pages of documentation in a single file will exceed fashions’ context home windows.

As a substitute of 1 huge file, we generate a separate llms.txt file for every top-level listing in our docs and the foundation llms.txt merely factors to those subdirectories.

We additionally take away tons of of directory-listing pages that present little semantic worth to an LLM, and we guarantee every web page has wealthy descriptive context (titles, semantic names, and descriptions).

For instance, we omit roughly 450 pages that solely function localized listing listings, like https://developers.cloudflare.com/workers/databases/.

These pages seem in our sitemap, however they comprise little or no data for an LLM. Since all baby pages are already linked individually in llms.txt, fetching a listing web page solely supplies a redundant record of hyperlinks, forcing the agent to make one other request to search out precise content material.

To assist brokers navigate effectively, every llms.txt entry should be wealthy in context however gentle on tokens. People may ignore frontmatter and filtering labels, however for an AI agent, this metadata is the steering wheel. That’s the reason our Product Content material Expertise (PCX) workforce has refined our web page titles, descriptions, and URL constructions in order that brokers all the time know precisely which pages to fetch.

Check out a bit from our root llms.txt.

Every hyperlink has a semantic title, an identical URL, and a high-value description. None of this required further work for llms.txt era. It was all already out there within the docs frontmatter. The identical goes for pages in prime stage listing llms.txt information. All of this context empowers brokers to search out related data extra effectively.

Moreover, we check our docs in opposition to afdocs, an rising agent-friendly documentation spec and open-source undertaking that permits groups to check docs websites for issues like content material discovery and navigation. This spec allowed us to construct customized audit tooling of our personal. By including just a few deliberate patches particular to our use case, we created a dashboard for straightforward evaluation.

We pointed an agent (Kimi-k2.5 through OpenCode) at different giant technical documentation websites’ llms.txt information and tasked the agent with answering extremely particular technical questions.

On common, the agent pointed at Cloudflare’s documentation consumed 31% fewer tokens and arrived on the right reply 66% quicker than the typical web site that’s not refined for brokers. By becoming our product directories into single context home windows, brokers can establish the precise web page they want and fetch it in a single, linear path.

Accuracy in LLM responses is usually a byproduct of context window effectivity. Throughout our testing, we noticed a recurring sample with different documentation units.

1. The grep loop: Many documentation websites present a single, huge llms.txt file that exceeds the agent’s quick context window. As a result of the agent can not “read” the entire file, it begins to grep for key phrases. If the primary search misses the particular element, the agent should suppose, refine its search, and check out once more.

2. Narrowed context and decrease accuracy: When an agent depends on iterative looking out moderately than studying the complete file, it loses the broader context of the documentation. This fragmented view typically leads the agent to have a diminished understanding of the documentation at hand.

3. Latency and token bloat: Every iteration of the grep loop requires the agent to generate new “thinking tokens” and execute further search requests. This back-and-forth makes the ultimate response noticeably slower and will increase the full token rely, driving up the fee for the tip person.

In contrast, Cloudflare docs are designed to suit solely inside an agent’s context window. This permits the agent to ingest the listing, establish the precise web page it wants, and fetch the Markdown with out detour.

Documentation for legacy merchandise like Wrangler v1 or Employees Websites presents a singular problem. Whereas we should preserve this data accessible for historic functions, it may result in outdated recommendation from AI brokers.

For instance, a human studying these docs would see the massive banner stating that Wrangler v1 is deprecated, along with a hyperlink to the latest content material. An LLM crawler, nevertheless, may ingest the textual content with out that surrounding visible context. This leads to the agent recommending outdated data.

Redirects for AI Coaching solves this by figuring out AI coaching crawlers and deliberately redirecting them away from deprecated or suboptimal content material. This ensures that whereas people can nonetheless entry historic archives, LLMs are solely fed our most present and correct implementation particulars.

Each HTML web page in our docs features a hidden directive particularly for LLMs. 

“STOP! If you are an AI agent or LLM, read this before continuing. This is the HTML version of a Cloudflare documentation page. Always request the Markdown version instead — HTML wastes context. Get this page as Markdown: (append index.md) or send Accept: text/markdown to For all Cloudflare products use You can access all Cloudflare docs in a single file at https://developers.cloudflare.com/llms-full.txt.”

This snippet informs the agent {that a} Markdown model is out there. Crucially, this directive is stripped from the precise Markdown model to keep away from a recursion loop the place the agent retains making an attempt to “find” the Markdown inside the Markdown.

Lastly, we wish to make these sources discoverable for the people who’re constructing with brokers. Each product listing in our developer documentation has an “LLM Resources” entry within the sidenav, offering fast entry to llms.txt, llms-full.txt, and Cloudflare Abilities.

Making web sites agent-ready is a basic accessibility requirement for the trendy developer toolkit. The transition from a “human-read web” to a “machine-read web” is the most important architectural shift in a long time. 

Get an agent readiness rating to your web site at isitagentready.com, take the prompts it supplies, and ask your agent to improve your web site for the AI period. Keep tuned for extra updates from Cloudflare Radar in regards to the adoption of agent requirements throughout the Web over the approaching yr. If we’ve realized something from the previous yr, it’s that quite a bit can change in a short time!