an inference layer designed for brokers

AI fashions are altering shortly: one of the best mannequin to make use of for agentic coding right this moment may in three months be a very totally different mannequin from a special supplier. On prime of this, real-world use instances usually require calling multiple mannequin. Your buyer help agent may use a quick, low cost mannequin to categorise a person’s message; a big, reasoning mannequin to plan its actions; and a light-weight mannequin to execute particular person duties.

This implies you want entry to all of the fashions, with out tying your self financially and operationally to a single supplier. You additionally want the suitable programs in place to observe prices throughout suppliers, guarantee reliability when certainly one of them has an outage, and handle latency regardless of the place your customers are.

These challenges are current everytime you’re constructing with AI, however they get much more urgent if you’re constructing brokers. A easy chatbot may make one inference name per person immediate. An agent may chain ten calls collectively to finish a single activity and all of the sudden, a single gradual supplier does not add 50ms, it provides 500ms. One failed request is not a retry, however all of the sudden a cascade of downstream failures. 

Since launching AI Gateway and Employees AI, we’ve seen unimaginable adoption from builders constructing AI-powered functions on Cloudflare and we’ve been delivery quick to maintain up! In simply the previous few months, we have refreshed the dashboard, added zero-setup default gateways, automated retries on upstream failures, and extra granular logging controls. At the moment, we’re making Cloudflare right into a unified inference layer: one API to entry any AI mannequin from any supplier, constructed to be quick and dependable. 

Beginning right this moment, you possibly can name third-party fashions utilizing the identical AI.run() binding you already use for Employees AI. When you’re utilizing Employees, switching from a Cloudflare-hosted mannequin to at least one from OpenAI, Anthropic, or some other supplier is a one-line change. 

const response = await env.AI.run('anthropic/claude-opus-4-6',{
enter: 'What's Cloudflare?',
}, {
gateway: { id: "default" },
});

For individuals who don’t use Employees, we’ll be releasing REST API help within the coming weeks, so you possibly can entry the total mannequin catalog from any surroundings.

We’re additionally excited to share that you’re going to now have entry to 70+ fashions throughout 12+ suppliers — all via one API, one line of code to change between them, and one set of credit to pay for them. And we’re shortly increasing this as we go.

You possibly can flick thru our mannequin catalog to seek out one of the best mannequin on your use case, from open-source fashions hosted on Cloudflare Employees AI to proprietary fashions from the key mannequin suppliers. We’re excited to be increasing entry to fashions from Alibaba Cloud, AssemblyAI, Bytedance, Google, InWorld, MiniMax, OpenAI, Pixverse, Recraft, Runway, and Vidu — who will present their fashions via AI Gateway. Notably, we’re increasing our mannequin choices to incorporate picture, video, and speech fashions with the intention to construct multimodal functions

Accessing all of your fashions via one API additionally means you possibly can handle all of your AI spend in a single place. Most corporations right this moment are calling a mean of three.5 fashions throughout a number of suppliers, which implies nobody supplier is ready to provide you with a holistic view of your AI utilization. With AI Gateway, you’ll get one centralized place to observe and handle AI spend.

By together with customized metadata along with your requests, you will get a breakdown of your prices on the attributes that you just care about most, like spend by free vs. paid customers, by particular person prospects, or by particular workflows in your app.

const response = await env.AI.run('@cf/moonshotai/kimi-k2.5',
      {
immediate: 'What's AI Gateway?'
      },
      {
metadata: { "teamId": "AI", "userId": 12345 }
      }
    );

AI Gateway provides you entry to fashions from all of the suppliers via one API. However generally it’s essential to run a mannequin you have fine-tuned by yourself information or one optimized on your particular use case. For that, we’re engaged on letting customers carry their very own mannequin to Employees AI. 

The overwhelming majority of our site visitors comes from devoted cases for Enterprise prospects who’re working customized fashions on our platform, and we wish to carry this to extra prospects. To do that, we leverage Replicate’s Cog expertise that can assist you containerize machine studying fashions.

Cog is designed to be fairly easy: all it’s essential to do is write down dependencies in a cog.yaml file, and your inference code in a Python file. Cog abstracts away all of the onerous issues about packaging ML fashions, corresponding to CUDA dependencies, Python variations, weight loading, and so forth.

Instance of a cog.yaml file:

construct:
  python_version: "3.13"
  python_requirements: necessities.txt
predict: "predict.py:Predictor"

Instance of a predict.py file, which has a operate to arrange the mannequin and a operate that runs if you obtain an inference request (a prediction):

from cog import BasePredictor, Path, Enter
import torch

class Predictor(BasePredictor):
    def setup(self):
        """Load the model into memory to make running multiple predictions efficient"""
        self.web = torch.load("weights.pth")

    def predict(self,
            picture: Path = Enter(description="Image to enlarge"),
            scale: float = Enter(description="Factor to scale image by", default=1.5)
    ) -> Path:
        """Run a single prediction on the model"""
        # ... pre-processing ...
        output = self.web(enter)
        # ... post-processing ...
        return output

Then, you possibly can run cog construct to construct your container picture, and push your Cog container to Employees AI. We’ll deploy and serve the mannequin for you, which you then entry via your standard Employees AI APIs. 

We’re engaged on some massive initiatives to have the ability to carry this to extra prospects, like customer-facing APIs and wrangler instructions with the intention to push your personal containers, in addition to sooner chilly begins via GPU snapshotting. We’ve been testing this internally with Cloudflare groups and a few exterior prospects who’re guiding our imaginative and prescient. When you’re eager about being a design accomplice with us, please attain out! Quickly, anybody will be capable to bundle their mannequin and use it via Employees AI.

Utilizing Employees AI fashions with AI Gateway is especially highly effective when you’re constructing stay brokers – the place a person’s notion of pace hinges on time to first token or how shortly the agent begins responding, fairly than how lengthy the total response takes. Even when whole inference is 3 seconds, getting that first token 50ms sooner makes the distinction between an agent that feels zippy and one which feels sluggish.

Cloudflare’s community of knowledge facilities in 330 cities world wide means AI Gateway is positioned near each customers and inference endpoints, minimizing the community time earlier than streaming begins.

Employees AI additionally hosts open-source fashions on its public catalog, which now consists of giant fashions purpose-built for brokers, together with Kimi K2.5 and real-time voice fashions. While you name these Cloudflare-hosted fashions via AI Gateway, there isn’t any further jump over the general public Web since your code and inference run on the identical world community, giving your brokers the bottom latency attainable.

When constructing brokers, pace shouldn’t be the one issue that customers care about – reliability issues too. Each step in an agent workflow will depend on the steps earlier than it. Dependable inference is essential for brokers as a result of one name failing can have an effect on the complete downstream chain. 

By way of AI Gateway, when you’re calling a mannequin that is out there on a number of suppliers and one supplier goes down, we’ll routinely route to a different out there supplier with out you having to jot down any failover logic of your personal. 

When you’re constructing long-running brokers with Brokers SDK, your streaming inference calls are additionally resilient to disconnects. AI Gateway buffers streaming responses as they’re generated, independently of your agent’s lifetime. In case your agent is interrupted mid-inference, it may well reconnect to AI Gateway and retrieve the response with out having to make a brand new inference name or paying twice for a similar output tokens. Mixed with the Brokers SDK’s built-in checkpointing, the top person by no means notices.

The Replicate crew has formally joined our AI Platform crew, a lot in order that we don’t even contemplate ourselves separate groups anymore. We’ve been onerous at work on integrations between Replicate and Cloudflare, which embody bringing all of the Replicate fashions onto AI Gateway and replatforming the hosted fashions onto Cloudflare infrastructure. Quickly, you’ll be capable to entry the fashions you liked on Replicate via AI Gateway, and host the fashions you deployed on Replicate on Employees AI as effectively.

To get began, take a look at our documentation for AI Gateway or Employees AI. Study extra about constructing brokers on Cloudflare via Brokers SDK.