Skip to main content

Insight · August 18, 2026

One door
for every model.

As agents multiply, every model call needs a place to be routed, capped, logged, and governed. That place is the AI gateway.

01 · The shift

The number of model calls just exploded.

For years an application talked to one model through one key. That era is ending. Gartner projects that 40% of enterprise applications will include agents built for specific tasks by the end of 2026, up from less than 5% in 2025, one of the fastest shifts in enterprise software since the move to public cloud.

Each of those agents makes many model calls. They reach different providers, they retry, they chain steps, they fall back when a provider is down. The traffic that used to be a single tidy line is now a crowd. Someone has to stand at the door and decide where each request goes.

02 · What it is

It sits between your apps and every model.

An AI gateway is one control surface for model traffic. Your applications call the gateway. The gateway calls the models. Everything you used to scatter across services, the routing rules, the keys, the budgets, the logging, now lives in one place with one policy model and one set of dashboards.

It is the same idea as the API gateway that has fronted web services for years, adapted to a different kind of traffic. An API gateway routes requests to services. An AI gateway routes requests to models, and adds what model traffic needs that web traffic never did: token accounting, prompt inspection, model fallback, and cost control. The application code stays simple. The gateway holds the complexity.

The one line to keep

“A gateway is not a wall you put in front of the models. It is the door they all already go through.”

03 · What the gateway actually does

Five jobs, one place.

01

Route

Send each request to the right model by rule, header, budget, or fallback order, without changing a line of application code.

02

Guard the keys

Provider credentials live in the gateway, not in every service. Rotate one key, not fifty scattered across teams.

03

Cap the spend

Per team and per app budgets, rate limits, and token accounting, set in one place and enforced on every call.

04

Log every call

Prompt, response, model, cost, and latency, recorded for every request. The audit trail writes itself.

05

Fail over

When a provider errors or slows, the gateway retries or routes to another model. The application never sees the outage.

04 · Not a rename

It is not your API gateway with a new label.

The instinct is to reach for the API gateway you already run. It does routing, keys, and rate limits, so why buy another box. The answer is that model traffic breaks the assumptions API gateways were built on.

An API gateway counts requests. Model cost is counted in tokens, and the same request can cost a cent or a dollar depending on the model and the length of the answer. An API gateway routes by path. Model routing depends on budget, latency, and which provider is healthy right now. An API gateway inspects headers. Model traffic carries prompts, which is exactly where injection and data leakage live. Different traffic, different control plane.

05 · One example that shows why it earns its place

One endpoint in front of many providers.

Cloudflare runs a gateway that sits in front of more than twenty model providers, from OpenAI and Anthropic to Amazon Bedrock and Azure. An application points at the gateway once. Behind that single endpoint the gateway caches repeated prompts, retries transient errors, fails over to a second provider when the first goes down, and records every request with its prompt, response, model, token count, cost, and duration.

Now picture the same setup with no gateway. Each application holds its own keys. Each team logs in its own format, if it logs at all. When a provider has an outage, every service that called it goes down at once, and the only way to learn what a model was asked is to find the engineer who wrote that service. The gateway does not add a step. It removes fifty copies of the same plumbing.

POST  gateway.yourco.com/v1/chat
route: cost_first      · pick the cheapest model that clears the bar
budget: team_growth    · stop when this team hits its cap
on_error: failover     · retry, then switch providers

one endpoint · every policy applied before the model ever answers

One call · routed, capped, logged, and retried by the gateway

06 · The governed fast path

The safe path has to be the fast path.

Governance fails when it is a checkpoint. If the safe way to call a model is slower or harder than the direct way, people route around it, and you get shadow AI, model use nobody approved and nobody can see.

A gateway inverts that. The governed path becomes the easiest path, one endpoint that already holds the keys, the budget, and the retries. You comply by using it, not by remembering to. That matters more every month. The EU AI Act began enforcement on 2 August 2026, with transparency and general purpose model duties now live and fines that reach 35 million euros or 7% of global turnover for the most serious violations. When a regulator or an auditor asks what your systems sent to which model, the answer has to exist. The gateway is where it lives, which is why it sits at the center of any real AI governance effort.

07 · When you need one

You do not need one on day one.

One app, one model, one key does not need a gateway. A plain client call is simpler, and simpler wins until it stops scaling. The gateway earns its place the moment you have more than one of anything: more than one model, more than one team, more than one environment that needs the same policy.

The tell is repetition. When you notice the same retry logic, the same key handling, and the same logging copied into a third service, you are building a gateway by hand, one service at a time. Better to run one on purpose than five by accident.

Closing

By the time you have many models, you already needed it.

Start with the calls you already make. Put them behind one endpoint that holds the keys, caps the spend, and logs the traffic. Add routing and fallback when a second model arrives. The gateway is not a project. It is the place all of this was always going to live.

Donny Smith

Written by · August 18, 2026

· Founder & Executive Creative Director, Bttr.

Over the past 15 years, he has led creative teams and contributed to products used by millions of people worldwide, working with companies including Apple, Opendoor, JP Morgan, GE Aerospace, Pepsi, and Alterra Mountain Company.

LinkedIn

Gartner press release, August 2025 · Cloudflare AI Gateway product documentation, 2026 · EU AI Act enforcement timeline, effective 2 August 2026

Share this perspective

More insights

Adjacent perspectives.

What Is LLM as a Judge

7 min read

What Is LLM as a Judge

LLM as a judge is the practice of using one language model to score another model’s output against a set of criteria. You hand the judge a rubric and either a single answer to rate or two answers to compare, and it returns a verdict: a number, a label, or a winner. It exists because the other options fall short. Anthropic’s evaluation guidance frames the tradeoff directly: code based grading is the fastest and most reliable but lacks nuance, human grading is the most flexible and highest quality but slow and expensive, and model based grading runs at the scale of code with something closer to the judgment of a person. The catch is that the grader is itself a model, so it carries a model’s biases. Zheng and colleagues named three in their foundational MT-Bench study: position bias, where a judge favors the answer shown first, verbosity bias, where it prefers the longer answer, and self enhancement bias, where it rates its own kind of output higher. The method still works. That same study found a strong judge agreed with human preferences more than 80 percent of the time, about the rate humans agree with one another, and 85 percent excluding ties against 81 percent among the people. The operator rule follows: validate the judge against human labels before you trust it, swap positions to cancel order effects, and grade with a different model than the one that wrote the answer.

What Is an AI Guardrail

7 min read

What Is an AI Guardrail

An AI guardrail is a check that runs around a model at the moment of a request. It reads what goes into the model and what comes out, screens for prompt injection, sensitive data, and off policy answers, and can block, redact, or retry when a check fails. It is enforcement code that sits outside the model, so the model itself does not have to be trusted to behave. Anthropic documents the core pattern directly: pass each input or tool result to a small, fast model such as Claude Haiku, ask only whether it carries an injection attempt, and let the content reach the main model only if the screen comes back clean. A guardrail is not an eval. An eval grades quality before you ship, offline and against a fixed set. A guardrail acts at run time, on the live request, one call at a time. What a guardrail is, how it differs from an eval, and how to layer a cheap deterministic check under a model based one.

Bttr. Field Brief

The brief Bttr. writes for senior buyers.

Monthly. One signal worth your time on Brand Operating Systems, AI search visibility, and the infrastructure buildout. No filler.