Insight · October 24, 2026
Not every call
needs your best model.
LLM routing sends each request to the model it actually needs, usually the cheapest one that can still clear the bar, and keeps a stronger one in reserve for the rest.
01 · The short answer
Routing picks the model per request, not per app.
LLM routing is the practice of sending each request to the model best suited to it, usually the cheapest one that can still clear your quality bar, rather than sending everything to one large model. A small component called a router reads the incoming request, judges how hard it is, and chooses which model answers.
The idea rests on a plain fact. Most of what an application asks a model to do is easy. A classification, a short rewrite, a lookup, a routine extraction. A frontier model answers those correctly, but so does a model that costs a fraction as much. Routing is how you stop paying frontier prices for easy work.
02 · Why it exists
One model for everything is the expensive default.
The usual way to ship an AI feature is to pick one capable model and send every request to it. It works, and it is the most common reason AI bills run higher than they need to. The hardest ten percent of requests set the model you pay for on the other ninety.
The model market made the alternative obvious. There is now a wide range of models at very different prices, from frontier reasoning models to small specialists that run cheaply. OpenRouter alone exposes over three hundred models through one API. When the options span that far in cost, sending every request to the top of the range is a choice, not a given.
Routing turns that range into a lever. The router sits in front of the models and decides, request by request, which one is enough. Nothing about the feature changes for the user. The only thing that changes is what you pay to serve it.
The one line to keep
“The question was never which model is best. It is which model each request actually needs.”
03 · How a router decides
Five moves, on every request.
01
Read
The router takes the request before any model sees it. The prompt, and often the context around it, are its only inputs. Nothing has been answered yet.
02
Score
It estimates how hard the request is. Some routers classify the prompt, some predict whether a small model would match a large one on this exact input.
03
Route
It compares that estimate against your bar and picks a model. Below the bar goes to the cheap one, above it to the strong one. The bar is a dial you set.
04
Call
The chosen model answers. To everything downstream it looks like a single call. The routing is invisible to the user and to the rest of your code.
05
Fall back
If the call fails, times out, or trips a limit, the router retries on another model instead of returning an error. That is why routing reads as reliability, not only savings.
04 · Router vs gateway
A router chooses the model. A gateway governs the call.
These two get merged in conversation, and they are different jobs. A router answers one question. For this request, which model. That is a quality and cost decision made fresh on every call.
An AI gateway is the control surface every model call passes through. It handles keys, logging, rate limits, budgets, and access, for all traffic, whoever sent it. Routing is one feature a gateway can offer, but a gateway without routing still does its job, and a router without a gateway still does its.
In practice they stack. The gateway is where every call is seen and controlled. The router is the decision inside it about which model gets the call. Tools like LiteLLM ship both, a single interface to over a hundred model providers, with a router that carries retry and fallback logic and tracks cost across them.
05 · The routing idea is already open
You do not have to take routing on faith.
The methods are public, and so is the code. Four open projects show the shape of the idea, from the research that proves the savings to the gateways that ship it in production.
RouteLLM
From the team behind Chatbot Arena. It trains routers on human preference data, then sends only the hard queries to the strong model. Its own benchmarks report cutting cost by up to eighty five percent while keeping ninety five percent of GPT-4 performance on MT Bench, and matching commercial routers for over forty percent less. It ships five routers, from a matrix factorization model to a BERT classifier.
LiteLLM
One interface to over a hundred model providers. Its router carries retry and fallback logic across deployments and tracks cost across all of them, so a single call can reach any vendor and recover from a failure without your code knowing which model answered.
vLLM Semantic Router
Maintained by the vLLM project and described as an open, programmable decision layer for models and compute. It selects or combines a model for each call, guided by a policy you set across quality, latency, and cost, and can keep calls inside approved locations.
OpenRouter
A single API in front of over three hundred models. The clearest proof of the premise. When the catalog spans that far in price and capability, choosing per request stops being an optimization and becomes the only sane default.
06 · What it actually buys
The savings are real, and not the headline number.
Up to eighty five percent sounds like a promise. It is a benchmark result on a specific task with a specific model pair, not a figure your bill will hit. The real number depends on your traffic. How many of your requests are genuinely easy, and how high you set the bar.
Two honest caveats come with it. The router is itself a cost and a point of failure. It runs on every request, and a bad route sends a hard question to a weak model and ships a wrong answer confidently. And a router is only as good as the bar you can measure it against. If you cannot tell a good answer from a bad one, you cannot tell whether the cheap model cleared it.
So routing is not a switch you flip. It is a decision you evaluate, with the same discipline you would bring to any model change. Measure quality on your own traffic, watch the routes, and move the bar until cost falls without a drop a user would notice.
07 · Where routing belongs
In the middle layer, not bolted onto every app.
The mistake is to bolt a router onto each feature. Then every team owns its own routing rules, its own fallback, its own cost tracking, and the picture of what you spend and why lives in a dozen places.
The cleaner shape is one layer that every call passes through, with routing as a job inside it. This is the layer we build at Bttr. as the token control plane, the middle layer between a company and every AI vendor. The router is one of its four jobs. For every question a team asks, it picks the right AI to answer it. Hard reasoning goes to one model, quick lookups to a cheaper one, long documents to whichever vendor priced them best this week, with the rules living in your repo, not a vendor console.
Put routing there and it stops being a per feature hack. It becomes a property of the whole system. One place that sees every call, routes it, and keeps the record so the next one is cheaper.
Closing
You never needed your best model for all of it.
Pick one feature with heavy, repetitive traffic. Measure the quality bar you actually need. Route everything under it to a cheaper model and watch the bill, not just the benchmark. The first time cost falls with no one noticing, routing stops being a theory.
Written by · October 24, 2026
Donny Smith · ECD, Founder, Bttr.
Over the past 15 years, he has led creative teams and contributed to products used by millions of people worldwide, working with companies including Apple, Opendoor, JP Morgan, GE Aerospace, Pepsi, and Alterra Mountain Company.
LinkedInFacts current as of October 2026 · LMSYS · RouteLLM · October 2026 · BerriAI · LiteLLM · October 2026 · vLLM Project · Semantic Router · October 2026 · OpenRouter · AI SDK Provider · October 2026
Share this perspective
More insights
Adjacent perspectives.
Bttr. Field Brief
The brief Bttr. writes for senior buyers.
Monthly. One signal worth your time on Brand Operating Systems, AI search visibility, and the infrastructure buildout. No filler.