Insight · October 22, 2026

Small models,
big enough.

NVIDIA Research says small language models are the future of agentic AI. The logic is hard to fault once you see what an agent actually does all day.

01 · The short answer

Small enough to run where the work is.

A small language model is a language model small enough to run on a laptop, a phone, or a single consumer GPU. Its parameters number from a few million to a few billion, not the hundreds of billions of a frontier model. It trades broad knowledge for speed, low cost, and running close to the work.

That is the whole idea. A frontier model knows almost everything and bills you accordingly. A small model knows less, answers faster, costs little, and can live on the device in your hand. For a narrow job, less is not a compromise. It is the right tool.

02 · Why the argument landed

An agent does narrow work, over and over.

In June 2025, a team from NVIDIA Research and Georgia Tech, led by Peter Belcak, published a position paper with a blunt title: small language models are the future of agentic AI. Their claim, in the paper's own words, is that small models are “sufficiently powerful, inherently more suitable, and necessarily more economical for many invocations in agentic systems.”

The reasoning starts with what an agent is. An agent does not hold one long, open conversation. It takes a goal and breaks it into many small steps. Read a field. Pick a tool. Format a result. Check a value against a rule. Most of those steps are narrow and repeated, and a small model handles them well.

A frontier model can do each step too. It is also overqualified for nearly all of them, and you pay frontier prices every single time it answers. The paper's point is that paying for general intelligence on work that does not need it is a habit, not a requirement.

The one line to keep

“The right question was never how big a model you can afford. It is how small a model can do the job.”

03 · What counts as small

A working line, and the models that sit under it.

There is no standards body for the word small. The common 2026 convention puts a small model under 10 billion parameters, often in the 1 to 7 billion range, small enough to run on consumer hardware without a cloud API. The boundary is fuzzy on purpose. A model that fits on a laptop today would have filled a server two years ago.

These are not toy models. Every major lab now ships a small tier built for exactly this work.

Microsoft Phi

Built to reason well above its weight on narrow tasks.

near 4 billion parameters in its smallest instruct variant

Google Gemma 3

The smallest variants fit on a phone and still take long inputs.

270 million to 27 billion across the family

Meta Llama 3.2

Shipped for the device in your hand rather than the data center.

1 billion and 3 billion

Alibaba Qwen3

Tool calling is built in, which is what an agent needs.

small dense models at 0.6, 1.7, 4 and 8 billion

Hugging Face SmolLM3

Open weights and open training, with tool calling supported.

3 billion, fully open

04 · One model is the wrong default

Not every call is the same size.

Most agents today route every step to one large model, because it is simple and it works. The paper calls for the opposite: a heterogeneous system, meaning an agent that invokes more than one model and sends each call to the smallest one that can handle it.

Think about the steps again. Deciding the plan for a hard, open request is real reasoning, and the big model earns its cost there. Pulling a date out of an email, choosing between three tools, or turning a result into clean JSON is not. Those are the calls a small specialist answers in a fraction of the time for a fraction of the price.

When an agent runs thousands of steps a day, the mix is where the cost lives. Sending the narrow work to a small model is not a downgrade. It is matching the tool to the task.

05 · How much of the work is actually narrow

Most of an agent's calls do not need the big model.

The paper does more than assert this. It takes real agents and estimates, step by step, how many of their model calls actually needed a large model and how many a specialized small model could have handled.

In MetaGPT, a coding agent, the estimate is that roughly 60 percent of the calls could be handled reliably by appropriately specialized small models. In Cradle, an agent that controls a computer screen, the estimate rises to about 70 percent. The rest, the genuinely open or hard calls, stay with the large model.

Read those numbers with care. This is a position paper, and these are estimates, not a measured benchmark. The point is not the exact percentage. It is the shape: in real agents, the majority of the work is narrow enough that a small model is enough.

large generalist   plan the hard, open request   ~30 to 40%
small specialist   read a field, pick a tool     ~60 to 70%
                   format a result, check a rule

route each call to the smallest model that holds the bar.

Estimated split from the paper's agent case studies

06 · What the setup looks like

A generalist to plan, specialists to do.

The practical shape is a mix. Keep a strong generalist for the calls that need broad reasoning, long novel context, or open conversation. Hand the narrow, repeated calls to small models tuned for them. A router in the middle decides which call goes where.

The small specialists are where you earn the gains. A model in the single billions can be taught a narrow job once and then run it cheaply forever. If you know when to fine tune a model, a small one is where it pays off fastest, because you are shaping behavior on a task you can describe, not trying to add knowledge the model never had.

This is also why small models suit the way an agent actually works. An agent is a loop of many small, bounded decisions. Bounded decisions are exactly what a small, fast, cheap model is good at.

07 · When to keep the big model

Small is not always enough, and that is fine.

This is not an argument to throw out large models. Broad reasoning across a long, novel problem, open conversation with a person, and the hardest planning steps are still where a frontier model is worth every token. The paper does not dispute that. It argues against using the big model for everything by default.

The line between the two is fuzzy, and it moves. A task that needs a large model this year may fit a small one next year as the small tier improves. The discipline is to check, not to assume. Measure where a small model holds the bar you set, and keep the large model only where it does not.

That is the real shift. Not small instead of large, but the right size for each call, chosen on evidence rather than habit.

Closing

For most of the work, small was always enough.

Look at one agent you run. Count how many of its calls are narrow, repeated steps and how many are genuinely hard. Move the narrow ones to a small model and watch the bill, and the latency, fall without the quality following them down.

Written by · October 22, 2026

Donny Smith · ECD, Founder, Bttr.

Over the past 15 years, he has led creative teams and contributed to products used by millions of people worldwide, working with companies including Apple, Opendoor, JP Morgan, GE Aerospace, Pepsi, and Alterra Mountain Company.

Sources, facts current as of October 2026 · arXiv · Small Language Models are the Future of Agentic AI, Belcak et al., NVIDIA Research · June 2025 · IBM · What Are Small Language Models · 2025 · Microsoft Azure · What Are Small Language Models · 2025 · Machine Learning Mastery · Introduction to Small Language Models · 2026

Share this perspective

More insights

Adjacent perspectives.

7 min read

What Is Test-Time Compute

Test-time compute is the processing a model spends at the moment you ask, rather than during training. For years a better model meant more training: more data, more parameters, more compute spent once. Test-time compute is the second axis, spent at inference on the one question in front of the model, and it comes in two shapes. Serial means one long chain of reasoning, what Anthropic describes as multiple sequential steps before the final output. Parallel means several independent attempts at once with the best one selected. The gains are measured, not felt: using the equivalent of 256 samples, a learned scoring model and a 64K token budget, Anthropic reported Claude 3.7 Sonnet reaching a GPQA score of 84.8 percent. The operator point is plain. Thinking is billed like any output, so the skill is not turning the dial to the maximum. It is matching the compute to the difficulty of the task.

7 min read

The Model Is Twenty Percent

A useful vertical agent system is four things besides the model: connectors to the industry data, evals written by people who know the work, rules for what the agent can read and change, and a clean review screen for a human. Treat the model as swappable.

8 min read

How to Evaluate a RAG System

Evaluating a RAG system means scoring its two stages on their own, because a retrieval augmented generation answer is only as good as the passages behind it, and a wrong answer has two possible causes a single number cannot tell apart. Retrieval is measured before the model writes anything. The LlamaIndex evaluation guide scores a retriever with hit rate, the share of questions where the right passage lands in the top results, and mean reciprocal rank, which rewards ranking it high. Generation is measured given what was retrieved. Ragas, the open evaluation toolkit now maintained by VibrantLabs, splits the answer into claims and scores faithfulness as the share the retrieved context supports, then scores answer relevance by generating questions from the answer and comparing them to the real one. Read as two pairs, the four metrics locate the fault: high recall with low faithfulness is a generation problem, low recall with high faithfulness is a retrieval one. Anthropic showed the payoff in September 2024, cutting a retrieval failure rate measured by recall at 20 from 5.7 percent to 1.9 percent by adding context, keyword search, and reranking. And it is not a one time check. Models drift, documents go stale, so the teams that trust their systems run these scores on live traffic the way they run unit tests.

View all insights →

Bttr. Field Brief

The brief Bttr. writes for senior buyers.

Monthly. One signal worth your time on Brand Operating Systems, AI search visibility, and the infrastructure buildout. No filler.

Right size the model to the work

Small models,big enough.