Skip to main content

Insight · August 14, 2026

You cannot fix
what you cannot see.

An agent makes dozens of hidden moves before it answers. Observability is how you watch every one.

01 · The problem

An agent is not one call. It is a chain of them.

A single model call is easy to watch. You send text, you get text back, you log both. An agent is different. It reasons, calls a tool, reads the result, decides what to do next, calls another, and only then answers. One question from the user becomes a sequence of hidden moves inside the machine.

When the answer comes back wrong, the cause is rarely the last step. A tool returned nothing at step three. The plan at step five acted on the nothing. Ten steps later the agent was confident and wrong. A log line that says the request failed tells you none of that. You need to see the whole run.

02 · What it is

Observability is seeing the run, not just the result.

AI agent observability is the practice of recording every step an agent takes so you can see what happened and why. It borrows the shape from how engineers already watch complex software. A request becomes a trace. Each step inside it becomes a span. The spans nest, so the picture is a tree that starts at the agent run and branches into every model call, tool call, and decision beneath it.

Each span carries its own record: what was asked, what came back, how long it took, how many tokens it burned, whether it errored. Read the tree top to bottom and the run stops being a black box. It becomes a timeline you can follow, a chain of choices with an address for every one.

The one line to keep

“Evals grade the answer. Observability shows the work.”

03 · What a trace is made of

Five kinds of span, nested into one tree.

01

The agent span

The whole run, from the request to the answer. It is the root of the tree, and every other span hangs beneath it.

02

The model span

One call to the model. It records the model asked, the tokens in and out, and the reason the model stopped.

03

The tool span

One tool the model asked to run. A search, a database read, a payment. It records the tool, the inputs, and what came back.

04

The plan span

The moment the agent decides what to do next. This is where a wrong turn is chosen, and where you look first when the answer is wrong.

05

The workflow span

Several agents coordinating on one job. It wraps the sub agents so a handoff between them is something you can see, not guess at.

These are not names one vendor invented. The OpenTelemetry project, the open standard for tracing software, now defines spans for agent runs, model calls, tool execution, and planning under its generative AI conventions. The same shape works whether the agent calls Anthropic, OpenAI, or a model you host yourself.

04 · Not the same as evals

Evals grade quality. Observability shows what happened.

These two get confused, and they should not. An eval is a test. It asks whether the answer was good against a bar you set. Observability is a record. It shows the steps the agent took to reach that answer, good or bad.

You need both, and they feed each other. An eval tells you the answer at step ten was wrong. The trace tells you why, by showing the empty tool result at step three that started the failure. Without the eval you do not know there is a problem. Without the trace you cannot find it.

This is the same discipline that turns a loop you can trust into one you can leave running. A loop that checks itself still needs a record of what it did, so that when it does fail, the failure has a place you can find it.

05 · One trace, read top to bottom

The failure has an address.

Here is one agent run, drawn as a trace. A user asked for the status of an order. The agent planned, called the model, ran two tools, and answered. The indentation shows what nested under what. The numbers are the time each span took.

invoke_agent  order-status                         1.84s
├─ plan       decide next step                     0.21s
├─ chat       gpt-4 · 512 tokens                    0.44s
├─ execute_tool  lookup_order(id)   returned null   0.05s
├─ execute_tool  search_orders(email)               0.31s
└─ chat       gpt-4 · 388 tokens   answer built     0.39s

Illustrative trace · every span carries timing, tokens, and result

The answer was wrong. Read the trace and you see it in seconds. The order lookup returned null, because the id was never the right key. The agent fell back to a search, found the wrong record, and built a confident answer on top of it. No line of code threw an error. Nothing crashed. The only evidence the failure happened is in the trace, and once you have it, the fix is obvious.

06 · What to record

Write down the model, the tokens, and the reason it stopped.

A useful span records more than a start and an end. The OpenTelemetry conventions give each field a standard name, so the data reads the same across tools. The model call records the model asked for, the tokens in and the tokens out, and the reason the model stopped. Cached tokens get their own count, because they cost less and hide a large part of the bill when you ignore them.

The tool span records the tool called, the call that triggered it, and what came back. The agent span carries a name and a stable id, so a run you are looking at today lines up with the same agent tomorrow. Token accounting rolls up the tree, which is the only honest way to answer what a single agent run actually cost.

The generative AI part of the standard is still marked in development, and moving. The move that ages well is to instrument against the open standard rather than one vendor's private format. Standard names travel. If you change platforms later, the traces you already recorded still make sense.

07 · Why it matters now

An agent you cannot see is not really in production.

A single model call fails loudly. An agent fails quietly. It keeps going, works around the gap, and hands back an answer that looks finished. The cost of not seeing it is not a crash you notice. It is a slow drift of wrong answers you never catch, and a bill that grows for reasons you cannot name.

The three questions a team cannot answer without traces are the three that matter most. Why did this run give the wrong answer. Where is the time going when a request is slow. What did this actually cost. Each one lives in the tree, and nowhere else.

As agents take on real work and act on real systems, this stops being a nice to have. It becomes the difference between an agent you can run in front of customers and one you can only demo. You cannot improve a run you cannot see, and you cannot trust one you cannot check.

Closing

Ship the agents you can see.

Take one agent you already run. Turn on tracing and send a single request through it. Read the tree it leaves behind, span by span. The first time a wrong answer turns out to have a clear address, you will stop shipping agents you cannot watch.

Donny Smith

Written by · August 14, 2026

· Founder & Executive Creative Director, Bttr.

Over the past 15 years, he has led creative teams and contributed to products used by millions of people worldwide, working with companies including Apple, Opendoor, JP Morgan, GE Aerospace, Pepsi, and Alterra Mountain Company.

LinkedIn

OpenTelemetry generative AI semantic conventions, agent and tool spans · Datadog LLM observability documentation · reviewed August 2026

Share this perspective

More insights

Adjacent perspectives.

What Is LLM as a Judge

7 min read

What Is LLM as a Judge

LLM as a judge is the practice of using one language model to score another model’s output against a set of criteria. You hand the judge a rubric and either a single answer to rate or two answers to compare, and it returns a verdict: a number, a label, or a winner. It exists because the other options fall short. Anthropic’s evaluation guidance frames the tradeoff directly: code based grading is the fastest and most reliable but lacks nuance, human grading is the most flexible and highest quality but slow and expensive, and model based grading runs at the scale of code with something closer to the judgment of a person. The catch is that the grader is itself a model, so it carries a model’s biases. Zheng and colleagues named three in their foundational MT-Bench study: position bias, where a judge favors the answer shown first, verbosity bias, where it prefers the longer answer, and self enhancement bias, where it rates its own kind of output higher. The method still works. That same study found a strong judge agreed with human preferences more than 80 percent of the time, about the rate humans agree with one another, and 85 percent excluding ties against 81 percent among the people. The operator rule follows: validate the judge against human labels before you trust it, swap positions to cancel order effects, and grade with a different model than the one that wrote the answer.

What Is an AI Guardrail

7 min read

What Is an AI Guardrail

An AI guardrail is a check that runs around a model at the moment of a request. It reads what goes into the model and what comes out, screens for prompt injection, sensitive data, and off policy answers, and can block, redact, or retry when a check fails. It is enforcement code that sits outside the model, so the model itself does not have to be trusted to behave. Anthropic documents the core pattern directly: pass each input or tool result to a small, fast model such as Claude Haiku, ask only whether it carries an injection attempt, and let the content reach the main model only if the screen comes back clean. A guardrail is not an eval. An eval grades quality before you ship, offline and against a fixed set. A guardrail acts at run time, on the live request, one call at a time. What a guardrail is, how it differs from an eval, and how to layer a cheap deterministic check under a model based one.

Bttr. Field Brief

The brief Bttr. writes for senior buyers.

Monthly. One signal worth your time on Brand Operating Systems, AI search visibility, and the infrastructure buildout. No filler.