Skip to main content

Insight · November 14, 2026

What is a
multimodal model?

One model that reads text, sees images, and hears audio in the same context. Not a reader with a vision tool bolted on. One mind, many senses.

01 · The definition

More than one sense, in one mind.

A multimodal model is an AI system that takes in and reasons across more than one kind of data, such as text, images, audio, and video, inside a single context. It maps each input into one shared representation, so it can relate a word, a picture, and a sound as parts of the same thing.

A model that only reads text is unimodal. It handles one modality and nothing else. The multimodal model widens that to several at once, and the point is not that it has an extra feature. It is that the senses meet inside the model, where it can reason over all of them together instead of processing each one alone.

02 · Native vs bolted on

Stitched together, or trained as one.

There are two ways to build one, and the difference decides almost everything. The older way stitches separate models together. When Google launched Gemini in December 2023, it described the usual method directly: “the standard approach to creating multimodal models involved training separate components for different modalities and then stitching them together.” A speech model passes text to a language model, which asks an image model. Each handoff is a translation, and each translation drops something.

The native way trains one network across the senses from the start. Google said Gemini was instead “pre-trained from the start on different modalities” so it could “seamlessly understand, operate across and combine different types of information including text, code, audio, image and video.” That is the line between a model that can see and a model that was taught to see while it learned to read.

This is why a vision tool bolted onto a text model is not the same thing. The bolt on can describe an image, then forget it and reason only over the words that came back. The native model keeps the image in the same context as the text, so a question about the words can look straight at the picture.

The one line to keep

“Stitching models together builds a pipeline. Training one across the senses builds a mind.”

03 · What makes it multimodal

Four things, whatever the model.

01

One context, not one pipe

Text, image, audio, and video land in the same window at once. The model does not hand a picture to a separate tool and wait for a caption. It reads the picture where it reads the words.

02

A shared representation

Each input is turned into numbers that live in one space, so the word dog, a photo of a dog, and a bark can sit close together. Meaning, not format, is what the model compares.

03

Trained across modalities

A native model learns the senses together from the start, so a reference in the text can point at a region of the image. A stitched system learns them apart and joins them later.

04

Input, and sometimes output

Most models read many modalities and answer in text. Newer ones also produce images and audio from the same network, so the senses run both directions.

04 · How the senses meet

Everything becomes numbers in one space.

The trick that makes it work is a shared representation. Each modality gets turned into a list of numbers, and those numbers are placed in one common space so that related things, a caption and the photo it describes, sit near each other. The model compares meaning, not file type.

CLIP, which OpenAI released in 2021, is the clean early example. It trained two encoders, one for images and one for text, on roughly 400 million pairs of images and captions until matching pairs landed close together in the same space. That is what let it sort images it had never been trained on, using plain text labels alone.

Where the senses get joined is a real design choice. A November 2025 survey of fusion strategies for vision language systems lays out the range: late fusion runs a separate encoder per modality and combines them at the end, the way CLIP does, while early fusion sends image patches and text tokens into the same network from the first layer, the way models such as LLaVA do. Earlier the merge, tighter the reasoning across senses, and heavier the compute.

05 · One example that shows the whole difference

Three models in a line, or one that hears directly.

Voice assistants used to be a pipeline. OpenAI described its older voice mode as three separate models in a row: one transcribed your speech to text, a language model read the text and wrote a reply, and a third turned that reply back into audio. The language model in the middle never heard your voice. Tone, a second speaker, a laugh, all of it was gone by the time the words reached it. That pipeline answered in about 2.8 seconds with one model and about 5.4 seconds with a larger one.

GPT-4o, which OpenAI released in May 2024, collapsed the line into one model. OpenAI said it “trained a single new model end-to-end across text, vision, and audio, meaning that all inputs and outputs are processed by the same neural network.” Because nothing waits in a queue, it answers audio in as little as 232 milliseconds and about 320 milliseconds on average, close to the pace of human conversation. The model hears the voice itself, not a transcript of it.

stitched    speech → text model → language model → text → audio
            the middle never hears the voice · about 5.4s

native      one model · text, vision, audio in and out
            hears the voice directly · about 0.32s

Same senses · different architecture · one of them keeps what the other drops

06 · What it unlocks in a product

People do not send the world as plain text.

By 2026, native multimodality is the default shape of a frontier model, not a feature added to one. Gemini shipped in three sizes for that reason, from a large model for complex work down to a small one meant to run on a device. The senses come built in.

For a product, that is the real change. A support tool can read the angry message and the screenshot attached to it in the same pass, instead of asking the person to describe what they already showed you. A field app can answer a question about a photo of a part. A model can watch a short clip and tell you what happened in it. None of that needs a separate vision service wired in by hand.

The cost is that a multimodal interface has more ways to confuse a person and more ways for the model to be wrong. An image read incorrectly is still answered with full confidence. That is a design problem as much as a model one, which is the ground we cover in design systems for AI products.

07 · Where the model meets the work

A sense is only useful once a person can act on it.

A multimodal model is raw capability. The product is what decides which inputs to accept, how to show what the model saw, and how a person corrects it when the read is wrong. The base term under this one is the AI agent, and the limit that shapes how much the model can hold at once is the context window, since an image and a minute of video spend far more of it than a line of text.

That interface layer is where Bttr. works, as AI operating systems, “the layer where models, workflows, agents, and humans actually meet.” A model that can see and hear is only worth what a person can do with what it saw and heard.

Closing

You never took in the world one sense at a time.

Neither should the model. A multimodal model is the step where the machine stops reading a description of the world and starts taking in the world itself, in more than one sense at once, the way the person it works for always has.

Donny Smith

Written by · November 14, 2026

· Founder & Executive Creative Director, Bttr.

Over the past 15 years, he has led creative teams and contributed to products used by millions of people worldwide, working with companies including Apple, Opendoor, JP Morgan, GE Aerospace, Pepsi, and Alterra Mountain Company.

LinkedIn

Share this perspective

More insights

Adjacent perspectives.

What Is Temperature in AI

7 min read

What Is Temperature in AI

Temperature is a setting that controls how random a language model’s output is. At each step the model scores every word it knows, turns those scores into probabilities with a softmax, and temperature is applied inside that step: it divides each score before the probabilities are worked out, which sharpens or flattens the gap between the leading word and the rest. A Hugging Face guide to text generation describes the move as making the distribution sharper by increasing the likelihood of high probability words and decreasing the likelihood of low probability ones, and notes that as temperature approaches zero the sampling becomes equal to greedy decoding, where the model takes its single highest scoring word every time. OpenAI’s API documentation puts the range between 0 and 2, says higher values like 0.8 make the output more random while lower values like 0.2 make it more focused and deterministic, and recommends altering temperature or top_p but not both. The catch most teams miss is that a low number is not a lock. Anthropic states in its glossary that even with temperature set to 0 the results will not be fully deterministic, and identical inputs might produce different outputs across API calls. The operator rule follows: run cold when the work has one right answer, run warm when you want range, and never read a warmer setting as a smarter model.

What Is Tool Calling

7 min read

What Is Tool Calling

Tool calling is how a language model acts on the world. You describe a function to the model as a name, a description, and a schema for its inputs. When a request needs it, the model does not answer in prose. It returns a structured call naming the function and its arguments, your code runs that call, and the result goes back so the model can answer from something real. Anthropic puts it plainly: tool use, also called function calling, lets a model call functions you define or that the vendor provides, and the model only decides what to run, never runs it. The names differ across vendors and the substance does not. OpenAI shipped the idea as function calling, Anthropic calls it tool use, and the Claude documentation settles the overlap in one line. This is the base layer under every agent, since each single act an agent takes is a tool call, and the Model Context Protocol standardizes the layer above it so one tool, described once, can be reached by any model that speaks the protocol.

Bttr. Field Brief

The brief Bttr. writes for senior buyers.

Monthly. One signal worth your time on Brand Operating Systems, AI search visibility, and the infrastructure buildout. No filler.