Insight · November 14, 2026
What is a
multimodal model?
One model that reads text, sees images, and hears audio in the same context. Not a reader with a vision tool bolted on. One mind, many senses.
01 · The definition
More than one sense, in one mind.
A multimodal model is an AI system that takes in and reasons across more than one kind of data, such as text, images, audio, and video, inside a single context. It maps each input into one shared representation, so it can relate a word, a picture, and a sound as parts of the same thing.
A model that only reads text is unimodal. It handles one modality and nothing else. The multimodal model widens that to several at once, and the point is not that it has an extra feature. It is that the senses meet inside the model, where it can reason over all of them together instead of processing each one alone.
02 · Native vs bolted on
Stitched together, or trained as one.
There are two ways to build one, and the difference decides almost everything. The older way stitches separate models together. When Google launched Gemini in December 2023, it described the usual method directly: “the standard approach to creating multimodal models involved training separate components for different modalities and then stitching them together.” A speech model passes text to a language model, which asks an image model. Each handoff is a translation, and each translation drops something.
The native way trains one network across the senses from the start. Google said Gemini was instead “pre-trained from the start on different modalities” so it could “seamlessly understand, operate across and combine different types of information including text, code, audio, image and video.” That is the line between a model that can see and a model that was taught to see while it learned to read.
This is why a vision tool bolted onto a text model is not the same thing. The bolt on can describe an image, then forget it and reason only over the words that came back. The native model keeps the image in the same context as the text, so a question about the words can look straight at the picture.
The one line to keep
“Stitching models together builds a pipeline. Training one across the senses builds a mind.”
03 · What makes it multimodal
Four things, whatever the model.
01
One context, not one pipe
Text, image, audio, and video land in the same window at once. The model does not hand a picture to a separate tool and wait for a caption. It reads the picture where it reads the words.
02
A shared representation
Each input is turned into numbers that live in one space, so the word dog, a photo of a dog, and a bark can sit close together. Meaning, not format, is what the model compares.
03
Trained across modalities
A native model learns the senses together from the start, so a reference in the text can point at a region of the image. A stitched system learns them apart and joins them later.
04
Input, and sometimes output
Most models read many modalities and answer in text. Newer ones also produce images and audio from the same network, so the senses run both directions.
04 · How the senses meet
Everything becomes numbers in one space.
The trick that makes it work is a shared representation. Each modality gets turned into a list of numbers, and those numbers are placed in one common space so that related things, a caption and the photo it describes, sit near each other. The model compares meaning, not file type.
CLIP, which OpenAI released in 2021, is the clean early example. It trained two encoders, one for images and one for text, on roughly 400 million pairs of images and captions until matching pairs landed close together in the same space. That is what let it sort images it had never been trained on, using plain text labels alone.
Where the senses get joined is a real design choice. A November 2025 survey of fusion strategies for vision language systems lays out the range: late fusion runs a separate encoder per modality and combines them at the end, the way CLIP does, while early fusion sends image patches and text tokens into the same network from the first layer, the way models such as LLaVA do. Earlier the merge, tighter the reasoning across senses, and heavier the compute.
05 · One example that shows the whole difference
Three models in a line, or one that hears directly.
Voice assistants used to be a pipeline. OpenAI described its older voice mode as three separate models in a row: one transcribed your speech to text, a language model read the text and wrote a reply, and a third turned that reply back into audio. The language model in the middle never heard your voice. Tone, a second speaker, a laugh, all of it was gone by the time the words reached it. That pipeline answered in about 2.8 seconds with one model and about 5.4 seconds with a larger one.
GPT-4o, which OpenAI released in May 2024, collapsed the line into one model. OpenAI said it “trained a single new model end-to-end across text, vision, and audio, meaning that all inputs and outputs are processed by the same neural network.” Because nothing waits in a queue, it answers audio in as little as 232 milliseconds and about 320 milliseconds on average, close to the pace of human conversation. The model hears the voice itself, not a transcript of it.
stitched speech → text model → language model → text → audio
the middle never hears the voice · about 5.4s
native one model · text, vision, audio in and out
hears the voice directly · about 0.32sSame senses · different architecture · one of them keeps what the other drops
06 · What it unlocks in a product
People do not send the world as plain text.
By 2026, native multimodality is the default shape of a frontier model, not a feature added to one. Gemini shipped in three sizes for that reason, from a large model for complex work down to a small one meant to run on a device. The senses come built in.
For a product, that is the real change. A support tool can read the angry message and the screenshot attached to it in the same pass, instead of asking the person to describe what they already showed you. A field app can answer a question about a photo of a part. A model can watch a short clip and tell you what happened in it. None of that needs a separate vision service wired in by hand.
The cost is that a multimodal interface has more ways to confuse a person and more ways for the model to be wrong. An image read incorrectly is still answered with full confidence. That is a design problem as much as a model one, which is the ground we cover in design systems for AI products.
07 · Where the model meets the work
A sense is only useful once a person can act on it.
A multimodal model is raw capability. The product is what decides which inputs to accept, how to show what the model saw, and how a person corrects it when the read is wrong. The base term under this one is the AI agent, and the limit that shapes how much the model can hold at once is the context window, since an image and a minute of video spend far more of it than a line of text.
That interface layer is where Bttr. works, as AI operating systems, “the layer where models, workflows, agents, and humans actually meet.” A model that can see and hear is only worth what a person can do with what it saw and heard.
Closing
You never took in the world one sense at a time.
Neither should the model. A multimodal model is the step where the machine stops reading a description of the world and starts taking in the world itself, in more than one sense at once, the way the person it works for always has.

Written by · November 14, 2026
Donny Smith · Founder & Executive Creative Director, Bttr.
Over the past 15 years, he has led creative teams and contributed to products used by millions of people worldwide, working with companies including Apple, Opendoor, JP Morgan, GE Aerospace, Pepsi, and Alterra Mountain Company.
LinkedInGoogle · Introducing Gemini, our largest and most capable AI model · December 2023 · OpenAI · Hello GPT-4o · May 2024 · OpenAI · CLIP, connecting text and images · January 2021 · Willis and Bakos · Exploring fusion strategies for multimodal vision language systems · arXiv · November 2025 · Facts current as of November 14, 2026.
Share this perspective
More insights
Adjacent perspectives.
Bttr. Field Brief
The brief Bttr. writes for senior buyers.
Monthly. One signal worth your time on Brand Operating Systems, AI search visibility, and the infrastructure buildout. No filler.