Insight · November 2, 2026
Taught by
preference.
A raw language model predicts the next token. RLHF tunes it toward the answers people actually rate higher, which is the step that turned raw models into assistants you can talk to.
01 · The idea
Learning from what people prefer.
RLHF, reinforcement learning from human feedback, is a training method that tunes a language model toward answers people prefer. Humans compare pairs of model outputs, those choices train a reward model, and the model is then optimized to score higher against it. It is the step that turned raw models into usable assistants.
The reason it exists is simple. A base model is trained to predict the next token, so it is good at continuing text and indifferent to whether the continuation is helpful, honest, or safe. RLHF supplies the missing signal: not what word comes next, but which whole answer a person would rather receive.
02 · How it works
A reward model, then a careful push toward it.
The hard part of alignment is that you cannot write down a formula for a good answer. So RLHF learns one. Labelers are shown two responses to the same prompt and pick the better one. A reward model is fit to those picks, trained to give a higher score to the answer a person chose. In practice the reward model starts from the fine-tuned model and swaps its word prediction head for a single score.
With a reward model in hand, the real model becomes a policy to be improved. It writes an answer, the reward model scores it, and reinforcement learning, usually the algorithm called PPO, nudges the model to produce answers that score higher next time. The process described by the original OpenAI method runs in three steps: collect demonstrations and fine-tune, collect comparisons and train a reward model, then optimize the policy against that reward model.
One piece keeps the whole thing from falling apart. The reward model is only a proxy for human taste, and a model left to chase it will find cheap tricks that score well and read badly. So the optimization carries a penalty that keeps the tuned model close to the one it started from, measured as the distance between the two. The reward pulls the model toward preferred answers, the penalty holds it near sane language, and the balance between them is the craft.
The one line to keep
“Scale makes a model capable. Human feedback makes it useful.”
03 · The pipeline
Five stages, from raw model to assistant.
01
Pretraining
The base model learns to predict the next token across a vast body of text. It ends up fluent and knowledgeable, and willing to continue almost anything, helpful or not.
02
Supervised fine-tuning
People write example answers in the voice you want, and the model trains on them. This teaches format and manner, but it cannot cover every prompt a model will ever see.
03
Reward modeling
Labelers compare two model answers to the same prompt and pick the better one. Those choices train a separate reward model that scores any answer the way a person would.
04
Reinforcement learning
The model is optimized to produce answers the reward model scores higher, usually with PPO, while a penalty keeps it close to where it started so it does not drift into nonsense.
05
Online iteration
Some teams repeat the loop. Fresh answers are compared, the reward model is updated, and the policy is tuned again, so the standard keeps moving with real use.
The first two stages give a model manners. The last three give it judgment, by teaching it to prefer the kind of answer a person would have chosen. Most of the behavior you notice in a modern assistant was set in the final three.
04 · What it is not
Not plain fine-tuning, and not a system prompt.
Supervised fine-tuning shows the model examples of the right answer and trains it to copy them. It works when you can write the answer. RLHF works on the much larger set of prompts where you cannot write the perfect answer but you can tell which of two attempts is better. Judging is cheaper than authoring, and that is the whole point.
It is also not a system prompt. A system prompt steers a model at run time, in words, every time it answers. RLHF changes the weights once, in training, so the behavior is baked in. The two stack: a model aligned with human feedback still reads its standing instructions, and the prompt rides on top of the disposition that training gave it.
The ingredient that makes RLHF its own thing is the reward model. Fine-tuning has a fixed target for every example. RLHF learns the target from comparisons, then optimizes against that learned target. A model can be fine-tuned and then aligned, which is the usual order, and the second step is where tone, refusal, and restraint come from.
05 · One result that shows the point
A small aligned model beat a giant raw one.
OpenAI’s InstructGPT work, from Ouyang and colleagues, is the clearest case. In human evaluations on their prompt distribution, the authors report that outputs from the 1.3B parameter InstructGPT model were preferred to outputs from the 175B GPT-3, a model with about a hundred times more parameters. The small model was tuned with human feedback. The large one was not.
The same work reports that the aligned models made up facts less often and produced less toxic output, with only small regressions on standard benchmarks. OpenAI announced in January 2022 that these models, trained with humans in the loop, had become the default on its API. Anthropic reported the same shape of result the same year, finding that a helpful and harmless assistant trained this way was both more helpful and much harder to provoke than one tuned for helpfulness alone.
1 · collect demonstrations → supervised fine-tune
2 · collect A/B comparisons → train reward model
3 · optimize policy (PPO) → maximize reward,
minus drift from startThe three step RLHF pipeline · as described in the InstructGPT work
06 · The newer methods
You can skip the reward model. Sometimes.
The classic pipeline has moving parts: a reward model to train, a policy to optimize, and the reinforcement learning loop between them. In 2023, Rafailov and colleagues introduced Direct Preference Optimization, which showed that the same preference data can tune a model in a single supervised step. Their paper states that it eliminates the need for fitting a reward model, sampling from the model during fine-tuning, or heavy hyperparameter tuning, while staying stable and lightweight.
DPO and its relatives made alignment reachable for far smaller teams, since you no longer have to stand up a full reinforcement learning stack. The tradeoff is that DPO works on a fixed set of comparisons, while the classic loop can keep generating new answers and collecting fresh feedback as it goes.
For reasoning tasks, online reinforcement learning came back. Methods that generate many answers and reward the ones that reach a correct result now sit alongside preference tuning. The honest summary for 2026 is that there is no single method. RLHF named the idea, DPO simplified it, and online RL extended it, and serious products use whichever fits the behavior they are trying to shape.
07 · Why it is a product decision
The preferences are the product.
The reason alignment is not just a research topic is that someone has to decide what counts as a better answer. That decision is encoded in every comparison a labeler makes, and it becomes the model’s disposition. Who writes the guidelines, what they reward, and where they draw refusal lines are product choices with consequences a user feels on every turn.
It is also worth knowing the failure modes. The reward model is a proxy, and a model can over-optimize it, scoring higher while getting worse, which is why the drift penalty and continued evaluation matter. Labelers disagree with each other, so the data carries their mix of taste and bias. A reward model trained on narrow preferences will make a narrow assistant.
The operator takeaway is plain. Capability comes from scale and from the base model you choose. Usefulness, safety, and voice come from the feedback you train on. If the model that speaks for your product has a personality, a set of refusals, and a sense of what a good answer looks like, those did not arrive by accident. They were chosen, one comparison at a time.
Closing
A model does not decide what good is. You do.
At Bttr. this sits inside product engineering, full-stack systems built for performance and longevity. Before you blame a model for the wrong tone, look at the preferences it was trained on. That is the dial, and it is one you can turn.
Written by · November 2, 2026
Donny Smith · ECD, Founder, Bttr.
Over the past 15 years, he has led creative teams and contributed to products used by millions of people worldwide, working with companies including Apple, Opendoor, JP Morgan, GE Aerospace, Pepsi, and Alterra Mountain Company.
LinkedInFacts current as of November 2026 · OpenAI · Aligning Language Models to Follow Instructions · January 2022 · Ouyang et al. · Training Language Models to Follow Instructions with Human Feedback · 2022 · Christiano et al. · Deep Reinforcement Learning from Human Preferences · 2017 · Anthropic · Training a Helpful and Harmless Assistant with RLHF · 2022 · Rafailov et al. · Direct Preference Optimization · 2023
Share this perspective
More insights
Adjacent perspectives.
Bttr. Field Brief
The brief Bttr. writes for senior buyers.
Monthly. One signal worth your time on Brand Operating Systems, AI search visibility, and the infrastructure buildout. No filler.