October 2, 2026 · 8 min read

What Is Reranking

A retrieval system usually finds the right passage and then leaves it in the wrong place. The first stage is a bi encoder that turns the query and every document into separate vectors and compares them, which is fast but never reads the two together, so the best passage often lands in the middle of the list. Reranking is the second pass that fixes the order. A cross encoder reads the query and one candidate as a single input, scores how they actually relate, and reorders the short list so the most relevant passages rise to the top. It is slower because it computes a fresh score for every query and document pair, so it runs on about a hundred candidates, never the whole corpus. On the public MS MARCO benchmark the sentence transformers cross encoder models trade speed for accuracy across sizes, from a tiny model near 9,000 documents a second to larger ones a fraction of that. Anthropic measured the payoff on a real pipeline: retrieve 150 chunks, rerank, keep the top 20, and the failure rate fell 67%. Retrieval finds the passage. Reranking makes sure the model reads it first.