How a retriever actually finds the right document

I ship real systems for a living, and lately I've been going deep on how RAG actually works. The thing that caught me off guard: the piece that decides whether you get a good answer isn't the model at all.

Everyone talks about the model. The prompt, the answer, which LLM is smartest this month. But in a real RAG system, the part that decides the answer runs before the model wakes up. It picks which documents the model is even allowed to see. Get it wrong and the smartest model on earth still answers badly, because it never saw the right page.

That part is called the retriever. I spent the last couple of weeks pulling it apart, one search at a time. Here's how it actually works, start to finish.

Start with a librarian#

You walk up to a librarian and ask, "how do I make New York style pizza at home?"

They don't read every book on the shelf. They understood what you meant and walk straight to the three books that matter.

A retriever does that over your documents. Under the hood it scores every document against your question and hands back the top few. Easy to say. Scoring them well is the whole job. And a good retriever doesn't run one search to do it. It runs two at the same time, then filters the results, then merges them. Let's take those one at a time.

Search one: the exact words#

The oldest trick in search is just counting words. No AI, no embeddings. Which document shares the most words with your question?

You start dumb. Take the question, take each document, throw away word order. All that's left is which words appear and how many times. Then for each word in your question, give a point to every document that contains it. Most words in common wins.

Two things fix the obvious flaws.

First, long documents cheat. A giant page has your words just because it has every word, so you divide by the document's length. Now it's the share of the text, not the raw count.

Second, and this is the one that matters, not all words are worth the same. Think about a lost bag at an airport. "It's black, it's big, it has a bright orange Pikachu keychain." Nobody searches on black or big. Every bag is black and big. You go straight for the Pikachu keychain, because almost none of them have it. Search does the same move: it counts how many documents each word appears in, and weights the rare ones higher. Match on "the" and you've learned nothing. Match on "pizza" and you've probably found the page.

Put those two halves together and you get TF-IDF, a decades-old formula that's still the baseline every fancy search engine gets measured against. The smartest thing about counting words isn't the counting. It's knowing which words to ignore.

There's one more upgrade worth naming. Plain counting is greedy: a page that says "pizza" 50 times looks way better than one that says it 5 times. In real life it isn't. After a few mentions you already know the page is about pizza, the rest is noise. The modern version, BM25, bakes that in. Each extra match still helps, but less than the one before it. Search got better the day it learned to stop counting so eagerly.

Search two: the meaning#

Keyword search has a wall. Ask about "cars" and it misses the document that only ever says "vehicles". Same idea, different words, zero shared terms.

The second search fixes that. It turns every piece of text into a point in space, placing similar meanings close together, and looks for the points nearest your question. That's semantic search.

Here's the part that surprised me: nobody teaches those models what words mean. Picture a huge party where everyone stands in a random spot. You walk around with one rule. Two people with a similar vibe, get closer. Two people with nothing in common, back away. Repeat it millions of times and the room sorts itself. The sports crowd drifts to one corner, the food people to another. Nobody assigned the corners. They just formed. That's how an embedding model learns meaning, by contrast, pulling similar text together and pushing different text apart. Meaning isn't a place. It's who you end up standing next to.

Two quiet gotchas live in here, and both bite people wiring up their first AI search.

One: "similar" doesn't mean "nearby". The points live in hundreds or thousands of dimensions, and in that many dimensions everything is far from everything, so raw distance stops telling you much. What survives is direction. Two ideas that mean the same thing point the same way, even if the line between them is long. So the system measures the angle between points, not the gap. Similar isn't close. Similar is aligned.

Two: you can only compare points made by the same model. Each embedding model builds its own space from scratch, like two people drawing a map of the same city, one centered on the town hall, one on the train station. Both maps are correct. Read a coordinate off one and point to it on the other and you land in the wrong place. Embed your documents with one model and your query with another and you get confident, ranked garbage, with nothing erroring out to warn you. Match the model on both sides. It's the boring rule that saves you a day.

The bouncer at the door#

Each search hands back 20 to 50 documents. Before anything gets ranked, there's a filter, and it barely looks at your question.

It checks who you are. Not a paid subscriber? Paid articles get cut before ranking even starts. In a different region? You only see documents for your region. These are hard rules at the door. It's a metadata filter, and at this stage content doesn't matter yet. Who's asking does.

Merging two lists into one#

Now you have two rankings, one from words and one from meaning, and you have to act on one list, not two.

The obvious move is to compare their scores. Don't. One search counts matching words, the other measures closeness in meaning, and those numbers live on scales that mean nothing side by side. So the standard trick throws the scores away and keeps only the order.

Picture two friends each handing you their top 10 restaurants. Turn each rank into points: 1st is 1 point, 2nd is half a point, 3rd is a third, on down the list. Add up each place's points across both lists. A spot that's 2nd on one list and 10th on the other scores 1/2 + 1/10 = 0.6. A place both friends rank near the top beats a place only one of them loved. That's reciprocal rank fusion. It never checks how much either friend liked the place, only where it landed. Position is the shared language. The numbers behind it aren't.

The dial nobody touches#

There's one knob most people never turn: how much each of the two searches gets to vote. In search it's called beta.

Think about hiring for a role. One reviewer scores the resume on exact keywords, does it literally say "Rust", "Postgres", the exact stack. The other reads between the lines for whether the person actually gets it. You decide whose vote counts more.

A common starting point is 70% meaning, 30% words. Why not 100% meaning? Because exact words matter more than people expect. Names, error codes, product IDs, rare jargon. Meaning-search blurs those together, "ERR_502" and "ERR_503" look almost identical to it, and the word-match search knows they're different. So the right split depends on your data. Legal docs, part numbers, code, lean toward words. General questions written the way people talk, lean toward meaning. There's no universally right setting. 70/30 is a starting guess, not an answer.

The tension nobody warns you about#

One question hangs over the whole thing: how many documents do you return?

Too many and the real answer drowns in noise and blows your context window. Too few and you drop the document that ranked 2nd or 3rd and was actually right. There's no fixed number that works. You tune it and watch it over time.

Keyword search, semantic search, a metadata filter, a merge, and a dial. Together that combo has a name: hybrid search. And a retriever running it is never "done". That's most of the work in a real RAG system, and all of it happens before the model writes a single word.

If you're building with RAG, which way do you tune it, toward missing a few good documents or toward handing the model some junk?