The RAG problem nobody likes to admit

I've noticed a pattern in the RAG writing I read this week: nobody's excited about finding relevant chunks anymore. The interesting part is throwing most of them away.

Take LinkedIn's production search architecture. The ranker is a 0.6-billion-parameter model, and it handles over 100,000 queries a second. Not because a bigger model would be worse — it's that at that volume, small and fast wins by a mile. I think about that number every time someone tells me they need a bigger embedding model before they've actually tried a smaller one.

Kapa.ai's write-up on pruning RAG context is the same idea from a different angle. They put a small model between the retriever and the generator, and it drops 68% of the retrieved text while keeping 96% of the useful answer. If you're paying to stuff twenty chunks into a prompt so the model can actually use three of them, this is worth copying — it's a genuinely simple thing to bolt on.

Two pieces this week made me rethink what "retrieval" even means. BlitzGraph bills itself as "Supabase for graphs" — retrieval by relationship instead of similarity, for the cases where "these two things are connected" matters more than "these two things sound alike." And ByteByteGo's comparison of RAG, Graph RAG, and Agentic RAG is the clearest explanation I've seen of why those three aren't interchangeable — pick the wrong one and you're rebuilding in six months.

On the research end, an analysis of SPLARE swaps out the usual embedding approach for sparse autoencoders and beats SPLADE on the MMTEB benchmarks. Early, but a sign dense embeddings aren't the only game in town anymore.

The one I keep thinking about, though, is the story of a search model that quietly started ranking the CEO's memos above everything else. No error, no warning — it just returned the wrong thing, confidently, until someone happened to notice. That's the real risk in RAG. It doesn't crash. It just quietly gets worse.

If I only did one thing from this list, it'd be pruning context before generation. A 68% smaller prompt with 96% of the recall beats swapping embedding models, and you can ship it this week.