RAG Is Retrieval. Context Engineering Is Architecture.
Systems Intelligence · Issue 3 · August 6, 2026
You upgraded your retrieval.
Moved from keyword search to dense vector embeddings. Tuned your section sizes. Got the recall metrics where you wanted them. The AI still returns wrong answers in production.
The problem isn’t retrieval.
We hit this wall building our LinkedIn content pipeline: the agent rebuilt brand voice from scratch on every run, re-establishing positioning decisions it had already confirmed the session before.
There’s a version of this that gets sold continuously: smarter embeddings fix the context problem. Better sectioning, higher recall scores, tighter relevance windows, and the AI will stop producing wrong answers at exactly the wrong moment. It’s the natural conclusion after your first RAG implementation breaks in the demo-to-production transition. And it misidentifies the problem entirely.
Retrieval and context engineering are different problems with different solutions.
RAG answers one question: given this query, what exists in the knowledge base that’s relevant? That’s a search problem. Context engineering answers a different question: given this query, this session history, this token budget, and this agent’s current task: what goes into the context window, how much of it, and in what form?
Most teams have built the first system. Almost nobody has built the second.
The retrieval ceiling
The first place the gap shows up is inside retrieval itself.
Keyword search (BM25, TF-IDF) is precise about exact terms. Search for “token budget” and you get documents containing those words. Dense vector embeddings are semantic: they surface “context window limit” as relevant to “token budget” even when the words don’t match. Each method finds what the other misses.
Most RAG implementations pick one. Hybrid retrieval uses both: keyword for precision, dense vectors for recall. Then it re-ranks the combined results by relevance to the actual query.
That re-ranking step is the difference between retrieved relevant documents and retrieved the right passages in the right order for this specific query. Without it, you’re handing the context window a pile. With it, you’re handing it ranked answers.
Most teams stop here. They’ve built the first stage of a six-stage architecture and shipped the stage, not the system.
The memory problem
Run any standard RAG system through a multi-turn session.
Ask five questions across an hour. By turn three, you’ve established context: this user is asking about receipt processing, they’re on the vendor reconciliation step, they’ve confirmed the date range. By turn five, the retrieval is surfacing foundational context the session established three turns ago.
The context window is filling with things it already knows.
In our LinkedIn automation, that looked like re-reading brand voice guidelines and tone decisions already confirmed two runs earlier, not because retrieval failed, but because nothing was carrying that context forward.
Memory decay is the architectural fix. The mechanism: turns with roughly 72% query overlap with previous turns get deduplicated. The slot occupied by redundant context gets freed for genuinely new information. Each turn, the context window carries history that matters, not history that happened.
Without it, multi-turn sessions degrade. Not immediately. Gradually, across turns, the agent works from a context that thickens with repetition. It produces answers that were accurate two turns ago and aren’t accurate now, not because retrieval failed, but because the context window is carrying yesterday’s conversation into today’s answers.
This isn’t a retrieval problem. The retrieval is working exactly as designed. The architecture above it isn’t.
The compression problem
Even with hybrid retrieval, even with memory decay, you eventually hit the token limit.
Most systems respond by cutting. You retrieved N documents; the context window fits M. The last (N − M) documents get dropped. The selection criterion is position in a ranked list, not relevance to the current query.
Extractive compression works from the other direction. Every sentence across all retrieved documents gets scored by token overlap with the query. The highest-scoring sentences, the ones most directly answering what was actually asked, get selected first. Selection continues until the token budget runs out.
What enters the context window is the most query-relevant extraction from across all retrieved content. Not the first M documents. The right sentences from all of them.
The quality difference isn’t marginal. Truncation gives the model the right documents, cut arbitrarily. Extractive compression gives it the right passages, selected deliberately. You’ve moved from choosing documents to choosing evidence.
And none of this works if nobody made the architectural decision about what the token budget actually is.
Three streams, one diagnosis
These aren’t three separate problems. They’re three manifestations of the same missing layer.
Retrieval surfaces what exists. Re-ranking orders it by relevance to the actual query. Memory decay prevents the context from accumulating dead weight across turns. Extractive compression ensures what survives into the context window is the most relevant evidence, not an arbitrary slice of a ranked list. Token budget control sets the ceiling everything else operates within.
Five decisions. All of them happen between “retrieval ran successfully” and “the model received its context.” Most teams have made one, maybe two.
Context engineering is the discipline of making all five, deliberately, before the agent runs.
The full pipeline: Retriever → Re-ranker → Memory decay → Extractive compression → Token budget control → Context window.
This architecture adds latency. Embedding generation alone adds roughly 85ms on CPU before a single token generates. For a single-turn query against a stable, bounded knowledge base (a documentation search, a legal corpus lookup), a well-tuned RAG system is sufficient. The five-stage stack is overhead you don’t need for that problem class.
But for agents running multi-turn sessions, operating on dynamic knowledge bases, working within bounded token budgets, and returning accurate answers when retrieved content exceeds what the context window holds, RAG is stage one. Context engineering is the architecture that makes the remaining five stages possible.
The question your architecture hasn’t answered
Retrieval answers: what exists?
Context engineering answers: what’s useful, right now, given what the system knows about this session, this query, and this constraint?
These are different questions. They require different architectural decisions. One is a search problem. The other is an engineering discipline most AI systems never touch.
If your AI produces wrong answers after your retrieval layer improved, you’re not looking at a retrieval failure. You’re looking at what happens when you build one stage of a six-stage system and ship it as the whole thing.
The question worth sitting with: who in your architecture made the decision about what gets into the context window?
If the answer is nobody, the decision was made by default. And defaults are what production failures are made of.
What’s the most expensive context you’re currently rebuilding every session? Reply with it. There’s a pattern worth naming.
Good context architecture makes agents reliable inside a session. Sessions still don’t compound. Next issue: why the composable stack has no hub, and why that’s the ceiling on everything else you’re building.
Systems Intelligence covers AI architecture for operators who need it working, not just running. New here? Subscribe.