43 DAGs. 9 batch branches. One migration window.

Here’s what those numbers looked like before the architecture existed.

5–6 hours per DAG. Done manually by engineers carrying institutional knowledge that disappeared at the end of every sprint. A 15–20% defect escape rate, which means 1 in 6 DAGs shipping with a bug the review process missed. Documentation coverage at 30–40%: the migration decisions were being made correctly, and then forgotten. By the next engineer who touched the same codebase, by the next batch that ran against the same patterns, by the next sprint that inherited the same technical debt without knowing why it existed.

Not a rough draft. That’s a production migration with experienced engineers doing careful work.

Here’s what the system produced.

8–10 minutes per DAG. Documentation coverage at 100%, generated as a byproduct of the validation pass, not as a separate sprint. 43 DAGs across 9 batch branches. Every defect caught in a local sandbox before anything moved to a shared environment. The only issues that surfaced downstream were data problems, not migration defects.

That’s what zero escapes means here. Not that nothing ever broke. That nothing broke somewhere expensive.

The numbers are the proof. But the numbers only make sense once you understand what the architecture was catching.

Two bugs passed every static review layer. Both got caught.

The first: a YAML syntax error. Duplicate key. The Developer didn’t flag it. The Manager reviewed the code and cleared it. QA ran the command. The parser rejected it. Ninety seconds.

The second: an undocumented legacy pattern, a deprecated module reference that wasn’t in the transformation table. The Developer applied the standard transformation. At runtime, ModuleNotFoundError. QA flagged it. The rule got added to the skill document. Every subsequent DAG with that pattern was caught before it ran.

Neither is a model failure. Both are architectural catches. Static analysis tells you what code says it does. Runtime tells you what it actually does. When only one runs, there’s a guaranteed gap. Not in the model. In the structure.

The math across the full migration window: roughly five engineer-hours back per DAG, and a 15–20% defect escape rate going to zero. Across a batch of three, that’s about two working days. At 50 DAGs it’s around 250 hours, call it six weeks of one engineer, and a migration that runs 4–6 weeks faster.

None of this is free.

The skill document took multiple sessions to build: 18-pattern transformation table, prohibited patterns list, 40+ item validation checklist. The Manager orchestration layer required real engineering work. The Docker QA environment had to be configured and maintained. That work happened once. Every DAG that ran through the system after paid down the investment.

The demo-first pitch describes the best case, not the production case: agent count, speed claims, impressive output. Build the validation architecture first, or you’re shipping the demo. The system works because of the engineering that happened before it ran, not despite it.

Now the part I didn’t expect.

After every DAG the batch stopped and waited for me. I read the diffs in the editor, read the status document, read the error logs when there were any. I watched the Manager work while it worked, so I saw it catch the Developer’s mistakes and QA’s failures in something close to real time.

I did find things it missed. Not many, but some. And here’s what I did about almost all of them: I had the Manager confirm the problem and fix that DAG, and then I put the rule in the skill document.

Common mistakes. The checklist. The prohibited patterns.

Because the interesting thing about the misses is what caused them. It usually wasn’t the agent being wrong. It was a DAG written by one engineer in a house style that differed from another engineer’s, in a way the transformation rules hadn’t anticipated. A codebase with a decade of individual habits in it.

Fix the DAG and you’ve fixed one DAG. Write the rule and DAG 30 can’t reproduce it.

So the gate’s real output wasn’t caught bugs. It was a skill document that got harder to fool every week. The system that migrated DAG 40 was measurably better than the one that migrated DAG 4, and the thing that improved it was a human reading diffs and writing down what he learned.

Take the human layer out and you don’t just lose a safety net. You lose the only part of the system that was getting smarter.

The architecture is not migration-specific. But the precondition is easy to miss, and it’s the thing that decides whether any of this transfers.

Both bugs were caught mechanically. A parser rejected one. A runtime threw on the other. Neither catch required judgment. That’s what lets the layers stay independent: QA can return pass or fail without an opinion, so nothing has to be negotiated between agents.

Where nothing can return pass or fail on its own, the third layer isn’t a guard. It’s another opinion, and Episode 2 covered what happens then.

So the question for your domain is whether that check exists. Schema normalization across legacy systems has it: constraint violations and type checks decide it. Code review has it for correctness, and not for design. Compliance documentation has it for form and not for substance, which is the harder half and the one worth being careful about. Contract extraction mostly doesn’t, and I’d want to solve verification before pointing this at it.

The architecture is the same. The skill document is specific to the domain. Whether a machine can settle the question is what decides if either one matters.

The architecture solves the structural problem: validation integrity across a batch transformation where one agent can’t see the full scope of its own work. That problem is solved.

The second problem is the one the gate exposed. The skill document is what makes any individual agent reliable before it ever reaches validation, and it compounds everything downstream. A strong one means the Developer’s transformations are correct before QA sees them. A weak one means QA is doing remediation instead of validation. The knowledge layer is what the system runs on, and mine only got better because a person was sitting there writing it down.

That’s what the newsletter is on right now: the AI capital you’re building, and who actually owns it. Then what it takes to make that knowledge compound instead of reset every session.

The teams shipping zero escapes at scale didn’t find a better model. They decided validation was architecture, and built it before the agents ran.

Two paths from here.

If you’re the one accountable for a migration not shipping bugs: which of the four layers would break first in your stack, the Developer, the Manager, QA, or the human gate? Leave a comment with which one, and what it’s guarding now.

If you’re mapping this onto your own batch transformation: comment with the domain, and I’ll work through whether the mechanical check exists for it.

Not the voice calibration that took weeks of prompts to get right. Not the domain context your AI absorbed from your real projects. Not the communication patterns it learned: the ones that make your outputs sound like you wrote them.

Nate’s Newsletter put a name to it: AI working intelligence. The AI that knows your voice, your projects, your pushback patterns, the way you need complexity framed for different stakeholders. Six months of serious work builds something qualitatively different from a fresh account.

Last month my receipt pipeline processed seven Lowe’s purchases. It extracted vendor, amount, date and line items from every one of them at full confidence. No manual review flagged, nothing ambiguous.

It recorded which property the money was spent on for zero of them.

Not because extraction failed. The field is hardcoded to null and deferred to whoever reviews the queue later. Every receipt carries the same store number, so there is nothing to infer from. And the rules table that is supposed to learn my vendors has been live four months and contains no rules at all.

Four months, a working engine, and not one rule in it. Writing rules is a task with no forcing function. It never blocks anything, so it never happens.

So I tried to fill it in myself, a month later. I got six of seven right. For the seventh I confidently named a property that had nothing to do with it. One that isn’t even in the system.

I had not forgotten. I had lost the ability to know something was missing.

I had already fixed this. When we added photo receipts, we put a property selector on the capture form, because the person standing at the register is the only one who knows. That fix landed on the channel I happened to be testing. The channel that runs by itself never got it, and that is the one processing everything.

Mine was one empty field. Most operators don’t meet their version of it until they switch tools.

What AI working intelligence actually is

Here’s what makes this different from losing a file. AI working intelligence has four components: domain knowledge absorbed from your field, communication patterns tuned to your voice and register, workflow preferences built up over months of use, and behavioral calibration. Professional capital that compounds with use.

Almost all of it is distributed across tools you can’t merge, stored in accounts you don’t control, and owned by companies whose terms of service are clear about what belongs to them.

Where it goes

There are four moments when AI working intelligence evaporates. None of them are edge cases.

The first is a tool switch. You move to a different platform because of a price change, a feature gap, or a directive from above. Every calibration, every trained pattern, every context stays behind.

The second is an enterprise mandate. IT rolls out an AI policy. Your account is disconnected. You start from zero inside a new environment with none of the context you built.

The third catches people off guard. You move to a new role. You can’t take your prompts. You definitely can’t take your account context. Six months of professional capital stays at the company you left.

The fourth is the subtlest, and Nate’s Newsletter named it: QuitGPT migration. Your company procures the enterprise tier. Your personal account stays personal, in a different data silo. The accumulated intelligence doesn’t transfer.

This isn’t a design flaw. It’s a deliberate structure. None of the major platforms have built the layer that would let your accumulated intelligence leave. The capital is real. The ownership isn’t yours.

The architecture that changes this

The fix has a name. Nate calls it the Open Brain: a Postgres database, connected via MCP, that creates a portable context bundle. Domain knowledge, communication patterns, workflow preferences, calibration. All stored in a layer you control, not in an account you rent.

When the tool changes, the context doesn’t. The AI you connect to tomorrow inherits the capital you built today.

This is what the hub’s memory layer is built to do. The file-based store running now is the starting version: what a portable context layer looks like before it has a database underneath it. The Supabase layer that makes it persistent, queryable, and tool-agnostic is in design and testing on our own infrastructure, with a set of enhancements still ahead of it. It is deliberately not in production.

That sequencing is part of the argument, not a caveat to it. The layer that holds everything your organization knows is the last thing you should ship on someone else’s schedule.

The infrastructure isn’t exotic. Postgres + MCP is buildable with available tools. What’s missing for most operators isn’t the technology. It’s the architectural decision to treat AI working intelligence as capital worth owning, not as a convenience that lives in a platform you tolerate.

The AI tools available to you will keep improving. The intelligence you build with them compounds the whole time.

The question is whether that compounding belongs to you or to the platforms it runs on.

For most operators right now, the answer is the platforms. Not because they chose that. Because nobody made the architectural decision to own the layer where the capital lives.

Which of the four layers costs you the most when you switch tools or start a fresh session: domain knowledge, communication patterns, workflow preferences, or behavioral calibration? Reply with one sentence. I’m mapping where operators lose the most ground.

Next issue: the Karpathy model for AI knowledge that compounds instead of resets, and what it takes to build in practice.

Last week on Systems Intelligence

Issue 4: The Composable Stack Has No Hub covered why the tools compose but the knowledge doesn’t, and what a hub does that a spoke can’t. This issue asks the harder version of the same question: the capital accumulating in that hub, whose is it?

Systems Intelligence covers AI architecture for the Pragmatic Operator: builders who need AI that works, not AI that demos. If this resonated, forward it to someone whose AI context lives in an account they don’t control. If you’re new, start with Issue 1.

— Jerry Shields · systemsintel.dev

We didn’t build a smart AI agent.

We built four layers that can’t trust each other.

That’s what makes it work.

When you hit your first agent reliability problem, the obvious answer is more context. Give the agents better information: what was done before, what decisions were made, what the others found. Coordination means shared knowledge. More shared knowledge means fewer mistakes.

That’s true for most tasks. For a review sequence, it’s the wrong trade.

I know it’s the wrong trade because I built the other version first.

The first architecture had a dedicated Code Reviewer agent. A peer to the Developer: one writes, one reviews, clean separation of duties. It’s how the org chart looks on paper, and it’s the design most people reach for.

It didn’t last long.

The reviewer shared context with the work it was reviewing. And a reviewer that already knows what the developer intended is not reviewing the code. It’s confirming a story it was told. The review came back agreeable, thorough, and structurally unable to disagree. Every finding traced back to something the Developer had already flagged itself.

So the review moved up a level, to the Manager that was orchestrating the batch. Which, on reflection, is where it sits on every engineering team I’ve worked on. The lead or the manager owns code review. Not because they’re a better engineer than the person who wrote it. Because they’re the one accountable for what ships, and their read isn’t anchored to the reasoning that produced it.

The fix wasn’t a better reviewer. It was a reviewer standing somewhere else.

The Manager reads artifacts, not reports.

That distinction is the whole design. When the Developer finishes, it produces a review document: a complete account of what was changed, what was validated, what decisions were made. The Manager reads that document and checks the code itself. It never asks the Developer how it went. An agent saying “everything looks good” is not a gate. The reviewed file is.

Here’s what that bought. The Developer ran Batches 6 and 7 with clean context, no memory of prior DAGs, no carryover from run to run. It applied a correct transformation. Inconsistently, across both batches, because its context window didn’t span the work it had already done.

The Manager’s did.

The Manager reviewed Batch 7 independently and flagged the config pattern inconsistency introduced across both runs. One read. One catch. One fix, before any of it left the sandbox.

The agent doing the work could not see the shape of its own output. The layer above it could.

Four layers. Four failure modes. No overlap.

Developer transforms fast, documents everything, doesn’t approve its own work. Its context is the skill document and the current DAG. Nothing from the run before. Blind spot: it can’t see across its own batches.

Manager reads the artifact and reviews the code against the transformation rules. Static analysis: what the code says it does. Doesn’t run it. Blind spot: errors that only appear under execution.

QA doesn’t read the Manager’s review before testing. Runs the output. Imports the DAG. Checks the task graph. What fails at runtime doesn’t care what the review said. Blind spot: it can only judge what runs, not whether the transformation logic was sound.

The human gate sees all of it and none of it in advance. After every DAG the batch stops and waits. Blind spot: attention, and the fact that a person reviewing the twentieth diff is not the person who reviewed the first.

Each layer is designed to be wrong in a way the others can’t be.

What makes the handoffs work without shared context is the artifacts.

The Developer writes the review document. The Manager reads it directly. QA can reference it if something fails, but it doesn’t read it before running the tests. It runs first, then consults the record if something breaks.

The artifacts serve the record. The clean contexts serve the integrity.

Give QA the Developer’s report in advance and the review changes. It starts from “everything looks right” and works backward from approval. The YAML duplicate key, the one that passed two rounds of static analysis, gets caught precisely because QA doesn’t know it was already approved. It runs the command. The parser rejects it. Ninety seconds.

When QA fails, the Manager decides what happens next. Small problems it fixes itself. Larger ones get a Bug-Fix agent spawned with a narrow brief: here is what failed, here is what was already done, change only what’s needed. It never re-migrates the DAG and it never reads the original source. And there’s a ceiling. Three QA runs on one DAG and the batch stops for a human, because an agent looping on the same error is not making progress, it’s producing motion. Across the whole migration that ceiling was never reached. One DAG needed a second iteration. That was the worst of it.

The common instinct when something breaks is to add context. Give agents more information about what each other found. Tighter coordination. Better handoffs.

For a review sequence, the fix runs the other direction.

The independence is what gives each gate real teeth. Any process that validates its own work is a single point of failure. The distrust is the load-bearing wall. Not a design quirk to optimize away, but the mechanism that closes the gap between what code says and what systems do.

Static analysis shows intent. Runtime shows truth. When only one runs, there’s a guaranteed gap. Not in the model. In the structure.

That structure holds for three DAGs. What it looks like across 43, and what the human layer actually produced over that distance, is Episode 3.

One question before then: what does your review process take on trust?

Comment with where it shows up. I read every one.

Code review passed. Two agents approved it. The DAG would have failed on import anyway.

That’s not a model failure. That’s a structural one. It took a third agent, running a single command, to understand the difference.

We were three DAGs into a 43-DAG Airflow migration from Azure to Google Cloud. Each DAG moved through the same sequence: a Developer agent transformed the code, a Manager agent reviewed it independently, a QA agent ran it. The Developer and the QA agent spawned fresh each time: clean context, no memory of the agents before them, no assumption carried forward from prior runs.

DAG 3 was an orchestration file. More complex than the first two: more task dependencies, more operator configurations, more moving parts. The Developer wrote it and produced a review document listing every change it made. The Manager read that document and checked the code against the transformation rules directly. Task operators confirmed, dependency chains traced. High confidence. Approved.

Then the QA agent ran one command: airflow dags list-import-errors.

Error. YAML: duplicate key.

A duplicate key in a mapping node. The kind of thing you can read ten times and not catch, because nothing in the visual layout signals it’s wrong. But YAML parsers don’t read files. They process them. And a duplicate key is a hard failure: the DAG doesn’t load, doesn’t run, doesn’t exist as far as Airflow is concerned.

The fix took 90 seconds. The QA agent identified the error, the Manager confirmed it and corrected it, the command re-ran clean. The batch moved forward.

Two agents approved that configuration. One of them exists specifically to review the other’s work. Both did exactly what they were designed to do. Neither of them could have caught this.

The Developer wrote code. The Manager read code. Both doing the same kind of work: static analysis. Reading the file, checking structure and logic against a set of expectations. What neither of them did, what neither of them could do inside their role, was run it.

YAML syntax errors don’t exist in the reading. They exist in the parsing.

You can read a duplicate key and not see it. The parser sees it immediately. The Manager, running the most careful static analysis possible, is structurally incapable of catching a class of errors that only appear under execution. More review of the same kind wouldn’t have changed the outcome. More diligence wouldn’t have changed the outcome. Only a different mode of verification could.

The QA agent isn’t a better reviewer. It does a different thing: runtime validation. One command: load the DAG, report what Airflow actually processes. Not what the code says. What the system sees.

There are two modes of verification, and most processes use only one.

Static analysis is fast, scalable, good at logical errors, structural problems, and known anti-patterns. It evaluates what’s visible in the code. It has a ceiling: it can only see what exists before execution.

Runtime validation runs the output. Parse errors, import failures, configuration problems that only appear when the system actually processes the file. An entire category of error that static review cannot see, structurally, ever.

When both modes run, they cover each other’s blind spots. When only one runs, there’s a guaranteed gap. Not in the model. In the architecture.

The alternate timeline isn’t hypothetical. A YAML error in an Airflow DAG takes the entire batch offline. With 43 DAGs queued, a failed batch means re-queuing, re-running, emergency triage inside a migration window that doesn’t have slack in it. Everything here was caught in a local sandbox, before any of it reached a shared environment. That was the point of running it this way.

“Code review passed” is the most useless thing anyone can say in that moment.

The architecture is built around that blind spot. Not because any individual agent is weak, but because the structure provides what no single agent can: two modes of verification, running independently, neither able to inherit the assumptions of the one before it.

The QA agent doesn’t know the Developer wrote the file. It doesn’t know the Manager approved it. It runs the output and reports what the system sees. That independence is the feature. Give it the prior review and you introduce confirmation bias: the tendency to find what the last pass found and miss what it missed.

Deliberate distrust is the design.

Static analysis shows intent. Runtime shows truth.

There’s a layer above all three. After every DAG, the whole thing stops and waits for a human to say continue. That gate is not a formality, and what it produced was not mostly bug catches. Episode 2 is the architecture underneath it, including the version of it we built first and had to throw away.

One question before then: what does your review process actually run, and what does it only read?

Comment with what you’re running. I read every one.

In the last days of March, OpenAI published a Codex plugin that runs inside Claude Code. Apache-licensed, built to operate as a subagent, shipped into a rival’s environment with no partnership and no announcement. Three days later, Cursor shipped an agent-first workspace: every running agent, local and cloud, across every repo, in one pane.

The coverage called it convergence. That’s not right.

Convergence means the tools are becoming one. What actually happened is each tool settled into a layer. Cursor took orchestration, managing fleets of agents across repos and environments. Claude Code took reasoning-heavy execution. Codex took code generation at scale. The pattern mirrors how infrastructure tooling matured a decade ago: Prometheus for metrics, Grafana for visualization, PagerDuty for alerts. Specialized tools, composable by design.

If you’ve been building this way, using different tools for different functions and resisting the urge to find one that does everything, the market just validated your instincts.

Then in June, Cursor announced it was pre-training a 1.5-trillion-parameter model from scratch. Not a fine-tune of somebody else’s base weights. A full training run, owned end to end. It still routes to Claude and GPT and Gemini; the new layer goes underneath rather than closing the doors. The reason given was dependency: the labs can change pricing, rate limits, and context behavior at any time, and everything downstream absorbs it.

Read that as a business decision and it’s unremarkable. A company reducing supplier risk.

Read it as an architecture decision and it’s the tell. A layer inside the composable stack looked at its own position and concluded that composing into someone else’s stack was not a durable place to stand.

The vendors won’t accept being a pure spoke. Worth asking why you would.

The composable stack solves horizontal composition. Tools talking to tools. Data flowing between them. Specialized layers doing their specific jobs.

What it doesn’t solve is vertical compounding.

Open a session in any of those tools tomorrow morning. It doesn’t know what you built last week. It has no memory of the architectural decision you spent three hours on Tuesday, no record of the vocabulary your team uses for your domain, no understanding of the non-negotiables your most important client put in writing six months ago. The tool didn’t forget. It never knew; there’s no layer in the composable stack designed to accumulate.

The tools compose. The knowledge doesn’t.

This isn’t a flaw in any of the tools. It’s an architectural gap. The composable stack is excellent at routing: data flows in, gets processed, flows out. Routing and accumulating are different functions, and the composable stack was built for the first one.

Consider what happens without the second. Every session, you rebuild context. Re-explain the domain. Re-establish what’s in scope. Re-surface decisions that already got made. You do this not because you forgot. The stack did. The overhead feels manageable at first. Four months in, it’s a tax on every session. Every person on your team is rebuilding context independently, inconsistently, from whatever happens to be in their head that morning.

The tools got faster. The knowledge didn’t accumulate.

What a hub does is structurally different from what a spoke does.

A spoke executes. It’s deterministic, auditable, reliable. It does the same thing every time. That’s the feature. Spokes are where the work happens.

A hub accumulates. It holds what the organization knows: brand voice, business rules, domain vocabulary, architectural decisions, team patterns. The knowledge that takes months to develop and used to live exclusively in a few people’s heads. Active context that shapes every decision the spokes make. It doesn’t need to be found. It’s already there.

The relationship is bidirectional. Spokes route data into the hub: inputs, events, results. Spokes route instructions out: approved actions, formatted outputs, triggered workflows. The hub holds the judgment that decides what the spoke does with what it receives.

Remove the hub, and the spoke still executes. It just executes without judgment.

More spokes don’t solve this. More tools in the stack don’t solve it either. Routing data in more directions doesn’t create a layer that accumulates. Confusing the two is how you end up with excellent infrastructure that doesn’t get smarter over time.

The operators who build a hub compound. The stack gets smarter as the hub grows. New tools plug into a center that already understands the business. Context isn’t rebuilt at the start of every session; it’s updated.

The operators who connect tools without a hub are on a performance treadmill. Better models, faster outputs. But the knowledge ceiling stays where it was. The advantage built with the AI stack is as durable as the last session, which is to say, not durable at all.

This is what “memory as moat” means in practice: compounding return on organizational knowledge that only accrues when there’s a center for it to accumulate in.

The composable stack validates the spokes. The moat is in the hub.

The composable stack event is real, and it matters. Three tools composing without coordination is the market telling you the right architecture is layered and specialized, not monolithic. That part is settled.

What the event didn’t settle: what lives at the center.

Spokes route data in. Spokes route instructions out. The hub is where it compounds.

The market built the first two. The third isn’t in any of the tools that composed this spring, and it wasn’t in the one that spent a pre-training budget trying to build its way to solid ground. That’s not a technical gap. A hub isn’t a tool. It’s a decision about what your organization is going to invest in building, maintaining, and feeding over time.

The operators who answer that question are building something their tools can’t replicate on their own.

The stack is now available to everyone.

The compounding is not.

What’s your current hub? Or are you operating without one? Reply with what you’re using.

The hub solves accumulation. The harder question: does the knowledge you’re accumulating belong to you? Next issue: the AI capital you’re building right now, and who owns it.

Last week on Systems Intelligence

Issue 3: RAG Is Retrieval. Context Engineering Is Architecture. covered the five architectural decisions most teams never make above the retrieval layer. Good context architecture makes an agent reliable inside a session. This issue is about what survives between sessions.

Systems Intelligence covers AI architecture for the Pragmatic Operator: builders who need AI that works, not AI that demos. If this resonated, forward it to someone assembling a stack. If you’re new, start with Issue 1.

— Jerry Shields · systemsintel.dev

Moved from keyword search to dense vector embeddings. Tuned your section sizes. Got the recall metrics where you wanted them. The AI still returns wrong answers in production.

The problem isn’t retrieval.

We hit this wall building our LinkedIn content pipeline: the agent rebuilt brand voice from scratch on every run, re-establishing positioning decisions it had already confirmed the session before.

There’s a version of this that gets sold continuously: smarter embeddings fix the context problem. Better sectioning, higher recall scores, tighter relevance windows, and the AI will stop producing wrong answers at exactly the wrong moment. It’s the natural conclusion after your first RAG implementation breaks in the demo-to-production transition. And it misidentifies the problem entirely.

Retrieval and context engineering are different problems with different solutions.

RAG answers one question: given this query, what exists in the knowledge base that’s relevant? That’s a search problem. Context engineering answers a different question: given this query, this session history, this token budget, and this agent’s current task: what goes into the context window, how much of it, and in what form?

Most teams have built the first system. Almost nobody has built the second.

The retrieval ceiling

The first place the gap shows up is inside retrieval itself.

Keyword search (BM25, TF-IDF) is precise about exact terms. Search for “token budget” and you get documents containing those words. Dense vector embeddings are semantic: they surface “context window limit” as relevant to “token budget” even when the words don’t match. Each method finds what the other misses.

Most RAG implementations pick one. Hybrid retrieval uses both: keyword for precision, dense vectors for recall. Then it re-ranks the combined results by relevance to the actual query.

That re-ranking step is the difference between retrieved relevant documents and retrieved the right passages in the right order for this specific query. Without it, you’re handing the context window a pile. With it, you’re handing it ranked answers.

Most teams stop here. They’ve built the first stage of a six-stage architecture and shipped the stage, not the system.

The memory problem

Run any standard RAG system through a multi-turn session.

Ask five questions across an hour. By turn three, you’ve established context: this user is asking about receipt processing, they’re on the vendor reconciliation step, they’ve confirmed the date range. By turn five, the retrieval is surfacing foundational context the session established three turns ago.

The context window is filling with things it already knows.

In our LinkedIn automation, that looked like re-reading brand voice guidelines and tone decisions already confirmed two runs earlier, not because retrieval failed, but because nothing was carrying that context forward.

Memory decay is the architectural fix. The mechanism: turns with roughly 72% query overlap with previous turns get deduplicated. The slot occupied by redundant context gets freed for genuinely new information. Each turn, the context window carries history that matters, not history that happened.

Without it, multi-turn sessions degrade. Not immediately. Gradually, across turns, the agent works from a context that thickens with repetition. It produces answers that were accurate two turns ago and aren’t accurate now, not because retrieval failed, but because the context window is carrying yesterday’s conversation into today’s answers.

This isn’t a retrieval problem. The retrieval is working exactly as designed. The architecture above it isn’t.

The compression problem

Even with hybrid retrieval, even with memory decay, you eventually hit the token limit.

Most systems respond by cutting. You retrieved N documents; the context window fits M. The last (N − M) documents get dropped. The selection criterion is position in a ranked list, not relevance to the current query.

Extractive compression works from the other direction. Every sentence across all retrieved documents gets scored by token overlap with the query. The highest-scoring sentences, the ones most directly answering what was actually asked, get selected first. Selection continues until the token budget runs out.

What enters the context window is the most query-relevant extraction from across all retrieved content. Not the first M documents. The right sentences from all of them.

The quality difference isn’t marginal. Truncation gives the model the right documents, cut arbitrarily. Extractive compression gives it the right passages, selected deliberately. You’ve moved from choosing documents to choosing evidence.

And none of this works if nobody made the architectural decision about what the token budget actually is.

Three streams, one diagnosis

These aren’t three separate problems. They’re three manifestations of the same missing layer.

Retrieval surfaces what exists. Re-ranking orders it by relevance to the actual query. Memory decay prevents the context from accumulating dead weight across turns. Extractive compression ensures what survives into the context window is the most relevant evidence, not an arbitrary slice of a ranked list. Token budget control sets the ceiling everything else operates within.

Five decisions. All of them happen between “retrieval ran successfully” and “the model received its context.” Most teams have made one, maybe two.

Context engineering is the discipline of making all five, deliberately, before the agent runs.

The full pipeline: Retriever → Re-ranker → Memory decay → Extractive compression → Token budget control → Context window.

This architecture adds latency. Embedding generation alone adds roughly 85ms on CPU before a single token generates. For a single-turn query against a stable, bounded knowledge base (a documentation search, a legal corpus lookup), a well-tuned RAG system is sufficient. The five-stage stack is overhead you don’t need for that problem class.

But for agents running multi-turn sessions, operating on dynamic knowledge bases, working within bounded token budgets, and returning accurate answers when retrieved content exceeds what the context window holds, RAG is stage one. Context engineering is the architecture that makes the remaining five stages possible.

The question your architecture hasn’t answered

Retrieval answers: what exists?

Context engineering answers: what’s useful, right now, given what the system knows about this session, this query, and this constraint?

These are different questions. They require different architectural decisions. One is a search problem. The other is an engineering discipline most AI systems never touch.

If your AI produces wrong answers after your retrieval layer improved, you’re not looking at a retrieval failure. You’re looking at what happens when you build one stage of a six-stage system and ship it as the whole thing.

The question worth sitting with: who in your architecture made the decision about what gets into the context window?

If the answer is nobody, the decision was made by default. And defaults are what production failures are made of.

What’s the most expensive context you’re currently rebuilding every session? Reply with it. There’s a pattern worth naming.

Good context architecture makes agents reliable inside a session. Sessions still don’t compound. Next issue: why the composable stack has no hub, and why that’s the ceiling on everything else you’re building.

Systems Intelligence covers AI architecture for operators who need it working, not just running. New here? Subscribe.

Sound familiar?

Here’s what nobody mentions when they sell you on AI agents: model failures are rare. Most agents don’t fail because they’re stupid. They fail because the environment they’re running in wasn’t built for them. The gap between “this AI can do that” and “this AI does that consistently, without supervision” is a plumbing problem, not a model problem.

We hit all three failure modes building the receipt intake workflow for Housewire. In testing, the agent processed expenses, categorized transactions, and surfaced anomalies. In production, it answered with high confidence from the wrong context, fell apart on edge cases nobody anticipated, and checked for new receipts every hour on a cron job. None of it was a model problem. All of it was plumbing.

The field’s default assumption is that reliability improves when you improve the model. Better model, better prompt, different provider. SI’s position is the opposite: the model is rarely the constraint. The environment is. And it fails in three specific ways.

Dumb RAG

Most teams think about retrieval like this: give the agent access to the knowledge and it’ll figure out what it needs. That works in a demo. In production, it creates context rot.

Accuracy degrades when you load too much into an agent’s working context: entire databases, full document libraries, unstructured dumps. Not because the model can’t handle it. Because the signal gets buried. The agent keeps answering, with high confidence, from the wrong part of the context.

The pattern that works in production: just-in-time retrieval. The agent gets structured access to domain knowledge and pulls exactly what the task requires. The benchmark from teams running this in production: keep working context under 8,000 tokens for task-execution agents. That number forces precision about what goes in.

This is an information architecture problem, not an AI problem.

Brittle Connectors

Most software isn’t built for agents. It’s built for humans, which means APIs designed for web UIs, undocumented field behavior, rate limits that made sense for human-paced interactions, and custom field semantics that live in one senior developer’s head.

Point an agent at that facade and you’re asking it to navigate a floor plan it wasn’t designed for. The result: the agent works when everything goes right and falls apart the moment it hits an edge case nobody anticipated.

Agent-ready facades look different. Workflow-level tools where one tool handles one complete operation, not one API call. Descriptions that explain purpose and consequences, not just parameters. Error messages in plain language. “The expense category doesn’t match any active source rules” is actionable. “Error 422: Unprocessable Entity” is not.

We built the receipt intake workflow for Housewire twice. The first design exposed raw database operations to the agent. The second defined workflow-level operations. The difference in reliability wasn’t incremental. It was categorical.

The Inspection Tax

Autonomous agents need to react to things that happen in your system. New record created. Status changed. Threshold crossed. The default approach is an inspection schedule: check for new items every N minutes.

It seems reasonable. It’s quietly catastrophic.

You can’t run autonomous agents on request-response infrastructure. Running on a fixed inspection cycle introduces latency, burns compute on empty checks, and converts what should be an event-driven process into a slow batch job. An agent checking for new receipts every hour isn’t an autonomous system. It’s a scheduled job with extra steps, and the most common reason implementations feel unreliable without anyone understanding why.

The fix is straightforward: agents acting on real records need data under five minutes stale. That means webhooks, realtime subscriptions, state change triggers, not cron jobs asking whether anything changed. This is a solved infrastructure problem. The hard part is recognizing your inspection schedule is the bottleneck before you’ve spent six months debugging the wrong thing.

Three questions that tell you more than any benchmark

If you’re building or running an AI agent integration, start here:

1. What does your agent load into context, and what happens to that as your knowledge base grows?

2. Are the tools available to your agent workflow-level, or are they raw API calls it has to chain together?

3. When something changes in your system, does the agent find out through a webhook, or by asking?

Most implementations fail question two or three on contact. Fixing them isn’t glamorous. These are the load-bearing walls: what decides whether your agent runs in production or gets quietly retired.

The demo always works. What you’re actually evaluating is the plumbing.

Which of the three failure modes have you hit? Reply with the one that cost you the most time. I’m compiling patterns.

The plumbing is fixed. The context window still has a problem most teams have never heard named. Next issue: why retrieval isn’t the layer you need to fix, and what is.

Systems Intelligence is the honest trade publication for AI automation practitioners. No demos. No hype. Just what it actually takes to make AI reliable in production. Subscribe.

You’ve seen the checklist. Someone forwarded it, someone else reposted it, and now it’s sitting in three of your channels with a countdown attached. Patch these headers, remap these error codes, do it before the 28th or your integrations fall over.

If you run remote MCP servers in production, that message is built to make your stomach drop. So let’s start with what’s true, because plenty of it is.

This is the largest revision of the protocol since launch. The maintainers say so in as many words: the release candidate locked May 21, and the final spec publishes July 28 (blog.modelcontextprotocol.io). The changelog is long and none of it is cosmetic. OAuth hardening. JSON Schema 2020-12 for tool schemas. W3C trace context. HTTP+SSE transport deprecated. This is a genuine reset, and treating it as a non-event is its own kind of malpractice.

So the deadline is real. What’s fake is the cliff.

Real vs. theater

Here’s the line nobody forwarding the checklist is drawing for you: the day the news breaks is not the day anything breaks.

MCP’s lifecycle policy guarantees at least twelve months between deprecation and removal. Servers built on the 2025-06-18 and 2025-11-25 revisions keep working after July 28. Nothing gets deleted day one. And if you’re running local STDIO servers, most of this doesn’t reach you at all. The disruptive changes are a remote-HTTP concern, because that’s where the problem they solve actually lived.

There is a real reason to move: you want to be current before the runway closes, and you want to reason about the change while it’s fresh instead of under duress a year from now. That’s a deadline for adoption. It is not a fire alarm.

The checklist collapses those two things into one. It takes a protocol reset with a year of runway and repackages it as a config patch due Tuesday. And once you can see that move, you can see what it’s hiding, which is the only part that actually matters.

One change takes state out of the protocol

Sessions are gone.

That’s the change that leaves every request standing on its own, and it’s buried on page four of every checklist because it can’t be written as a step.

Two proposals do the work. SEP-2567 removes sessions. SEP-2575 removes the initialize/initialized handshake. Identity and capabilities now travel through _meta, with a new discovery method replacing the old negotiation. WorkOS, writing on the authentication changes, puts it plainly: the protocol is going stateless. Any request can now land on any server instance.

Sit with that for a second, because it is not a version bump. It’s an architecture decision someone already made for you.

A stateful protocol lets you assume continuity. The same client talks to the same instance, the handshake establishes who they are, and each exchange after leans on that. Stateless removes the assumption. Every request arrives cold. Which means the questions you thought were settled are open again. Where does state live now? How do you establish identity per-request instead of per-session? How does a load balancer route traffic it can no longer tie to a session?

The spec answers that last one directly. The Mcp-Method and Mcp-Name headers become mandatory (SEP-2243) so a load balancer can route without cracking open the request body. That isn’t a chore to tick off. That’s the protocol telling you it now expects to run behind infrastructure that treats every call as independent. Caching moves the same direction: ttlMs and cacheScope appear on results because in a stateless world the server can no longer remember what it already told you.

Read those details as tasks and you’ll patch them and move on. Read them as evidence and they say the same thing in unison. The model of what an MCP connection is just changed underneath you.

The specimen

The forwarded checklist is worth keeping: as a specimen, not a map.

It optimizes for looking responsive. Fifteen bullets, a deadline, a sense of motion. What it can’t do is tell you why any of it is happening, because the “why” isn’t a task you can complete. The posture treats a protocol reset as a config scramble, and a config scramble has one seductive property: you can finish it. You can close the tickets and feel done, having never once asked what stateless does to your routing, your identity model, or your cache.

That’s the trap. Migrate reactively to a list and you’ll rebuild your integration twice. Once now, in a hurry. Again in six months, when you realize removing sessions quietly rewrote your entire state and routing model while you were treating it like a header change.

One more for the “why” pile, since it also isn’t a step: Akamai has flagged that the new MCP Apps feature introduces a stored-XSS class of risk, and that cheap task creation opens a denial-of-service vector. Security researchers are reading the changelog as architecture. The checklist is reading it as chores.

The actual question

Adopt while it’s current. The reason isn’t to dodge a cliff that doesn’t exist. It’s that reasoning about a stateless protocol is the work, and doing it deliberately now costs less than doing it reactively later.

The deadline on your feed is answering the wrong question. The real one was never did I patch the headers before the 28th. It’s does my architecture still hold when any request can hit any instance and nothing remembers the last one.

One question before the next issue: where does your MCP server keep state today: in the session, in a database, or somewhere you’d have to go check? Reply with one sentence. I’m compiling how people answer, because it predicts almost exactly how much the stateless shift is going to cost them.

94% of those same enterprises say agent sprawl is their fastest-growing security risk.

12% have a governance system to manage it.

One survey. 1,900 IT leaders. The report called it “agentic AI going mainstream.” That’s one way to read it. Another: a structural guarantee that most of these programs end badly.

The speed is real. The demand is real. The governance infrastructure isn’t there yet. Deployment has already outpaced it.

How this happens

Every enterprise AI meeting has the same pressure: move faster, ship more, prove ROI before the next budget cycle. The agents that get approved are the ones that solve visible problems quickly. The question nobody asks in that meeting is “what happens when we have forty of these?”

The result is exactly what the data shows: agents deployed team by team, use case by use case. Different vendors. Different authentication patterns. Different assumptions about data access. An agent for HR, one for finance, one for customer success, one the engineering team spun up without telling anyone.

No one intended to build a security liability. They just didn’t build a system.

What sprawl costs

The 12% number is the one that matters.

It’s not that the other 88% aren’t trying. 97% are actively planning org-wide AI agent strategies. But planning and infrastructure are different things. You can’t governance-plan your way out of 88 ungoverned agents running on a hope that nothing goes sideways.

Gartner expects more than 40% of agentic AI projects to be canceled by 2027. Not failed. Canceled. The distinction matters.

Failed means it didn’t work. Canceled means it worked well enough to scale, then the organization couldn’t manage what it scaled into.

The mechanism is predictable. An agent gets access to more data than intended. Or it writes to systems it shouldn’t. Or the team that built it turns over and nobody knows what it’s doing anymore. Not by accident. Ownership was never assigned. These aren’t edge cases. They’re structural outcomes of building deployment velocity before building governance.

The 94% who flagged sprawl as a risk aren’t paranoid. They’re accurate. They just haven’t stopped deploying.

What a governance system actually is

Here’s what it isn’t: a platform. The 12% aren’t there because they bought a governance tool. Most governance platforms in this category are either too immature or too enterprise-priced to matter at this stage. They’re there because they treated “what governs these agents” as a first-class design question from day one, not an afterthought to be solved once the deployment count got uncomfortable.

That means three things.

Ownership. Every agent has an owner. Not a team. A person. Someone accountable when it behaves unexpectedly. Someone whose name is attached to the access rights it holds. Someone who gets paged when it fails. If you can’t name the owner of every agent in your stack, you don’t have a governance system. You have a hope.

Audit. Every agent’s actions are logged and reviewable. Not by default. By design. “What did this agent do between 2am and 4am?” should have a deterministic answer. If it doesn’t, you don’t know whether it’s working correctly or just working quietly. Those aren’t the same thing.

Scope. Every agent has explicit boundaries: what it can read, what it can write, what systems it can touch. Not what it’s supposed to do. What it’s technically permitted to do. That distinction is where the risk lives. Agents don’t malfunction on purpose. They operate at the edge of their permissions, wherever you put that edge.

Ownership, audit, scope. Not a product. A system. Three questions every agent deployment should be able to answer before it goes anywhere near production.

The math nobody wants to do

96% deployed is a headline. 12% governed is the story underneath it.

The organizations that are still running functional agent programs in 2027 aren’t the ones that deployed the most, the fastest. They’re the ones that built the system before they needed it.

Most didn’t. Most won’t. Not until an incident makes governance the priority it should have been from the start.

Governance isn’t the brake on your AI strategy. It’s the foundation that lets the strategy hold weight.

One question, and I read every reply: how many agents are running in your operation right now, and could you name the owner of each one? Hit reply.

Systems Intelligence covers AI architecture for operators who need it working, not just running. New here?

Subscribe now