Making AI Agents Think Before They Retrieve

An operating system without a scheduler would still run programs. It would just run every one the instant it arrived, full-throttle, no priorities, no admission control, no asking whether anything needs to run at all. The machine would work, badly, and choke the moment the load went up. Nobody would call that an operating system. They’d call it a loop with ambitions.

The standard memory-augmented agent shipping today is that machine.

It has an execution layer: embed the query, search the store, stuff the results into the prompt, generate. What it’s missing is the layer above, the one that decides whether to run any of that in the first place. The scheduler. The thing that asks, before it reaches for the vector database, the only question that actually matters: do I already know this, or do I need to fetch it?

The pipeline never asks. It’s a reflex:

User Query  ==>  Retrieve Memory  ==>  LLM  ==>  Answer

Retrieve first, think later. The query comes in, the store gets pinged, the chunks come back, the model gets stuffed, the answer comes out. Retrieval is treated as a primitive, an atomic action that just happens, never as a choice with a cost and an expected benefit. That’s the expensive bug hiding in plain sight across the entire RAG ecosystem, and fixing it means building the missing layer: a cognitive controller that thinks before it retrieves.

The pipeline everyone ships

What’s actually happening under the hood of a memory-augmented agent hides a lot of waste behind that simple three-step arrow.

A user turn comes in. The agent takes the query, embeds it, runs a k-nearest-neighbor search against a vector store (Pinecone, Weaviate, FAISS, whatever), gets back the top-$k$ chunks, prepends them to the conversation context as “retrieved memories,” and hands the whole inflated prompt to the LLM. The LLM generates. You ship the answer. Repeat forever.

Here’s what that looks like in practice. A user asked an agent I’d been running whether they’d mentioned any allergies. The answer was already in the conversation. Three turns earlier they’d said they had no food allergies. The agent retrieved anyway, pulled back seven old chunks, one of which mentioned a friend’s shellfish allergy at a dinner months back, and confidently reported that the user was allergic to shellfish. Retrieval worked perfectly. The agent got confused by what it found. That’s a retrieval-shouldn’t-have-happened failure, and it gets worse the longer a conversation runs.

The costs compound in ways nobody measures:

• Token bloat. Every retrieval dumps $k$ chunks into the context window. With $k=5$ and chunks of ~300 tokens, that’s 1,500 extra tokens per turn, whether the retrieval helped or not. Over a 200-turn session, you’ve paid for 300,000 tokens of context that was mostly noise.
• Latency. Every retrieval is a network round-trip plus an embedding call plus a vector search plus a bigger LLM forward pass. Do that on every turn and your time-to-first-token balloons.
• Hallucination injection. Retrieved chunks aren’t free information. They’re additional information the model has to disambiguate from the actual conversation. When a retrieved chunk contradicts or merely resembles something in context, the model gets confused. The shellfish case above wasn’t a retrieval failure. It was a retrieval-shouldn’t-have-happened failure.
• Context entropy. As you keep stuffing retrieved chunks into the window, the signal-to-noise ratio drops. The model spends attention budget separating relevant facts from retrieved clutter. Naïve context accumulation grows token cost quadratically; smart compaction brings it back to linear. The gap is enormous.

So here’s the question that launched this whole investigation for me. If a meaningful fraction of queries can be answered from the context the agent already has, why are we retrieving for all of them?

How retrieval became a reflex

Retrieval didn’t start as a reflex. It started as a fix. Early RAG solved a specific problem: models hallucinated because they didn’t know things, so you fetched the things and handed them to the model. That worked. Then an entire ecosystem grew up around making it work better. Better embeddings. Better vector indexes. Better chunking. Better re-ranking. Hybrid search. Graph retrieval. The work was relentless, and almost all of it was about the how of retrieval: how to find the right chunks, how to rank them, how to fit them in the window.

Almost none of it questioned the whether. Retrieval became the entry point of every agent turn, the thing that runs before thinking. You build the pipeline, wire up the vector store, call it on every query. The defaults in every framework bake this in. The benchmarks measure agents on the assumption that retrieval happens. The field got very good at retrieving and rarely asked whether a given turn needed retrieving at all.

That’s the gap I kept hitting. The agent I was running had a perfectly good retriever. It used it on turns where the answer was already in the conversation, and got worse answers because of it. The fix wasn’t a better retriever. It was a layer above the retriever that decides when to call it.

Once I added a simple confidence gate to that agent, the shellfish-class failures stopped, and the retriever got called about half as often. That’s not a benchmark. It’s the anecdote that made me take the idea seriously and start asking what the general version looks like. The rest of this post is that general version.

Formalizing the retrieval decision

Time to be precise. Vague ideas about “smart retrieval” don’t ship. I’ll define the problem the way the people who build operating systems would, because that’s the right analogy, and I’ll come back to it.

At time $t$, an agent faces:

• A query $q_t$ (from the user, or an internal sub-goal).
• A contextual state: the conversation so far, any facts already retrieved this turn, tool outputs in the window.
• A set of memory stores $M$: short-term buffers, long-term vector DBs, knowledge graphs, entity stores, whatever the agent has.
• An uncertainty estimate $U_t$: how confident the agent is it can answer $q_t$ from current context alone. Could be the LLM’s own log-probabilities, an ensemble disagreement, a verbalized confidence, or a learned score.
• A context entropy $E_t$: a measure I’m proposing for how cluttered the current context is. More on this below.

The Retrieval Decision Problem is to choose an action $a_t$ that maximizes expected answer correctness minus expected cost:

where $\pi$ is the controller policy and the action space is roughly:

1. AnswerDirect: answer from current context, no retrieval.
2. Retrieve($q', k, \text{mode}$): formulate a memory query (possibly transformed from $q_t$), fetch $k$ items, in a chosen mode (semantic, lexical, graph, tool/web), then answer.
3. Iterate: retrieve, update state, retrieve again with a refined query. Multi-hop.
4. Verify: after answering, retrieve supporting evidence and fact-check.
5. Anticipate: predict likely future queries and prefetch in the background.
6. Compact: don’t retrieve; instead summarize/prune current context to lower $E_t$, then answer.

That’s the whole decision surface. Three questions get answered at once: whether to retrieve, how much to retrieve, and which mode to use. Today’s agents answer all three the same way every time: yes, $k=5$, semantic.

Context entropy, a metric nobody’s measuring

I want to dwell on $E_t$ because it’s the piece I haven’t seen anyone define, and I think it matters.

The intuition: a context window isn’t just “full” or “empty.” It has a quality. You can have a 4,000-token window that’s clean and coherent, or a 4,000-token window that’s full of contradictions, duplicate facts stated three different ways, stale information, and chunks that have nothing to do with the current question. Same token count, wildly different usefulness. The LLM’s job is much harder in the second case.

Context entropy tries to capture that. Concretely, it could aggregate:

where $C_t$ is the current context. Duplication is measured by semantic similarity clustering of context chunks. Contradiction by a small entailment model flagging pairs that can’t both be true. Staleness by timestamps on memory entries. Irrelevance by embedding distance between context chunks and the current query.

High $E_t$ means the context is messy. And here’s the non-obvious move: when $E_t$ is high, the right action might not be “retrieve more.” It might be “compact first, then maybe retrieve.” Stuffing more chunks into a confused context makes it more confused. This is exactly the failure mode that produces the shellfish bug. The retrieved chunk muddied an already-cluttered window.

I’m being deliberately conceptual here. I’m not claiming this exact formula is optimal. I’m claiming the axis exists and nobody is measuring it. Once you measure context entropy, you can act on it. Until you do, your agent is flying blind on context quality.

The CPU-scheduler analogy

Here’s the mental model that made everything click for me.

An operating system doesn’t run every process the instant it arrives. It has a scheduler. The scheduler decides who runs, for how long, with what priority, and sometimes whether a process should run at all (swapped out, deferred, killed). The scheduler is a control plane sitting above the execution layer. It’s the thing that makes a time-shared machine work.

Memory-augmented agents have an execution layer (embed, search, generate) and no scheduler. Every query is a process that gets run immediately, full-throttle, no questions asked. We built the CPU and forgot the OS.

The cognitive controller I’m proposing is the scheduler. It sits above the memory subsystem, looks at the request, looks at the state of the machine, and decides what to do. Sometimes it runs the retrieval process. Sometimes it says “we already have this in cache, just answer.” Sometimes it says “the cache is dirty, clean it first.” Sometimes it prefetches because it can see what’s coming.

This isn’t a cute metaphor. It’s an architecture. Call it a MemOS, a memory operating system. The controller is its scheduler. Once you frame it that way, a lot of design questions answer themselves: you want priority levels (a question about the user’s medical history deserves retrieval even at high cost; “what’s 2+2” doesn’t), you want admission control under load (token budgets), you want prefetch and cache invalidation. We know how to build these. We’ve been building them for sixty years in a different domain.

The controller action taxonomy

To design controllers systematically, here’s the full action space mapped out. A controller at each turn picks from this menu:

The controller’s policy $\pi(s_t)$ maps state to one of these. 0 and 5 are the two that avoid retrieval entirely. Today’s agents never take them, because today’s agents don’t have a controller to take them with.

A single turn can chain these. A common pattern I expect to emerge: Retrieve → answer → Verify. Or Compact → Retrieve → answer. The controller drives a loop, not a one-shot classification.

Four ways to build the controller

This is where it gets concrete. There are four realistic ways to build $\pi$, and they trade off interpretability, adaptiveness, and implementation cost in different ways. I’ll show code for each.

1. Rule-based: the honest starting point

Hard-coded heuristics. Embarrassing to admit in 2026, but this is where I’d start every time, and I’ll defend it.

def rule_controller(query, context_summary, memory_stats):
    # If the query is chitchat, just answer. Don't burn a retrieval.
    if is_smalltalk(query):
        return ("ANSWER_DIRECT", {})

    # If we already have a high-similarity fact in context, answer directly.
    if semantic_similarity(query, context_summary) > 0.75:
        return ("ANSWER_DIRECT", {})

    # If the LLM is confident on the current context, trust it.
    if llm_confidence(query, context_summary) > 0.7:
        return ("ANSWER_DIRECT", {})

    # Factual lookup with low confidence -> retrieve, but stay cheap.
    if is_factual_query(query) and llm_confidence(query, context_summary) < 0.5:
        return ("RETRIEVE", {"query": query, "k": 5, "mode": "semantic"})

    # Context is cluttered -> compact before doing anything else.
    if context_entropy(context_summary) > ENTROPY_THRESHOLD:
        return ("COMPACT", {})

    # Default: answer without retrieval. Cheaper, and often right.
    return ("ANSWER_DIRECT", {})

Inputs: the query, a summary of current context (topics, embedding, token count), and memory metadata (entry count, recency). Outputs: an action and its parameters.

The case for it: simple, transparent, no training, no GPU, debuggable in a REPL. The case against: brittle. Hand-tuned thresholds drift as your user base and query distribution change. A rule that says “skip retrieval if similarity > 0.75” will be wrong for a subclass of queries you didn’t anticipate. But even a dumb rule-based controller will likely beat always-retrieve on cost while matching it on accuracy, because the always-retrieve baseline is so wasteful that almost any gating helps. I’d ship this first, measure, and use the failures to motivate the learned version.

2. Learned policy: when rules aren’t enough

Frame retrieval as a learning problem. Binary classification is the simplest framing (model $P(\text{RETRIEVE} \mid q_t, s_t)$ and threshold it), but the cleaner formulation is a multi-action policy over the taxonomy above.

def learned_controller(query, context_embedding, memory_embedding):
    # State: query embedding + context embedding + memory stats,
    # all concatenated into one feature vector.
    state = concat([
        embed(query),
        context_embedding,
        memory_features(memory_embedding),
    ])
    probs = controller_model.predict(state)   # softmax over actions
    action_id = int(np.argmax(probs))

    if action_id == ACTION_ANSWER:
        return ("ANSWER_DIRECT", {})
    # If we're retrieving, sub-models pick mode and k.
    mode = mode_head.predict(state)
    k = int(k_head.predict(state))
    return ("RETRIEVE", {"query": query, "k": k, "mode": mode})

The model is small, a few-layer MLP or a tiny transformer over the state vector, not the main LLM. Training is where it gets interesting. Two realistic paths:

• Supervised. Mine your logs. You have turns where retrieval happened and the answer was right, and turns where retrieval happened and the answer was wrong or unchanged. Label “retrieval was necessary” vs “retrieval was wasted” and train a classifier. Cheap, works, but your labels are noisy because you never observed the counterfactual (what would have happened if you hadn’t retrieved).
• Reinforcement learning. Treat each turn as a bandit. Reward $= +1$ if the answer is correct, $-0.5$ if partially correct, minus a small penalty $\epsilon$ for every retrieval action taken. Let the policy learn when retrieval’s expected benefit exceeds its cost. Use verifier-based RL to align the control decisions with task success. It’s the right instinct: you want the controller optimized for the thing you actually care about (right answers, cheaply), not a proxy.

The strength: a learned controller captures patterns you’d never hand-code, like “this kind of query in this context state is almost always answerable, even though it looks factual.” The weakness: it needs data, it can mis-generalize to out-of-distribution queries, and it’s opaque. When it decides to skip retrieval on a question it shouldn’t have, you’ll have a hard time explaining why.

3. Hybrid: rules gate, learned decides

This is what I’d actually build for production. Rules handle the obvious cases so the learned model never sees them; the learned model handles the ambiguous middle.

def hybrid_controller(query, context_summary, memory_stats):
    # Rules first: trivial cases never reach the model.
    if is_smalltalk(query) or is_arithmetic(query):
        return ("ANSWER_DIRECT", {})

    state = encode_state(query, context_summary, memory_stats)
    decision = learned_model.predict(state)

    # Confidence calibration: if the model is super sure either way, trust it.
    if decision["ANSWER"] > 0.9:
        return ("ANSWER_DIRECT", {})
    if decision["RETRIEVE"] > 0.9:
        return ("RETRIEVE", {"query": query, "k": 3, "mode": "semantic"})

    # Ambiguous middle: fall back to a conservative rule.
    if llm_confidence(query, context_summary) < 0.5:
        return ("RETRIEVE", {"query": query, "k": 3, "mode": "semantic"})
    return ("ANSWER_DIRECT", {})

You get the safety of rules at the extremes and the nuance of the learned model in the middle. The cost is implementation complexity: two systems to debug, two places where a bug can hide, and you have to be careful that the rule’s “obvious” cases are actually obvious. (They rarely are. Spend time on the boundary.)

4. Hierarchical: different decisions at different scopes

For multi-turn or multi-session agents, control naturally splits into layers. A session-level controller plans what to prefetch across the whole conversation. A turn-level controller handles each query. A two-stage controller decides whether to retrieve first. If yes, a separate policy decides how.

def hierarchical_controller(query, context, session_state):
    # Stage 1: should we retrieve at all?
    if not should_retrieve_policy(query, context):
        return ("ANSWER_DIRECT", {})

    # Stage 2: if yes, pick the mode and budget.
    mode = mode_policy(query, context)
    k = budget_policy(query, context, session_state.remaining_budget)
    return ("RETRIEVE", {"query": query, "k": k, "mode": mode})

The session-level policy is where anticipation lives. At session start, it might decide to load the user profile and recent preferences into context proactively. Per-query, the turn-level policy handles the small stuff. This decomposition mirrors how the OS scheduler works: there’s a long-term scheduler (which jobs get admitted) and a short-term scheduler (which process runs right now). Same idea, different substrate.

How they compare

My recommendation, having sat with this for a while: start with rule-based, measure relentlessly, promote the cases the rules get wrong into a hybrid controller, and only go full learned/hierarchical when you have the log volume to justify it. Most teams will get most of the benefit from a well-tuned rule-based controller and never need the rest.

The control loop, end to end

Here’s the full agent loop with the controller wired in. This is the thing I’d actually ship.

state = init_memory_state()

for turn_t, query in enumerate(dialogue):
    # 1. Encode the agent's state for this turn.
    context_repr = summarize_context(state.history, state.memory)
    uncertainty  = estimate_uncertainty(query, context_repr)   # U_t
    entropy      = compute_context_entropy(context_repr)       # E_t

    # 2. The controller thinks. This is the line today's agents don't have.
    action, params = controller.decide(query, context_repr, uncertainty, entropy)

    # 3. Execute the chosen action.
    if action == "ANSWER_DIRECT":
        answer = llm.answer(query, context_repr)

    elif action == "COMPACT":
        context_repr = compact_context(state.memory)
        answer = llm.answer(query, context_repr)

    elif action == "RETRIEVE":
        mem_query = params.get("query", query)
        k, mode   = params.get("k", 5), params.get("mode", "semantic")
        docs = memory_store.search(mem_query, k=k, mode=mode)
        answer = llm.answer(query, merge_context(context_repr, docs))

    elif action == "ITERATE":
        answer = iterative_retrieve_and_answer(query, context_repr, controller)

    # 4. Optional verification on high-stakes or low-confidence answers.
    if should_verify(answer, uncertainty):
        evidence = memory_store.search(answer, k=3, mode="semantic")
        if not verify_answer(answer, evidence):
            answer = llm.answer_with_evidence(query, evidence)

    # 5. Write back to memory if there's something worth remembering.
    state.memory = update_memory(state.memory, query, answer)
    yield answer

The controller is one line, controller.decide(...), but it’s the line that changes the whole shape of the system. Everything else is execution. The controller intercepts the “query → retrieve → answer” reflex and replaces it with “query → think → maybe retrieve → answer.”

A note on the should_verify step: cap the iterations. A controller that always thinks “one more retrieval might help” is an infinite loop waiting to happen. Hard-limit verification to one or two rounds and move on.

When the controller is an LLM

There’s a fourth architecture I glossed over because it deserves its own section: don’t build a separate controller module at all. Use the LLM itself as the controller, via a carefully crafted prompt. It’s the lowest-infrastructure way to get started.

[CONTEXT]: "<conversation so far>"
[QUERY]:   "<user question>"

You have an external memory store containing facts from past sessions.
Q: Do you have enough information in CONTEXT to answer QUERY confidently?
   If yes, answer now.
   If no, output a search query to fetch the missing information,
   and nothing else.
A:

The LLM either answers directly (skip retrieval) or emits a refined retrieval query. You’ve turned the memory decision into a language task, which is the thing LLMs are best at. No training, no separate model, no feature engineering, just prompt engineering and a parsing step on the output.

This is powerful and I don’t want to undersell it. The risk is the usual prompt-engineering risk: the LLM can hallucinate the decision just as easily as it hallucinates the answer. It might say “I have enough” when it doesn’t, or invent a retrieval query that’s worse than the original. Fine-tuning on a “should I retrieve?” task, with pairs of (context, query, correct decision), cleans this up a lot. If you’re going the LLM-controller route, budget for the fine-tune. Off-the-shelf prompting will leak failures on exactly the hard cases where the controller matters most.

Wiring it into real memory backends

The controller is memory-agnostic by design. It sits above whatever stores you have and emits actions; the stores execute them. A few concrete pairings:

• Vector DBs (Pinecone, Weaviate, FAISS). The RETRIEVE action is a semantic k-NN search. The mode parameter switches between pure semantic, hybrid semantic+lexical, or filtered search. The controller can also inspect retrieval quality. If the top results are all near-duplicates, that’s a signal the store doesn’t have what you need, and an ITERATE with a reformulated query is in order.
• Graph stores (Neo4j, Amazon Neptune). mode = "graph" turns the retrieval into a structured query. The controller compares the confidence of entity links extracted from context against the confidence of graph-derived answers. If the context already has the entities and the graph would just confirm them, ANSWER_DIRECT.
• Typed memory stores. Memory entries carry tags: user preference, factual knowledge, event, etc. The controller can scope retrieval by type: for a preference question, only query the preference partition; don’t drag in unrelated events.
• Pluggable memory frameworks. Any framework that exposes memory as stages (formation, retrieval, utilization) is the cleanest integration story. You register a “Decision” stage after formation and before retrieval. If the controller returns ANSWER_DIRECT, the framework skips the retrieval stage gracefully. You get the controller for free atop any compatible memory stack.

The API I’d want, if I were designing the interface:

class MemoryController:
    def decide(self, query: str, context: ContextSummary) -> Action:
        """
        Returns Action.ANSWER_DIRECT, Action.COMPACT,
        or Action.RETRIEVE with {query, k, mode}.
        """
        ...

The agent loop calls decide() and branches on the result. The controller knows nothing about how retrieval is implemented. The memory store knows nothing about why it’s being asked. Clean separation, and it means you can swap either side independently.

How you’d actually evaluate this

Claims without measurement are blog noise. Here’s the experimental framework I’d run, and the one I think the field should converge on.

Benchmarks

The benchmarks I’d run this on, the standard long-horizon agent-memory ones:

These three cover the spectrum: LoCoMo for raw long-conversation recall, MemoryAgentBench for structured multi-step memory, LongMemEval for the personalization angle that real chat assistants care about. Some of these include adversarial or unanswerable questions (the “I don’t have that information” case), which is exactly where a controller that skips retrieval has to be careful.

Metrics

You have to measure both sides. Did you get the answer right, and what did it cost you? Skip either and you’ll fool yourself.

The trap is optimizing retrieval count in isolation. A controller that never retrieves has a retrieval count of zero and an accuracy of garbage. You have to hold accuracy roughly constant and then compare cost. The honest framing is a Pareto frontier: accuracy on one axis, token cost on the other, and you want your controller on the frontier, up and to the left of the always-retrieve baseline.

Baselines and ablations

Run these, or your numbers don’t mean anything:

• Always-Retrieve. The standard RAG agent. Queries memory every turn. This is your upper bound on retrieval usage and the thing to beat on cost.
• Static mode-routing. Route every query to a fixed retrieval mode by type, with no pre-retrieval check. Tells you how much routing alone buys you (probably some, but not the skip cases).
• No-Memory. LLM-only, sliding context window, no external memory. Lower bound on accuracy; shows what pure parametric knowledge gets you.
• Off-the-shelf memory frameworks. Most default to always-retrieving, so expect them to cluster near Always-Retrieve.
• Ablations. Strip features one at a time, disable anticipation, disable verification, disable compaction, to see which controller behaviors are actually pulling weight. Vary the confidence threshold and the retrieval budget. For learned controllers, compare with and without RL fine-tuning.

What I expect the numbers to say

I haven’t run the full sweep yet, so these are hypotheses grounded in the structure of the problem, not results. But I’d bet on them.

Accuracy stays flat or ticks up. A controller that concentrates retrieval on the turns where it actually helps and skips the turns where it injects noise (the shellfish case) should hold accuracy or tick it up. The risk is on the under-retrieval side: set the threshold too high and you skip retrieval on questions you needed, and accuracy drops. The sweet spot is a controller that’s slightly conservative: retrieve when unsure, skip when sure.

Cost drops hard. If, say, 50% of turns can be answered from current context (and my anecdotal experience says it’s at least that), you’ve halved retrieval calls and roughly halved the tokens pulled from the store each turn. The total token bill drops by whatever share of the prompt those chunks were. The speedup comes from early stopping: skip the retrievals you don’t need. I’d predict 30–50% token-cost reduction at matched accuracy for a well-tuned controller. That’s the headline number, and it’s the one that matters for anyone paying an LLM API bill.

Latency tracks cost. Skip a retrieval, skip a network round-trip and a bigger forward pass. Average time-to-first-token should drop in proportion to the retrieval skip rate. The wins are biggest on the easy turns (chitchat, follow-ups, arithmetic), which today pay the same retrieval tax as hard factual questions.

Hallucination is the wild card. Done right, the controller reduces hallucination, because it stops injecting retrieved noise into contexts that already had the answer. Done wrong, with overconfident skips, it increases hallucination, because the agent answers from a context that was missing a needed fact. This is the metric to watch most closely. If you see accuracy hold but hallucination tick up, your threshold is too aggressive.

The picture I expect: accuracy versus token cost, with the controller on the Pareto frontier, up and to the left of the always-retrieve baseline.

The controller lands at roughly the same accuracy as always-retrieve, but shifted left on cost. Always-retrieve and static-routing sit below it, dominated. No-memory is on the frontier too, at the low-cost, low-accuracy end, cheap but unable to answer the hard questions. The controller is the point that matters: baseline accuracy, roughly half the cost. That’s a Pareto improvement, and it’s the whole argument.

How it fails, and what to do about it

No system is honest without a failure-mode section. Here’s where controllers break.

Over-retrieval. The controller misjudges and retrieves when it didn’t need to. Cost goes up, accuracy doesn’t. Boring failure, low damage. Mitigate with cost penalties in training and conservative thresholds in rules.

Under-retrieval (overconfidence). The dangerous one. Controller skips, context was insufficient, answer is wrong or hallucinated. This is the shellfish bug in reverse. Mitigate with a post-answer confidence check: if the answer comes back low-confidence, trigger a fallback retrieval and re-answer. The should_verify step in the loop exists for exactly this.

Stale knowledge. Controller decides not to retrieve for a topic it “already knows,” but the memory was cleared or never updated. The agent answers from stale or missing information. Mitigate by carrying last-retrieval timestamps in memory metadata and refreshing when they exceed a staleness window.

Model drift. A learned controller trained on one query distribution mis-classifies when the distribution shifts (new user cohort, new product surface). Mitigate with diverse training scenarios, regular retraining, and the hybrid fallback so rules catch the learned model’s blunders.

Verification loops. A controller that always thinks “one more retrieval might help” will spin. Hard-cap iterations. This is a one-line fix that prevents a production outage, so don’t forget it.

Catastrophic forgetting of essential context. If the controller’s COMPACT action gets too aggressive, it can prune a fact that turns out to matter three turns later. Mitigate by protecting certain context (user identity, stated preferences, safety-critical facts) from compaction, a “pinned” set that the compactor can’t touch.

The bigger picture

Step back. The thing I keep coming back to is that we’ve been building memory-augmented agents upside down.

We treat memory as a store and retrieval as a primitive. The whole RAG ecosystem is built on this assumption: better embeddings, better indexes, better chunking, better re-ranking, all of it optimizing the how of retrieval. Almost nobody questioned the whether. We built a magnificent retrieval engine and bolted it onto a reflex.

The shift I’m arguing for is small in code and large in framing. Memory isn’t storage. Memory is a decision. Every retrieval is a choice with a cost and an expected benefit, and an agent that never evaluates that choice is an agent that’s bleeding tokens and injecting noise on every turn. Add one layer, a controller that asks “do I need this?” before fetching, and the whole cost profile changes.

The OS analogy isn’t decoration. When time-sharing systems appeared, the breakthrough was the scheduler, the control plane that decided which process ran when, not a faster CPU. Memory-augmented agents are at the same moment. The execution layer is good enough. What’s missing is the scheduler.

For practitioners: don’t start with a learned controller. Start with rules. Gate your retrievals with a confidence check and a context-similarity check, measure tokens and accuracy, and watch the cost curve bend. Slot a decision stage into your existing pipeline. Track context entropy even if you only log it at first. You can’t manage what you don’t measure. The first time you see a query that retrieved seven chunks and answered from the conversation anyway, you’ll know why this matters.

For researchers: this opens a direction. Retrieval as a controlled action, akin to planning or meta-reasoning. The open questions are real: how to learn optimal retrieval policies with RL, how to formalize context entropy rigorously, how multi-agent systems coordinate their memory controllers without trampling each other, how to do this safely so a controller never skips retrieval on a safety-critical fact. The framework in this post (the problem statement, the taxonomy, the architecture options, the eval plan) is a starting blueprint, not a finished answer.

I started this whole investigation because of a shellfish hallucination. One bug, one user, one moment where the agent reached for memory it didn’t need and got confused by what it found. What fixed it was a question the agent never asked itself: do I already know this? A better embedding model wouldn’t have caught it. A bigger context window wouldn’t either. The reflex was the bug.

That’s the layer we’re missing. Build the scheduler. Let your agents think before they retrieve.