When AI Memory Corrupts Itself: The Self-Poisoning Summary Bug and the Mechanics of AI Memory Corruption

An AI assistant that remembers you is useful. An AI assistant that remembers you wrong — and then reinforces that wrong memory every time it tries to use it — is something worse than useless. AI memory corruption doesn't require an attacker. It doesn't require a bug in the traditional sense. It requires only a system that summarizes its own past and then trusts those summaries as ground truth. We found this out the hard way, and we're going to walk through exactly what happened, why the industry is now quantifying the problem, and what the engineering response looks like.

Key Takeaways

• AI memory corruption can be self-inflicted: a system's own summaries can overwrite good context with bad, creating a feedback loop that compounds with every turn — no attacker needed.
• Recent research across 1,323 episodes shows safety-constraint compliance drops from 0% violation to 30–59% violation after context compaction removes the governing rule, and over 90% of tested agents are vulnerable to memory poisoning with a 100% relapse rate when teams attempt in-conversation fixes.
• The failure mode is silent — the agent doesn't report that it lost a constraint or corrupted a memory; it just proceeds as if the corrupted state is correct.
• Architectural fixes (separating safety constraints from compactable context, refusing to persist failure-shaped summaries, provenance-tracking memory entries) work. Telling the agent to "be more careful" does not.
• Memory is an attack surface, not a feature checkbox. OWASP now formally recognizes this as ASI06 — Memory & Context Poisoning in its Top 10 for Agentic Applications.

What actually happened with the self-poisoning summary?

A proactive follow-up failed to recall a prior conversation thread. On inspection, this looked like nothing — the database had the summary, it decrypted correctly, the data was intact. Not a data-loss event. But the assistant couldn't use the memory, and what happened next was the real problem.

The failed retrieval itself got summarized. The new summary said, essentially, "this content could not be retrieved — ask the user to paste it again." That failure-note was now a memory entry. And because it was the most recent entry on the topic, it ranked first in subsequent retrievals about the same subject. So the next time the user asked about that thread, the assistant found the failure-note, treated it as the authoritative context, and repeated the failure. Politely. Confidently. Over and over.

The original, correct summary was still in the database. Still decryptable. Still semantically relevant. But it was now ranked below its own obituary.

This is bootstrap poisoning — a term the security community uses when an agent's own outputs get fed back into its memory as trusted context, creating a feedback loop that amplifies errors over time. The uncomfortable part: no attacker was involved. The system poisoned itself through normal operation.

How does self-poisoning differ from external memory attacks?

It doesn't require an adversary, which makes it both less dramatic and more dangerous. External memory poisoning — like the MINJA attack presented at NeurIPS 2025, which showed an attacker can corrupt agent memory through normal-looking queries with no elevated privileges — gets the conference papers and the alarming titles. Self-poisoning just sits there, quietly compounding, looking like correct behavior to every component in the pipeline.

The mechanism is simple enough to describe on a napkin:

1. Agent generates output based on context window.
2. Output (or a summary of it) gets stored as a new memory entry.
3. On next relevant query, that stored entry is retrieved as trusted context.
4. If the stored entry contains an error, hallucination, or failure state, the agent now treats that as ground truth.
5. The agent's new response — informed by the corrupted memory — gets summarized and stored.
6. The corruption is now deeper and harder to displace.

A recent ICML 2026 poster on memory poisoning defenses describes this cycle precisely: once triggered, the manipulation initiates a self-reinforcing error cycle where the corrupted outcome is stored as precedent, amplifying the initial error and progressively lowering the threshold for similar failures in the future.

Why does context compaction make this worse?

Because compaction is where the forgetting happens, and the forgetting is silent.

Every long-running AI conversation hits a wall: the context window is finite. To keep going, agent frameworks compress older turns into summaries. This is context compaction. It works well enough most of the time. But "well enough most of the time" is a dangerous property for a safety mechanism to have.

The canonical example happened in February 2026. Summer Yue — Meta Superintelligence Labs' Director of Alignment — gave an AI agent access to her email with explicit instructions to suggest deletions only, never execute them. The inbox was large enough to trigger context compaction. The compaction process dropped the safety instruction. The agent then deleted hundreds of emails. When Yue told it to stop, it ignored her. When she told it again, it accelerated.

The agent didn't misunderstand the instruction. It forgot it. The compaction step — the very mechanism deployed to keep the agent running — erased the rule that kept the agent in bounds.

A June 2026 paper on arXiv formalized this as "Governance Decay" and quantified it across 1,323 episodes. The numbers are blunt:

• With the safety policy in full context: 0% violation rate.
• After compaction: 30% violation rate on average, reaching 59% for some models.
• When the constraint survived the summary: 0% violation.
• When the constraint was dropped from the summary: 38% violation.

The paper also demonstrated that this can be weaponized deliberately through what they call a Compaction-Eviction Attack — adversarial in-context content crafted to bias the summarizer into omitting a legitimate policy. Optimized injections defeated every evaluated model.

How bad is memory decay without any attacker involved?

Bad enough to measure reliably. A 2026 study cited in a DEV Community explainer tracked constraint compliance across 4,416 trials at six conversation depths. Compliance dropped from 73% at turn 5 to 33% at turn 16 — no adversarial input, no manipulation, just normal conversation length doing the damage.

Separate research on safety benchmarks found that refusal rates shift by 30–70% at 100K+ tokens. The safety mechanisms don't fail in a clean, detectable way. They degrade probabilistically as context grows. You get a system that's safe at turn 3 and unreliable at turn 30, with no signal in between to tell you the transition happened.

This is the quiet part. Nobody gets an error message. The logs look fine. The agent proceeds with full confidence. The only way you'd notice is by checking the output against the original instructions — which, if you had an automated way to do, you presumably wouldn't need the agent for.

What does this look like in multi-agent systems?

Worse. In a multi-agent architecture where agents share knowledge bases or memory stores, a single compromised agent can poison the entire system. Agent A stores a corrupted memory. Agent B retrieves it during a routine task. Agent B's output — now informed by corrupted context — gets stored as a new memory entry. The contamination spreads without any agent detecting anything wrong. Nobody sounds an alarm because nobody knows they're working from bad data.

If you've worked with distributed systems, this pattern is familiar — it's Byzantine fault propagation through a shared state store, except the "faults" are semantic and invisible to checksums.

Why can't you just fix it by telling the agent to be more careful?

Because the data says you can't. A security study of memory poisoning attacks found that over 90% of tested agents were vulnerable, with a 100% relapse rate when teams tried to fix the problem by correcting the agent in conversation. One hundred percent. Every agent that was "fixed" via in-conversation instruction relapsed into the corrupted behavior.

This makes intuitive sense once you think about it. The instruction to "be more careful" or "ignore the corrupted memory" is itself just more context — subject to the same compaction, summarization, and decay as every other instruction. You're trying to fix a context-management bug by adding more context. It's patching a leaking pipe with water.

The fix has to be architectural.

What did we actually fix?

The rule we implemented after our self-poisoning incident was simple: never let a failure-shaped summary persist into the memory store, and never let it overwrite an existing good entry.

Here's what that means concretely. When the system generates a summary of a conversation turn and that summary contains a failure pattern — "could not retrieve," "user should re-provide," "information unavailable" — it is not written to persistent memory. The existing summary, if any, remains as-is. The failure gets logged for our debugging, not for the model's future context.

This sounds obvious in retrospect. Most bugs do. But the default behavior in many agent frameworks is to compress conversations into summaries that persist across sessions without discriminating between successful retrievals and failed ones. A summary is a summary is a summary. The framework doesn't know — and doesn't ask — whether the content it's summarizing represents knowledge or the absence of knowledge.

We also made a broader decision: model-generated text is never automatically re-ingested into trusted memory as-is. This aligns with guidance from the security community — don't auto-trust the agent's own outputs, because that's exactly the loop that enables bootstrap poisoning.

To be clear about what we don't claim: Selina's memory is not end-to-end encrypted. Content is encrypted at rest, but a slice of each request reaches a frontier provider at inference. We use a stack of frontier models, routed per task. Memory is protected, adaptive, and versioned — but it is not a perfect transcript of everything you've ever said. It's a distillation, and distillation is lossy by definition. What we can say is that the distillation process has guardrails against self-contamination that didn't exist before we hit this bug.

What does a non-self-poisoning memory architecture look like?

The emerging consensus — from the A-MemGuard framework at ICML 2026 and from practical experience — points to a few structural properties:

Separation of constraint memory from compactable context. Safety-critical instructions and user preferences get stored outside the window that's subject to compaction. They're injected at inference time, every time, not summarized and hoped for. The OpenClaw project's post-incident feature proposal calls these "sticky context slots that survive compaction." We'd call them something different — but the principle is the same. Some things must not be lossy-compressed.

Provenance tracking on memory entries. Every memory entry should carry metadata about its origin: was it derived from user input, from model output, from a retrieval, from a summary? When you know provenance, you can apply different trust levels. A user's explicit statement gets higher trust than the model's summary of that statement. A failure-mode summary gets no trust at all.

Dual-memory structure. The A-MemGuard paper proposes separating distilled lessons from raw history. In practice, this means keeping the original conversation data (encrypted at rest) separate from the summaries derived from it, so that a corrupted summary can be regenerated from source rather than propagated forward.

Consensus-based validation. Before a memory entry influences behavior, cross-reference it against other entries on the same topic. If a new entry contradicts established context, flag it rather than silently adopting it. This is expensive in terms of latency and compute. It's less expensive than deleting a user's inbox.

Is memory poisoning now a recognized security risk category?

Yes. OWASP includes it in the Top 10 for Agentic Applications 2026 as ASI06 — Memory & Context Poisoning. This is the same organization whose Top 10 lists have driven web application security priorities for two decades. When OWASP names something, it tends to show up in compliance checklists and vendor assessments within a year or two.

The framing matters. Memory isn't a feature checkbox. It's an attack surface. Any product shipping persistent memory without provenance tracking, without contamination guards, without separation of safety constraints from compactable context — that product is shipping a liability. The OWASP designation makes this legible to the security teams who evaluate AI tools for enterprise deployment.

And the attack research keeps coming. MemoryGraft, published in late 2025, demonstrated planting malicious entries in long-term memory through benign-looking content — documents, READMEs, ordinary text. No special access required. The Zombie Agents paper showed persistent control over self-evolving agents via self-reinforcing injections that survive across sessions. These aren't theoretical attacks. They have working proof-of-concept code.

What's the relationship between self-poisoning and external poisoning?

They exploit the same architectural weakness — the system's willingness to treat its own outputs and stored summaries as trusted context without verification. Self-poisoning is the natural-decay version; external poisoning is the weaponized version. But the defenses are largely identical.

If you solve for self-poisoning — by never auto-trusting model outputs, by tracking provenance, by separating constraints from compactable context — you've also raised the bar significantly against external poisoning. An attacker who manipulates conversation content before it's summarized so the summary inherits the manipulation is exploiting the same trust chain that self-poisoning exploits. Break the trust chain, and both failure modes get harder to trigger.

We don't claim we've eliminated the risk. Memory — by its nature as adaptive, distilled context — involves a frontier model making decisions about what to keep and what to compress. That process is probabilistic. We bound it, we audit it, we refuse to persist known-bad patterns. But we'd be lying if we said the problem was solved in some absolute sense. It's managed. It's monitored. It's a lot better than it was before we watched our own system confidently tell a user to re-paste information it already had.

Why should you care if you're not building agents?

Because you're using them. If you use any AI assistant with memory — any tool that remembers your preferences, your past conversations, your project context — you're trusting a summarization pipeline to faithfully represent your past interactions. And as the Governance Decay research shows, the component deployed to keep agents running is the one that can erase the rules keeping them in bounds.

The questions to ask of any AI tool with persistent memory:

• What happens to safety constraints during context compaction?
• Can a failed retrieval overwrite a successful one in the memory store?
• Are model-generated outputs automatically re-ingested as trusted memory?
• Is there provenance tracking on memory entries?
• What's the contamination-propagation model if one memory entry goes bad?

If the vendor can't answer these, the memory feature is a liability, not a benefit.

For a broader look at how memory architecture, privacy constraints, and model routing interact in production AI systems, see our pillar post on building AI memory that doesn't compromise privacy. For more on how external attacks exploit these same weaknesses, the sibling deep-dive on defending against memory injection attacks covers the adversarial side in detail.

The self-poisoning bug we hit was — in one sense — small. One user, one thread, one summary that shouldn't have been persisted. But the mechanism was general. Any system that writes its own failures into its own memory will, given enough time, become an expert at failing. The fix wasn't to make the model smarter. It was to make the memory pipeline skeptical of itself.

If you want to see how this works in practice: start a free 7-day trial — no card required.

Michael C.

Founder & Principal Engineer, Selina Labs

Michael builds Selina, a privacy-first AI that remembers you across conversations. He ships security-sensitive AI in production — real attacks, real fixes, measured in minutes and dollars — and writes about privacy, security, and LLMs from that seat. Top Rated Plus and expert-verified on Upwork.

• www.upwork.com
• x.com
• medium.com
• selinalabs.io