Multi-turn jailbreaks beat every model tested: what Cisco's benchmark gap means for how founders should evaluate security

If you're picking an LLM vendor based on a model card's safety score, you're answering the wrong question. Cisco tested 15 closed frontier models and found that security under single-turn evaluation bears almost no relationship to security under multi-turn attack. The rankings change. The gaps are large. And the configuration details that matter most aren't documented anywhere a buyer can find them. This piece walks through the findings, explains why the measurement itself is broken, and proposes a concrete evaluation practice you can run before signing your next vendor contract.

Key Takeaways

• Cisco's May 2026 study found multi-turn attack success rates ranging from 7.89% to 88.30% across 15 closed frontier models, while single-turn rates on the same models ranged only from 2.19% to 64.91%. Eight of 15 models showed a gap greater than 15 percentage points between the two regimes.
• Model rankings flipped between single-turn and multi-turn testing. A model that looks strong on the benchmark in its model card may be among the weakest when an attacker gets a second turn.
• A single configuration toggle (reasoning mode) cut one model's multi-turn attack success rate roughly in half, from 88.3% to 43.5%. No public model card or benchmark documented this effect.
• Single-turn safety benchmarks were designed to answer "does this model refuse an obvious bad prompt." Your actual question is "does this model hold policy across a conversation with a motivated adversary." These are different questions with different answers.
• Founders should test their own production configuration (system prompt, temperature, guardrail stack) against multi-turn attack families, not rely on upstream marketing numbers.

What did Cisco actually find?

Cisco's AI security team published "Proprietary Problems: No Frontier Model Is Multi-Turn Immune" in late May 2026, evaluating 15 closed frontier models from OpenAI, Anthropic, Google, Amazon, and xAI. They ran both single-turn and multi-turn attack batteries against every model. The results were unambiguous: no model was multi-turn immune.

The numbers tell the story flatly. Multi-turn attack success rates spanned 7.89% to 88.30%. Single-turn rates on the same models spanned 2.19% to 64.91%. Eight of the 15 models showed an absolute gap greater than 15 percentage points between the two testing regimes. That's not noise. That's a measurement failing to capture the thing it claims to measure.

This study followed Cisco's earlier open-weight benchmark, "Death by a Thousand Prompts" from November 2025, which found multi-turn attacks against eight open-weight models succeeded between 25.86% and 92.78% of the time. The pattern held across both open and closed models: multi-turn attacks succeeded at roughly 2x to 10x the rate of single-turn baselines.

Why do model rankings flip between single-turn and multi-turn tests?

Because the two tests measure different capabilities, and those capabilities are largely independent. A single-turn safety test measures whether a model's refusal classifier fires on an isolated prompt. A multi-turn test measures whether the model can maintain policy state across a conversation where the adversary is strategically building context, shifting frames, and decomposing harmful requests into innocuous-looking pieces.

Models that looked strong on the single-turn benchmarks typically used in model cards and procurement reviews did not necessarily hold up once an attacker could keep talking. This is the core finding, and it has a direct structural cause: single-turn safety alignment is mostly a pattern-matching problem (does this prompt match known attack patterns), while multi-turn safety is a state-tracking problem (does the accumulated context across turns constitute a policy violation, even if no individual turn does).

One specific multi-turn tactic illustrates this cleanly. Breaking a harmful request into innocuous pieces across turns and reassembling them later achieved a 95% success rate against one tested model. No single turn in that attack sequence would trigger a refusal. The harm is in the composition, and composition requires the model to hold and evaluate state across turns. Most alignment training doesn't optimize for this.

What does this mean for the benchmarks on model cards?

The benchmarks on model cards are answering a question you didn't ask. They were designed to measure single-turn refusal rates, typically against a fixed taxonomy of harmful prompt categories. That's a useful signal if your deployment is a single-shot API call (content classification, structured extraction, one-turn Q&A). It's close to irrelevant if your deployment involves multi-turn conversations, which is most products that users actually interact with.

The disconnect is a measurement-validity problem, not a vendor-honesty problem. The benchmark was built for one purpose and is being used for another. When you see a model card claiming a 97.8% refusal rate (the complement of the 2.19% single-turn ASR floor Cisco found), that number is real. It just doesn't tell you what happens on turn five, or turn twelve, when someone patient is working the conversation toward a specific outcome.

On the single-turn side specifically, Cisco found failures cluster in a few attack styles: "Imposter AI" (persona/role-play) procedures produced the highest weighted attack success rate at 37.50%, followed by Soft Paraphrase at 29.21% and System Prompt attacks at 27.69%. Even within single-turn testing, the aggregate number hides meaningful variation by attack family. Multiply that variation by the single-to-multi-turn gap and you start to see how little a single headline score tells you about your actual exposure.

How does configuration affect safety, and why isn't it documented?

Dramatically, and because no one requires it to be. One model's multi-turn attack success rate dropped from 88.3% to 43.5% simply by toggling a reasoning mode on. That's a configuration flag, not a model change. Same weights, same training, same RLHF. A single boolean parameter cut the attack surface roughly in half.

Cisco's report notes that this effect wasn't captured in any public benchmark or model card they reviewed. The implication is worth sitting with: the difference between "catastrophically vulnerable" and "moderately vulnerable" was a config toggle that no procurement team would know to ask about, because no documentation mentioned it.

This finding extends beyond reasoning modes. Temperature, system prompt content, guardrail tier selection, context window management: all of these affect multi-turn safety properties, and none of them are reflected in the benchmark numbers that show up in vendor pitch decks. When a vendor tells you their model scores well on safety, the correct follow-up is: "at what temperature, with what system prompt, with which guardrail configuration, and over how many turns?" If they can't answer all four, the score is decorative.

Why is this a privacy problem specifically?

Because your production configuration is where user data lives. If a multi-turn jailbreak succeeds against your deployed system, the attacker isn't just getting the model to say something harmful in the abstract. They're potentially extracting information from the conversation context, the system prompt (which often contains business logic and sometimes credentials), or the user's prior turns. The safety failure is also a data-exposure failure. A model that holds safety policy over extended conversations is doing something functionally identical to access control: maintaining invariants about what information flows where, under adversarial pressure, over time.

What should founders actually do differently when evaluating LLM vendors?

Test your own configuration, against multi-turn attack families, before you buy. Cisco's report recommends that organizations ask labs to publish attack success rates broken down by strategy family on every release, gate deployments on regressions above a three-percentage-point threshold in top attack categories, and flag any model with a cross-regime gap larger than 15 percentage points for manual review. That's a reasonable starting point. Here's how to operationalize it.

Step 1: Define the attack surface your product actually exposes

If your product is a single-turn API behind a classification pipeline, single-turn benchmarks are relevant to you and multi-turn testing is less urgent. If your product involves conversation (chat, assistants, agents, support bots), you need multi-turn evaluation and single-turn scores are insufficient. Most products shipping today fall into the second category. Be honest about which one you are.

Step 2: Test at the deployment layer, not the model layer

Don't test the base model. Test your production stack: your system prompt, your temperature setting, your guardrail configuration, your context window management. The Cisco reasoning-mode finding proves that model-level benchmarks diverge from deployment-level reality by margins large enough to change your risk assessment entirely. If you're using a frontier model with a custom system prompt and a guardrail proxy, test that assembly. Not the model. The assembly.

Step 3: Break evaluation into attack strategy families

Aggregate scores hide the structure of your vulnerability. Cisco's single-turn data shows Imposter AI attacks succeeding at 37.50% while other categories succeed at much lower rates. Your product's exposure to each category depends on your UI, your user base, and your conversation flow. A customer support bot has different exposure to persona-based attacks than a code assistant. Map attack families to your product's actual interaction patterns and weight your evaluation accordingly.

Step 4: Require cross-regime comparison from your vendor

Ask for both single-turn and multi-turn ASR on the model version you're deploying. If the gap exceeds 15 percentage points (Cisco's threshold for flagging), that model deserves manual review before production deployment. If the vendor can't provide multi-turn numbers, that's information too. It means they haven't run the test, or they have and don't want to share the results. Neither is reassuring.

Step 5: Retest on every model update

Cisco recommends gating deployments on regressions above three percentage points in top attack categories. This means you need a repeatable evaluation pipeline, not a one-time assessment. If you're swapping model versions quarterly (which most teams shipping on frontier APIs are), your safety evaluation needs to run at least quarterly. Automate what you can. The OWASP framework for evaluating AI red-teaming vendors provides a useful baseline for what "reproducible multi-turn evaluation" should look like.

Does independent research confirm Cisco's findings?

Yes. MultiBreak, an independent academic benchmark published in May 2026 and presented at ICML, found that some attack categories which look benign under single-turn testing became substantially more effective in multi-turn scenarios. This comes from a different research group using a different methodology, which makes the convergence of findings more credible than either study alone.

The red-teaming industry has moved accordingly. Multiple 2026 surveys of AI red-teaming vendors now treat multi-turn and agentic coverage as a baseline requirement rather than a differentiator. Strong vendors are distinguished by reproducible multi-turn evaluations and reporting that maps technical failures to business impact, not by whether they test multi-turn at all. The bar has moved. If your evaluation vendor doesn't offer multi-turn testing, they're behind.

What about regulatory pressure?

The EU AI Act, effective August 2026, classifies AI systems used in regulated decisions as high-risk and requires robustness testing against adversarial inputs as a specific legal requirement. If you're deploying in the EU (or serving EU users, which is most SaaS products), "we checked the model card" is not a compliance defense. You need documented evidence that you tested your deployed configuration against adversarial inputs, including multi-turn adversarial inputs, and that you have a process for retesting on model updates.

This isn't hypothetical regulatory risk. It's a dated compliance obligation. If you're a founder planning to ship an AI product in the EU in the second half of 2026, your evaluation process needs to produce artifacts that an auditor can review. A vendor's model card is not your artifact. Your own test results, against your own configuration, are.

How does this apply to privacy-first architectures?

Multi-turn safety failures are particularly acute when the model has access to persistent context, memory, or user data across turns. The more state the model carries, the more surface area a multi-turn attacker has to work with. This creates a tension: memory and personalization make the product better, but they also make multi-turn attacks more consequential when they succeed.

We deal with this at Selina by treating safety as an architectural property of the deployed system, not a property of any individual model. We use a stack of frontier models, routed per task, with runtime controls that bound behavior independently of any single model's alignment training. Memory is encrypted at rest, though we'll state plainly that memory is not end-to-end encrypted (a slice of each request reaches a frontier provider at inference). Files and transfers via SelinaSEND are end-to-end encrypted. Non-content operational metadata is kept for a short retention window.

We don't claim this makes us immune to multi-turn attacks. No one should claim that. What it means is that we don't rely on a model's refusal classifier as the sole line of defense, and we don't treat a model card's safety score as evidence that our deployed system is safe. We test our own configuration. We recommend you do the same with whatever vendor you choose.

What's the practical checklist?

Here's a concrete procurement checklist, derived from Cisco's recommendations and adapted for founders evaluating LLM vendors for production deployment:

1. Request multi-turn ASR by attack strategy family for the specific model version and configuration you plan to deploy. If the vendor provides only aggregate single-turn numbers, note the gap in your risk assessment.
2. Compute the cross-regime gap. If the difference between single-turn and multi-turn ASR exceeds 15 percentage points, flag the model for manual review before deployment.
3. Document the configuration. Record the exact system prompt, temperature, guardrail tier, reasoning mode, and context window settings used in both the vendor's benchmark and your own testing. Differences between these configurations invalidate the comparison.
4. Test decomposition attacks. Multi-turn attacks that break a harmful request into innocuous pieces across turns are the highest-leverage attack family and the one most invisible to single-turn scanning. Include them explicitly.
5. Set a regression gate. Define a threshold (Cisco suggests three percentage points in top attack categories) above which a model version update does not deploy to production without review.
6. Retest on every update. Model versions change quarterly or faster. Your evaluation pipeline needs to match that cadence.
7. Map technical ASR to business impact. A 30% ASR against persona attacks matters differently for a children's education product than for an internal code review tool. Weight your evaluation by consequence, not just by rate.

What don't we know yet?

Several things worth being honest about. Cisco's study tested 15 models, which is comprehensive but not exhaustive. The attack taxonomy they used is one of several possible frameworks, and different taxonomies might produce different rankings. The multi-turn attacks were automated, and human-operated attacks (which are slower but often more creative) might show different patterns. The interaction between safety and capability is still poorly characterized: it's possible that some of the configuration changes that improve multi-turn safety degrade capability in ways that matter for production use.

We also don't know how quickly the frontier providers will respond to this data. If the labs begin optimizing for multi-turn safety specifically (and there's some evidence this is already happening in post-training pipelines), the gap may narrow. But measuring whether it narrows requires exactly the kind of cross-regime, configuration-aware, strategy-family-level evaluation that doesn't currently exist in model cards. The measurement problem comes first.

The bottom line for founders

The benchmark you're citing in your procurement review was designed to answer a question about single-turn refusal. Your product exposes a multi-turn conversation surface. These are different threat models with different measurement requirements, and Cisco's data shows they produce different, sometimes contradictory, rankings of the same models.

Stop evaluating models. Start evaluating deployed configurations, against multi-turn attack families, at the cadence your model versions change. The score on the model card is real. It just isn't yours.

If you're building a product where conversation memory and user privacy intersect, we've spent time thinking about this. Start a free 7-day trial, no card required.

Michael C.

Founder & Principal Engineer, Selina Labs

Michael builds Selina, a privacy-first AI that remembers you across conversations. He ships security-sensitive AI in production — real attacks, real fixes, measured in minutes and dollars — and writes about privacy, security, and LLMs from that seat. Top Rated Plus and expert-verified on Upwork.

• www.upwork.com
• x.com
• medium.com
• selinalabs.io