You Cannot Patch What You Cannot Red Team: Why Adversarial Testing Is Now a Production-Day Activity for AI Agents
The most consequential number from NIST's March 2026 red-teaming competition with Gray Swan and the UK AI Security Institute: 250,000 attack attempts in five days, against four frontier agents, by a coordinated community of red teamers. The headline result was the discovery of novel jailbreaks and prompt injection patterns that the model's developers had not anticipated. The structural result was more important: a test that passes on Monday may fail on Friday if the model is fine-tuned, the system prompt changes, or a new tool is added to the agent's toolkit. Red teamers working on AI systems must shift from point-in-time assessments toward continuous adversarial evaluation cycles that track the agent's behavior surface as it evolves.
This is the operational reality of AI agent security in 2026. The traditional penetration testing model — annual or quarterly assessments, point-in-time snapshots, reports delivered to leadership — does not work for AI agents. The agent's behavior surface changes whenever the model is updated, the system prompt is modified, a tool is added, a policy is changed, or the agent's data sources shift. Each change introduces new attack surface; each attack surface requires testing. The testing cannot be a quarterly event; the testing must be continuous.
The organizations that will defend AI agents against 2026-era attacks are the ones that recognize adversarial testing as a production-day activity — built into the deployment pipeline, integrated with the runtime's audit trail, and continuously operating against the deployed agent. The traditional penetration testing model is too slow, too expensive, and too narrow. The continuous adversarial evaluation model is fast enough to keep pace with the agent's evolution, cost-efficient enough to operate continuously, and broad enough to cover the agent's full behavior surface.
The Five Properties of AI Agent Attack Surface
AI agent attack surface has five properties that distinguish it from traditional software attack surface. The properties are the reason traditional penetration testing is insufficient.
Property 1: The surface is language-based, not code-based. Traditional software is attacked through code: buffer overflows, SQL injection, deserialization flaws. AI agents are attacked through language: prompts, instructions, requests. The attack surface is the space of all natural language inputs that the agent will accept and act on. The space is unbounded; the testing cannot enumerate it.
Property 2: The surface is reasoning-based. A traditional vulnerability is a bug in code: the code does something specific, the bug has a specific location. An AI agent vulnerability is a reasoning pattern: the agent reasons in a specific way under specific conditions, the reasoning can be steered through language. The reasoning pattern is not localized; the reasoning pattern is distributed across the model's weights and the agent's prompt context.
Property 3: The surface is contextual. A traditional vulnerability is reproducible: the same input produces the same behavior. An AI agent vulnerability is contextual: the same input produces different behavior depending on the agent's history, the conversation's state, the model's randomness. The attack surface changes with the context; the testing must account for the context.
Property 4: The surface evolves with the model. Traditional software has a version-controlled behavior surface: the v1.2 software has the v1.2 vulnerabilities, and the v1.3 release documents the differences. AI agent behavior surface evolves whenever the model is fine-tuned, retrained, or replaced. The evolution is not always documented; the testing must cover the current state, not the last known state.
Property 5: The surface includes tool interactions. Traditional software has function call APIs that can be tested directly: the function takes specific inputs and produces specific outputs. AI agent tool interactions are mediated by language: the agent decides which tool to call, with what arguments, in what sequence. The tool interactions are part of the language-based surface; the testing must address them as part of the language-based attack surface.
These five properties together define what adversarial testing for AI agents must address. The traditional penetration testing approach — point-in-time, code-focused, deterministic — does not address any of the five properties adequately. Continuous adversarial evaluation is the model that addresses all five.
The Three-Tier Evaluation Hierarchy
The Cloud Security Alliance's ARIA framework (published in March 2026) established a three-tier evaluation hierarchy that has become the de facto reference for structuring AI agent adversarial evaluation programs. The tiers are not sequential; the tiers operate continuously, in parallel, against the deployed agent.
Tier 1: Model testing. Automated, continuous, low-cost testing of the underlying model's behavior against a battery of adversarial inputs. The inputs are designed to elicit known vulnerability patterns: jailbreaks, prompt injection, harmful content generation, information extraction. Model testing is fast (thousands of tests per minute), cheap (computed at inference cost), and broad (covers the model's full input space). Model testing catches regressions, identifies changes in model behavior, and provides the foundation for higher-tier testing.
Tier 2: Adversarial red teaming. Human-led, expert-driven, focused testing of the agent against sophisticated attacks designed to find novel vulnerabilities. Red teaming is slow (hours or days per campaign), expensive (skilled human time), and narrow (focused on the most likely attack scenarios). Red teaming finds the vulnerabilities that automated testing misses: novel attack patterns, multi-step exploits, social engineering at the agent level. Red teaming provides the depth that automated testing lacks.
Tier 3: Field testing. Production-environment, real-world testing of the agent against actual attacks observed in the wild. Field testing captures the attacks that the red teamers anticipate: novel jailbreaks circulating in research communities, attack patterns observed against other organizations, exploit chains discovered by security researchers. Field testing requires sharing of threat intelligence; the testing benefits from the broader community's experience.
The three tiers together provide comprehensive coverage of the agent's attack surface. Tier 1 catches the regressions; Tier 2 finds the novel vulnerabilities; Tier 3 captures the real-world attacks. The tiers operate continuously; the coverage is always current.
The Adversarial Evaluation Pipeline
A production adversarial evaluation pipeline has seven stages. Each stage is automated; the stages operate continuously against the deployed agent.
Stage 1: Test corpus curation. The pipeline maintains a corpus of adversarial test inputs. The corpus includes: known jailbreak patterns, prompt injection templates, tool misuse scenarios, memory poisoning payloads, social engineering attempts. The corpus is updated continuously as new attacks are discovered; the corpus is shared with the community (with appropriate controls for sensitive content).
Stage 2: Input generation and mutation. The pipeline generates new test inputs through mutation and combination of existing tests. Mutations include: paraphrasing, code-switching, embedding in unrelated contexts, multi-step chaining. The mutations expand the test corpus without manual authoring; the expansion is guided by the failure modes observed in prior tests.
Stage 3: Agent evaluation. The pipeline submits the generated inputs to the deployed agent. The agent's responses are captured; the responses are evaluated against the test's success criteria (did the agent reveal sensitive information, did the agent take a forbidden action, did the agent reason in a vulnerable pattern). The evaluation is automated.
Stage 4: Failure analysis. The pipeline analyzes the failures — the responses that violated the success criteria. The analysis identifies the reasoning pattern that led to the failure, the input features that triggered the failure, and the conditions under which the failure occurs. The analysis is the input to the remediation stage.
Stage 5: Remediation. The pipeline tracks remediation activities: prompt updates to prevent specific failures, tool restrictions to limit agent capabilities, policy changes to enforce safer behavior. The remediation is not always complete; the pipeline maintains a list of known failures with their remediation status. The list is the agent's residual risk.
Stage 6: Reporting. The pipeline reports adversarial evaluation results continuously: the number of tests executed, the failure rate, the trend over time, the open failures, the remediation progress. The reports are integrated with the security operations tooling; the reports trigger alerts when failure rates spike or new failure patterns emerge.
Stage 7: Feedback. The pipeline feeds failure data back into the test corpus generation. Failures become tests; tests are mutated; mutations are evaluated. The feedback loop is what makes the testing continuously improving rather than static.
The seven stages operate continuously; the pipeline runs every day against the deployed agent; the failures discovered today inform the tests run tomorrow. The testing is continuous; the security posture is current.
The Attack Categories That Red Teamers Find
Red teamers working on AI agents consistently find vulnerabilities in five categories. Each category requires different remediation; each category requires dedicated testing.
Category 1: Jailbreak patterns. The agent is steered to bypass its safety training or operational guidelines. The patterns include role-play, character adoption, hypothetical scenarios, multi-step escalation, and recently popularized techniques that exploit the model's instruction hierarchy. Red teamers find new jailbreaks continuously; the jailbreak corpus grows weekly.
Category 2: Prompt injection. The agent's input contains instructions that override the agent's intended behavior. The injection may come from user input, from retrieved documents, from tool responses, or from external content the agent processes. The injection may be direct (explicit instructions) or indirect (subtle steering). Red teamers find injection patterns that exploit specific tool configurations, specific retrieval pipelines, specific reasoning patterns.
Category 3: Tool misuse. The agent uses its tools in ways the agent's designers did not intend. The agent may call tools in unexpected sequences, pass unexpected arguments, chain tool calls to achieve forbidden outcomes. Tool misuse is the most common vulnerability category in 2026 red-teaming reports; the agent's tool inventory is the attack surface.
Category 4: Memory poisoning. The agent's long-term memory is corrupted with content that influences future reasoning. The poisoning may come from a single user (the agent trusts one user's input more than warranted), from a retrieved source (the agent ingests poisoned content), or from an adversary (the agent is socially engineered into storing malicious content). Memory poisoning persists; the poisoning affects future sessions.
Category 5: Multi-agent hijacking. In multi-agent architectures, one agent is steered to send malicious content to other agents; the other agents execute the malicious content as if it were legitimate. The hijacking exploits the trust between agents; the hijacking propagates through the agent network.
The five categories are not independent. A successful red-team attack may combine category 2 (prompt injection) with category 3 (tool misuse) to achieve category 5 (multi-agent hijacking). The combination is what makes the attacks sophisticated; the combination is what makes the testing hard.
The Tool Stack for Adversarial Evaluation
The tool stack for AI agent adversarial evaluation in 2026 has matured. Several open-source and commercial tools are widely deployed.
PyRIT (Python Risk Identification Tool for LLMs). Microsoft's open-source framework for adversarial evaluation. PyRIT provides the infrastructure for automated testing, including test corpus management, response capture, failure analysis, and reporting. PyRIT integrates with major agent frameworks and model APIs. PyRIT is the foundation of many organizations' evaluation pipelines.
Garak. NVIDIA's open-source LLM vulnerability scanner. Garak provides a library of probe templates, attack generators, and evaluation metrics. Garak is focused on Tier 1 model testing; it is the automated regression suite. Garak's probe library is continuously updated with new vulnerability patterns.
Cogensec / Global Red Teaming Network. Community platform for sharing adversarial test cases, red-teaming methodology, and findings. The network connects red teamers across organizations; the platform is the infrastructure for Tier 3 field testing. The network's value grows as more organizations participate.
Commercial platforms. Several vendors (Galileo, Arthur, Robust Intelligence, Patterns) have built commercial adversarial evaluation platforms that provide the pipeline as a managed service. The commercial platforms offer sophisticated analysis, integrated reporting, and the operational scale that larger organizations require.
The tool stack is not a single tool; the tool stack is a layered ecosystem. PyRIT for the framework, Garak for the library, community platforms for the network, commercial platforms for the operations. The combination is what production adversarial evaluation requires.
The Integration with the Runtime
Adversarial evaluation produces results; the results must be acted upon. The integration with the runtime is what closes the loop.
Failure → Policy update. When the adversarial evaluation discovers a failure, the failure's analysis is fed to the runtime's policy engine. The policy is updated to prevent the specific failure; the policy update is version-controlled and auditable. The next adversarial evaluation run verifies that the policy update prevents the failure.
Vulnerability → Tool restriction. When the adversarial evaluation discovers a tool misuse pattern, the vulnerability's analysis is fed to the runtime's tool configuration. The tool may be restricted (scoped more narrowly), the tool may be removed (if the risk outweighs the benefit), or the tool may be wrapped (with additional validation). The next adversarial evaluation run verifies that the restriction prevents the misuse.
Pattern → Detection rule. When the adversarial evaluation discovers a reasoning pattern that led to failure, the pattern is added to the runtime's anomaly detection. The detection rule identifies the pattern in production; the rule's alerts are routed to the security operations team. The rule's effectiveness is validated against future adversarial evaluations.
Test → Continuous monitoring. The adversarial evaluation's tests are integrated with the runtime's continuous monitoring. The monitoring runs a subset of the tests against the deployed agent continuously; the monitoring alerts on failures. The monitoring is the production-time adversarial evaluation.
Facio (the HITL-first agent runtime) implements the integration as a first-class architectural component. Adversarial evaluation results flow into the runtime's policy engine, tool configuration, and anomaly detection. The runtime's audit trail (covered in the Facio analyses from June 2026) records the evaluations, the failures, the remediations, and the verifications. The integration is what makes the adversarial evaluation cycle closed.
The Organizational Challenges
Adversarial evaluation is technically feasible; organizationally, it requires sustained investment. Three challenges consistently arise.
Challenge 1: The vulnerability discovery rate. Continuous adversarial evaluation discovers vulnerabilities daily. Each vulnerability requires triage, analysis, and remediation. The volume is high; the remediation capacity is limited. The organization must prioritize: critical vulnerabilities are remediated immediately; medium vulnerabilities are batched; low vulnerabilities are accepted as residual risk. The prioritization is a continuous activity.
Challenge 2: The model update cycle. When the underlying model is updated (fine-tuned, retrained, replaced), the entire test corpus must be re-run. The re-run is necessary because the model update may have introduced new vulnerabilities or fixed existing ones. The re-run is expensive (in computation and in analysis time). The organization must budget for the re-runs as a recurring cost.
Challenge 3: The red teamer talent. Skilled AI red teamers are scarce. The talent must be hired, trained, and retained; the talent is competitive with the offensive AI research community. The organization must invest in the red team as a strategic capability, not as a one-time project.
These challenges are organizational; they are not technical. The organizations that solve them are the ones that treat adversarial evaluation as a continuous program, not as a quarterly assessment.
The Bottom Line
You cannot patch what you cannot red team. AI agents are attacked through language, reasoning, context, model updates, and tool interactions; traditional penetration testing addresses none of these adequately. Continuous adversarial evaluation is the model that addresses all five.
The three-tier hierarchy — model testing, adversarial red teaming, field testing — provides comprehensive coverage. The seven-stage pipeline — test corpus curation, input generation, agent evaluation, failure analysis, remediation, reporting, feedback — provides continuous operation. The five attack categories — jailbreaks, prompt injection, tool misuse, memory poisoning, multi-agent hijacking — define the scope. The tool stack — PyRIT, Garak, community networks, commercial platforms — provides the infrastructure. The integration with the runtime — failure to policy, vulnerability to tool restriction, pattern to detection — closes the loop.
The organizations that will defend AI agents in 2026 are the ones that have deployed continuous adversarial evaluation, integrated the results with the runtime's defenses, and built the organizational capability to operate the program sustainably. The NIST-Gray Swan-UK AISI competition's 250,000 attack attempts is the warning: the attack surface is large; the attacks are sophisticated; the testing must match.
Facio (the HITL-first agent runtime) integrates adversarial evaluation with the runtime's policy engine, tool configuration, and anomaly detection. Placet.io (the HITL inbox and messenger) routes novel vulnerability discoveries to human reviewers for rapid remediation. Together, they close the loop between test discovery and runtime defense. The architecture is the difference between a hardened agent and a defensible one.
Further reading:
- Stingrai: AI Red Teaming — How to Test LLM and Agentic Apps in 2026
- GLACIS: AI Red Teaming Guide — Testing AI Systems (April 2026)
- Cloud Security Alliance: NIST AI Agent Security — Red-Teaming Guidance and Enterprise Compliance
- Baeseokjae: LLM Red Teaming Guide 2026 — Security Testing for AI Agents
- AI Agent Runtime Guardrails: Why Policy at the Model Layer Fails
- When One Compromised Agent Becomes a Hundred Compromised Systems