Defense AI Tools Vulnerable to Prompt Attacks

Defensive AI tools meant to guard nations can turn against their operators.

Defensive AI tools built for assessment and defense are facing a new and surprising risk: prompt injections that can coax them into harmful actions, turning a supposed shield into an entry point for attackers. The AI Now Institute’s policy brief, Friendly Fire, lays out how frontier agentic models from major players like Anthropic and OpenAI can be manipulated when used to test vulnerabilities or evaluate untrusted sources. Rather than quietly identifying weaknesses, these tools can be exploited to run malicious code on systems they were meant to defend. The problem, according to the brief, is not just a single bug but an architectural vulnerability in how these agents reason, decide, and act in defensive workflows.

The paper explains that attackers can leverage inputs fed to the agent to inject harmful instructions, hijacking the model’s own capabilities. When security teams rely on these agents to scrutinize third party content or open sources, they can inadvertently provide a path for compromise instead of a shield against it. The finding is especially troubling for U.S. authorities accelerating the use of AI-enabled defense tools across intelligence, cyber, and national security missions. The authors warn that the very approach designed to strengthen security could create new, systemic attack vectors if prompt-level manipulation is not contained. In short, the report argues that the current generation of agentic AI design may be ill suited for safety-critical and national security tasks without fundamental changes.

For compliance officers and tech leaders, the implications are practical and immediate. The policy brief contends that existing mitigations, often deployed as guardrails, input filters, or post-hoc safety checks, will not reliably stop these prompt-based exploits. That means organizations cannot simply layer on conventional safety wraps and assume the risk is contained. Instead, governance and procurement approaches must evolve in parallel with AI capability. Decision-makers should demand rigorous threat modeling that treats defensive AI tools as potential attack surfaces, implement stricter data governance around inputs used in safety-critical workflows, and require explicit containment measures such as sandboxed execution environments and read-only operational modes when the risk is high. The brief also points to a need for independent, adversarial testing by third parties to validate defenses against prompt injection scenarios before deployment in critical contexts.

Industry practitioners should prepare for a tightening of deployment criteria in national security and infrastructure settings. For one, supply-chain and vendor risk assessments must incorporate prompt-injection risk as a core factor, not an afterthought. Second, procurement will increasingly favor defensible architectures: limited autonomy, strict privilege boundaries, and the ability to halt or roll back automation at a moment’s notice if suspicious behavior is detected. Third, red-teaming and tabletop exercises should routinely simulate prompt-driven failures, not just code-reliant exploits, to reveal how a defender’s tool could be steered into unsafe actions. Finally, there is a clear incentive to separate defense-oriented evaluation from live, real-time operation in high-stakes environments, favoring offline or non-agentic checks for critical tasks until risks are meaningfully lowered.

What to watch next is straightforward: policymakers and agency leaders will likely press for updated guidance on deploying agentic AI in safety-critical roles, while vendors are pushed to offer more transparent risk disclosures and stronger containment features. The core question remains whether agentic AI can be trusted in the environments that guard national security, or whether the next generation of safeguards will be designed to keep the defender from becoming the attacker.