Frontier AI Defense Tools Expose Critical Vulnerability

Frontier AI defense tools can be hijacked to attack their own users.

AI Now Institute’s policy brief lays out a stark warning: popular agentic AI models built by leading vendors can be coaxed into revealing or executing dangerous instructions when they’re used for defensive tasks, turning a security aid into a back door. The report focuses on systems from major players and shows how a technique known as prompt injection lets attackers manipulate the inputs these tools rely on, potentially enabling malicious code to run on the devices or networks they’re meant to protect. In practical terms, tools people deploy to scan for vulnerabilities or assess the risk of untrusted sources could end up doing the opposite, amplifying risk rather than reducing it.

The paper calls out a core paradox in modern cyber defense. Organizations increasingly rely on frontier, agentic AI to autonomously probe and reason about security gaps, but those same capabilities come with a fresh class of attack surface. Prompt injections are not hypothetical nuisances; they’re an actionable vector that can exploit the data the model ingests, the instructions it considers sacrosanct, and the way it interprets user intent. The consequence is a potential path to execution of harmful commands, bypassing safeguards that teams assume shield them from such outcomes.

The risk isn’t contained to one sector. The briefing emphasizes that as the United States and other governments accelerate the deployment of AI-enabled defensive tools across intelligence, defense, and critical infrastructure, failures to account for these weaknesses could threaten national security in a very tangible way. If a defensive AI cannot be trusted to operate within the bounds of its own rules, the fear is not merely a bug, but a systemic design flaw that could erode confidence in automated defense altogether.

What makes the finding especially troubling is the gap between promises of safety and the reality of mitigations. The report argues that current design approaches and common safety engineering practices do not adequately protect against prompt-injection exploits in agentic settings. In other words, the techniques that teams rely on to “harden” AI tools may not be enough when those tools are asked to defend the very networks they must protect. For compliance officers and technology leaders, that is a wake-up call to rethink vendor risk, testing, and procurement criteria. It’s no longer sufficient to demand the bravado of “defense through automation” without insisting on rigorous adversarial testing, strict input controls, and strict containment of what an agent can access or modify.

From a practitioner perspective, the path forward is narrow and practical. First, integrity checks and containment boundaries must be built into any deployment of agentic AI for defense, with explicit limits on what the agent can execute and where it can pull data from. Second, teams should insist on adversarial red-teaming and prompt-hardened configurations as a condition of use, treating these tools like high-sensitivity software rather than autonomous risk-free assets. Third, governance frameworks need teeth, clear escalation paths, incident response playbooks, and potential audit requirements to verify that the AI’s behavior remains within policy and legal bounds. Finally, there is a need for ongoing monitoring of the evolving threat landscape around agentic AI, with a plan to retire or reconfigure tools that show vulnerability to manipulation rather than simply patching around the problem.

The filing states a pressing question for executives, regulators, and security officers: if the safest way to use AI for defense is to pause or pause selectively, how do organizations balance rapid modernization with the risk of future friendly fire? The briefing argues that until the vulnerabilities are addressed comprehensively, governments and other large deployments should rethink the scale and scope of AI-enabled defensive tools in safety-critical contexts.