Agentic AI Risks Backfire in Defense Use

A defense tool built on frontier AI can be turned against its user through prompt injections. The AI Now Institute’s policy brief "Friendly Fire" argues that popular agentic AI models from Anthropic and OpenAI can be hijacked when deployed for defensive purposes, turning the very tool designed to shield systems into a backdoor for attackers.

The filing states that when these models are used to assess the security of open or third party sources, they can expose users to the risks they are meant to defend against. Attackers can manipulate the models via prompt injections, which feed harmful instructions into the system through the data or input the tool ingests. In practice, this means a system that relies on a defensive AI for vulnerability discovery or threat assessment could inadvertently execute malicious code or create new footholds for intruders. The risk surface is not theoretical; it sits inside the operational workflows that are meant to harden networks and critical infrastructure.

The briefing also stresses a broader, worrying context. As the United States accelerates the use of AI enabled defensive tools across intelligence and defense circles, ignorance of these weaknesses could produce existential risks for critical systems. The authors argue that current AI designs and the common mitigations safety engineers deploy will not reliably protect against these prompt injection exploits. In other words, the risk isn’t a patch you can apply tomorrow; it demands a rethink of how agentic AI is used in safety critical and national security contexts. The filing states that governments and other organizations racing to embed these tools for cyber defense must reconsider their deployment until the vulnerabilities are meaningfully addressed.

For practitioners, the implications are concrete. Compliance programs and tech leadership should treat agentic AI deployments in security contexts as high risk experiments with potential, unintended consequences. The brief highlights that the defense utility of these agents may be compromised precisely when they are most needed, creating a tension between speed of adoption and assurance of safety. The result is a tricky balance between gaining useful cyber threat insights and preserving strict control over what the AI can execute in a live environment.

From a practitioner perspective, several realities jump out. First, the attack surface is not limited to external threats; the tool itself can become a vector if prompt input is not properly contained. Second, mitigations that rely on post hoc safety filters are unlikely to stop sophisticated prompt injections, so teams should seek architecture level safeguards, such as strict sandboxing and strict separation between discovery tasks and execution environments. Third, any deployment in critical infrastructure or national security contexts should be accompanied by rigorous governance, independent risk assessments, and enhanced monitoring of AI prompts and outputs. Fourth, organizations must plan for an uncertain horizon where the speed of AI enabled defense tools outpaces policy and standard setting, which can leave operators exposed to evolving exploit techniques.

The briefing also points to policy steps that could help close the gap between capability and safety. It calls for careful reconsideration of deployment in sensitive contexts and suggests that policymakers may need to establish compliance deadlines and robust enforcement mechanisms to ensure that risk controls (logging, access controls, and containment measures) are actually in place before extensive use. In short, the era of bluntly deploying agentic defense tools without guardrails may be over, at least for critical applications.

As the industry weighs next moves, the core message is clear: these tools offer powerful defensive potential but also introduce new attack paths. The window for sloppy, rapid deployments is closing, especially where national security and critical infrastructure are concerned. The policy brief urges a measured, governance driven approach that pairs architectural safeguards with enforceable policy steps to prevent friendly fire from becoming a national security problem.