Frontier AI defensive tools backfire on users

AI defense tools can turn against their users, a new policy brief warns.

A policy brief from the AI Now Institute titled Friendly Fire argues that frontier AI agents (such as those built by major players like Anthropic and OpenAI) pose a real and actionable risk when used for defense. The core finding is striking, that these agents can be manipulated through prompt injections and other weaknesses to carry out malicious actions on the very systems they are meant to shield. In practical terms, tools designed to help organizations assess vulnerabilities or defend against open or third party sources could be leveraged to compromise the defender instead. The paper emphasizes that attackers do not need to break into a system from the outside. They can exploit the AI's own modeling and input handling to inject harmful instructions that the agent then executes.

The report highlights a troubling paradox as the United States accelerates the adoption of AI-enabled defensive capabilities across its intelligence and defense infrastructure. As these tools become more integrated into critical decision loops and security operations, their intrinsic vulnerabilities translate into existential risks for the operating networks and data they are meant to protect. The vulnerabilities are not hypothetical edge cases, they hinge on well-known adversarial techniques like prompt injections that can take advantage of the AI's reliance on large language models and automated reasoning. The concern is that current designs of LLMs and the mitigations safety engineers typically deploy do not fully shield users from these attack vectors. In other words, the very frontier technologies intended to harden defenses could, if deployed without deeper risk controls, create new pathways for compromise.

One of the most important takeaways for tech leaders and compliance officers is that this is not a theoretical debate about future capabilities. It is a call to reevaluate deployment strategies for AI-driven defense in high-stakes environments. The briefing argues that until these risks are better understood and mitigated, governments and other organizations, especially those responsible for national security and critical infrastructure, should reconsider how and where these frontier tools are deployed. In practice, that means rethinking risk assessments, red-teaming practices, and the governance structures that oversee AI-enabled defense programs. It also implies heightened scrutiny of the data these agents ingest, the contexts in which they operate, and the potential for unintended actions if adversaries succeed in manipulating inputs or surrounding workflows.

For practitioners in the field, the brief suggests concrete but cautious steps to manage risk. First, broaden adversarial testing beyond conventional privacy and safety checks to include prompt injection and other exploitation scenarios specifically tailored to defense and critical infrastructure use cases. Second, tighten governance around data provenance, model updates, and supply-chain dependencies so that vulnerable components do not propagate weaknesses into security workflows. Third, implement layered defenses that assume the AI system could be compromised, including manual overrides, independent verification of critical decisions, and robust incident response playbooks that can isolate AI-driven processes quickly. Fourth, ensure compliance programs reflect the evolving risk landscape by embedding AI risk into enterprise risk management, with explicit accountability for deployment decisions in safety-critical environments. While these measures do not erase the risk, they provide a clearer path for maintaining resilience as organizations navigate the balance between rapid AI deployment and robust defense against innovative attack methods.

In short, the Friendly Fire briefing reframes a familiar tension, the same AI tools that promise stronger cyber defense can, under certain conditions, become a liability. As agencies and enterprises weigh the benefits of automation against the possibility of self inflicted harm, the question becomes not only how to advance AI capabilities, but how to govern them responsibly when the stakes are national security and critical infrastructure.