Defensive AI Risks Prompt Policy Reconsideration

Defensive AI can turn against its user, new research warns.

The AI Now Institute’s policy brief on friendly fire exposes a chilling paradox: frontier agentic AI tools, designed to defend networks, can be exploited to subvert the very systems they protect. The report focuses on large language model agents developed by high profile players like Anthropic and OpenAI. It shows that attackers can leverage the models’ own weaknesses to inject harmful instructions, potentially unleashing malicious code on a defender’s own network when those tools are used to assess risk, vet sources, or monitor threats. In plain terms, a tool meant to guard critical infrastructure can become an entry point for compromise if prompt injection or other input-based exploits slip through the cracks. The finding is not a minor bug but a systemic concern: the very design that enables rapid defense responses also expands the attack surface in ways current safety controls cannot fully cover. As the United States and allied governments push AI enabled defenses across intelligence and national security apps, the paper argues that the risks are existential for critical systems if we proceed without a more robust governance framework.

The implications hit hard for who bears the risk and who benefits. National security agencies, operators of critical infrastructure, defense contractors, and AI vendors all stand at a crossroads. If frontier agents become routine in cyber defense, a successful prompt injection could cascade into system outages, data exfiltration, or control loss, undermining mission readiness at the moment of greatest need. The brief makes clear that the vulnerabilities cannot be cured by ordinary mitigations layered on top of current model designs. In other words, the problem is not a patchable nuisance but a fundamental design concern that could undermine confidence in agentic AI as a safety tool in high stakes contexts.

What should policy and industry do next? The paper implies a shift in both deployment philosophy and governance. First, it argues for a reexamination of when and where such agents are trusted to operate in safety critical domains. Second, it calls for rigorous, independent testing and red teaming that specifically target prompt injection risks and the possibility of code execution through defensive workflows. Third, it urges safer by design requirements, including strict containment controls, auditable decision trails, and layered fail safe modes that can halt or quarantine AI actions when unexpected prompts or inputs arrive. Fourth, governance should tie deployment to formal risk management: clear ownership, liability, and accountability for outcomes when agentic tools are used in cyber defense. Finally, incident reporting and post incident reviews must be mandatory so lessons can drive continual improvement rather than piecemeal fixes.

From the practitioner perspective, a few concrete takeaways matter now. One, any deployment in safety critical contexts should begin with a formal risk assessment that treats prompt injection as a primary threat vector, not a peripheral concern. Two, vendors and operators should fund and publish independent red team assessments that stress test defense-oriented workflows against adversarial inputs. Three, organizations should require safe by design features and explicit containment boundaries for agents running in production, along with clear kill switches and audit trails. Four, regulators and industry bodies should consider licensing, certification, and post market surveillance for AI defense tools, with enforcement mechanisms that ensure timely remediation when risks are identified. The overarching theme is a tradeoff: speed and automation in defense versus the hard limits of current AI safety guarantees. The policy brief suggests that until these risks are meaningfully addressed, governments and critical infrastructure operators should pause or tightly constrain deployment in high consequence contexts and demand stronger oversight before broader rollout.

The bottom line is blunt: the very tools intended to shield networks can create new routes to compromise if used in safety critical settings without robust governance. Policymakers and industry leaders must act now to define deadlines, enforcement, and a disciplined path forward that reconciles rapid defense with sound risk management.