EU moves to ban AI tools for sexual deepfakes

AI deepfakes weaponize women's images, and Europe moves to ban the tool.

The filing states that generative AI is increasingly being used to produce sexualized content about real people without consent. Men already use AI to create deepfakes, generate undressed bodies, and share callously via social platforms such as X, while AI chatbots replay sexist insults. Regulators frame this not as an isolated tech problem but as a continuation of long standing gender based violence and patriarchal power, now amplified by new tools. The consequences for women are severe, ranging from psychological distress to personal and professional harm, and the pathways to defense remain limited.

On the policy front, the European debate centers on how to square protection with innovation. The Digital Services Act currently governs how platforms disseminate illicit content, but it does not fully address the production side of the problem. The European Union is also weighing an overhaul to the AI Act that would set obligations on providers and deployers of AI systems, with the aim of curbing tools used to create sexualized deepfakes. The drafting shows a push toward a more explicit framework that could constrain the kinds of AI capabilities available to the public and the ways those tools are deployed on online services. Yet the landscape is still fragmented. In Germany, experts point to a lack of reliable data on gender based violence in the digital space and an absence of a robust legal framework to address it, illustrating how gaps at the national level coexist with broader EU level debates.

For compliance officers and platform leaders, the implications are concrete. First, there is a need to catalog which AI tools within a company or platform could be misused to create nonconsensual sexual content. That means not just tracking what is marketed as a general purpose tool, but understanding how it could be repurposed for deepfake creation or image manipulation. Second, the action set will hinge on who bears responsibility when such content is produced and shared. Under the DSA, platforms face moderation duties for disinformation and abusive content, while the AI Act would place duties on providers and deployers of AI systems depending on the risk classification of the technology. The practical effect is a push toward stricter governance of AI features that enable image manipulation, with compliance teams needing to enforce consent, privacy and safety guarantees across product lines. Third, there is an incentive to invest in safer-by-design workflows, including red-teaming of AI models for misuse, enhanced user reporting, faster takedowns, and clearer user controls around the generation and distribution of AI assisted content. Finally, enforcement will likely involve cross border cooperation and platform level accountability, given the EU's emphasis on platform duty of care and provider accountability.

What to watch next is where the EU lands on explicit prohibitions or tight licensing for AI tools that can be used to create sexual deepfakes, and how those rules interact with ongoing DSA obligations and the AI Act’s risk based requirements. For operators, the key is to align product development and moderation practices with a shifting regulatory boundary that aims to limit a particularly harmful use of AI while preserving legitimate creative and analytical applications. The tension will be in balancing innovation with meaningful protection, ensuring data subjects’ rights are respected, and staying compliant as rules evolve at EU level and across member states.