Urgent call to ban AI apps that enable fake nude content

The Children’s Commissioner Dame Rachel de Souza is calling on the government to introduce a total ban on apps that use Artificial Intelligence (AI) to generate sexually explicit ‘deepfake’ images of children.

The Children’s Commissioner’s new report exposes how Generative Artificial Intelligence (GenAI) is being misused to create sexually explicit deepfake images of real people.

While it is illegal to create or share a sexually explicit image of a child, the technology enabling them remains legal – and it is no longer confined to corners of the dark web but now accessible through large social media platforms and search engines.   

Speaking to children about this emerging technology, the Commissioner’s new report analyses the threat of nudification technology, assessing its use online and the impact on children. In focus groups, children told the Commissioner about their biggest concerns. 

This includes the high risk of harm to children: AI that generates naked or other sexually explicit deepfake images disproportionately targets girls and young women, and many tools appear only work on female bodies – contributing to a culture of misogyny both online and offline.  

The easy access to such harmful tools was also raised. These AI tools are widely available via mainstream platforms, including the biggest search engines and app stores, with GenAI making the creation of harmful content easier and cheaper than ever.  

Girls spoke about taking preventative steps to keep themselves safe from becoming victims of nudification tools by limiting their online participation – in the same way that girls often take steps to protect themselves in real life, such as not walking home alone.

The Children’s Commissioner is calling for urgent action, including banning bespoke nudification apps that enable users to generate sexually explicit images of real people; as well as creating specific legal responsibilities for developers of GenAI tools to identify and address the risks their products pose and to mitigate the risks to children.

The report also calls for the establishment of effective systems to remove sexually explicit deepfake images of children from the internet, and for the recognition of deepfake sexual abuse as a form of violence against women and girls and taking it seriously in law and policy.

Paul Whiteman, general secretary of school leaders' union NAHT, said: “School leaders share the Children’s Commissioner’s concerns over this technology, both in its use against children and against school staff. Our members will be debating this subject at our conference next week, calling for the creation and distribution of non-consensual deepfake content to be criminalised. This is an area that urgently needs to be reviewed as the technology risks outpacing the law and education around it.”