Grok undressed women and children on demand. Europe's rules are now being tested on it.

On January 26 the European Commission opened a formal investigation into X under the Digital Services Act over Grok, the AI chatbot built into the platform. Since late December, users had been able to ask Grok to put a real woman in a bikini or remove her clothes, and it did, including for images of children. Henna Virkkunen, the Commission’s executive vice-president for tech, called non-consensual sexual deepfakes a violent, unacceptable form of degradation. The case matters beyond X, because it tests how far a company is responsible for what its AI feature produces once it is switched on for millions of users.

Three weeks of image abuse

The numbers are hard to pin down and bad either way. According to Forbes, a New York Times analysis counted 1.8 million sexualised images of women produced in nine days from late December. The Centre for Countering Digital Hate put the figure at roughly 3 million images of women and children within days. The methods differ, and I would use the lower number. It is still a volume no moderation team could have handled after the fact.

The response was slow and piecemeal. In early January the Commission ordered X to preserve all documents related to Grok. Around January 8 X restricted image generation to paying subscribers, which limited the volume without addressing the problem, and later added technical measures to stop edits that put real people in revealing clothing. By then ten countries had acted. Indonesia and Malaysia blocked Grok outright, France and Ireland opened their own investigations, the UK regulator Ofcom began an inquiry, and California’s attorney general demanded that xAI stop.

What the Commission is actually investigating

The Commission is not investigating the images one by one. Under the Digital Services Act, very large platforms must assess the risks of their services and put mitigation in place before they launch features that change those risks. The question in this case, as The Register describes it, is whether X assessed and mitigated the risks of Grok’s image generation before deploying it in the EU. The Commission is also examining X’s recommender systems. The maximum fine is 6% of global annual turnover, and the Commission has already fined X €120 million under the same law for breaches of its transparency rules.

That framing is the part to pay attention to. A company that ships an AI feature to a large audience has to show it thought about the obvious misuse first. “Remove her clothes” is about as obvious as misuse gets. Image models that refuse such requests have been on the market for years, and xAI chose to ship one that did not.

The skeptical view deserves a hearing. Enforcement under the DSA is slow (the Commission opened its earlier case against X in December 2023 and fined it two years later), and the platforms argue that the users who type the prompts are the ones committing the abuse. Both points are true. Neither changes the design choice at the start.

Denmark already has a bill for this

Denmark saw this coming. The government’s bill giving people copyright over their own face and voice was expected to pass early this year. It lets anyone demand that a platform take down a realistic digital imitation of them made without consent, with exceptions for parody and satire, and it threatens platforms that fail to comply with severe fines. Culture minister Jakob Engel-Schmidt has argued that deepfakes nobody can get removed undermine democracy, and France and Ireland have shown interest in copying the approach.

The Grok episode is the strongest argument yet for it. The DSA deals with the platform’s systems. A Danish right over your own likeness would give the person in the image a direct claim.

What a business deploying generative AI should take from this

Most companies will never ship an image generator to millions of users. Many are putting generative AI in front of customers, though, in chatbots, product configurators, marketing tools and support, and the logic of the Grok case applies to them at a smaller scale.

Test for misuse before launch, and write down what you found. The DSA’s duty to assess risks in advance only covers the largest platforms, but from August 2026 the AI Act requires anyone who deploys a system that makes deepfakes to disclose that the content is artificially generated or manipulated. In any dispute, a record showing you tried to break your own system before customers could is what separates a company that took care from one that did not.

Check what your supplier’s model will do. If a vendor’s image or text model sits inside your product, you inherit its behaviour. Ask what it refuses, how that was tested, and what happens when someone finds a way around it.

Treat your people’s likeness as something to protect. The same tools that undressed strangers on X are used to fake executives’ voices and faces in payment fraud. Agree an internal rule that no transfer or sensitive decision is made on the strength of a voice or video call alone.

And think about where your brand appears. X is under investigation in several countries at once for how it handled this. Whether your company keeps advertising or posting there is a decision someone should make deliberately.