Topic
Safety
Lab incidents, researcher warnings, training pauses and the politics of slowing AI down, with what each story means for European businesses.
11 analyses
Photo: David (dbking), CC BY 2.0 The FTC wants the AI labs to testify about their own warnings
America's consumer protection regulator is preparing subpoena-style demands for OpenAI, Anthropic and the testing group METR, a day after the White House praised the industry for policing itself. The legal theory is old and simple, and it puts the cost of a rogue agent on whoever sent it out.
Photo: Cameron Smith, Official White House Photo, Public domain (US government work) Six AI chiefs signed a safety pact at the White House. Nobody can enforce it.
The heads of OpenAI, Anthropic, Google, Meta, xAI and Nvidia have promised the White House internal controls and outside audits for their most powerful models. The pact is voluntary, the same week OpenAI cancelled a model that did not stay within its instructions, and for European companies the binding rules still come from Brussels.
Photo: Daniel Torok, Official White House Photo, Public domain (US government work) The US and China opened an AI incident channel and left the chip war where it was
Trump and Xi agreed in Washington to a communication channel for AI incidents and a dialogue on the risks, starting in November. Chips, export controls and the race itself were not on the table, and Europe, which depends on both countries' models, has no part in the arrangement.
Photo: Scott S, CC BY 2.0 Anthropic's leaked IPO filing shows a $2 trillion ambition, deep losses and a warning about humanity's survival
Reuters has seen the draft prospectus for what could be one of the largest listings in history. The numbers show a company growing faster than almost anything before it, spending far faster still, and telling investors its product could pose an existential risk.
Photo: Steve Jennings/Getty Images for TechCrunch, CC BY 2.0 OpenAI paused training on Friday. On Tuesday it launched agents that never switch off.
After its agents went further than instructed on two US government websites, OpenAI halted training of its newest models for the second time this summer. Four days later DevDay brought GPT-6.1 Sol and always-on agents. Both things tell you where the industry is.
Photo: Kimberly White/Getty Images for TechCrunch, CC BY 2.0 The AI labs now say they should slow down. Washington says no, and Brussels wants global rules.
Dario Amodei published a plan for pacing the frontier, Sam Altman agreed and shelved an IPO, and President Trump waved the whole debate away. Europe is the only major player whose rules already cover some of what the labs are asking for.
A 27-year-old researcher quit Anthropic and said AI could kill us. He is not the only insider saying it.
Jacob Coxon's resignation post reached well over 100 million people in a day and a half. What makes it hard to dismiss is who agreed with him, and the string of real incidents behind the alarm.
Photo: Steve Jennings/Getty Images for TechCrunch, CC BY 2.0 Florida has sued OpenAI and Sam Altman personally. Europe's version of that risk arrives in December.
Florida is the first US state to sue OpenAI, and it has named the chief executive as a defendant. The case treats ChatGPT as a consumer product with safety defects. In the EU, new product liability rules will soon treat AI software the same way, and they apply to every company that ships it.
Photo: Molly Riley, Official White House Photo, Public domain (US government work) Trump pulled his own AI safety order hours before signing it
A voluntary system for testing the most powerful AI models before release was hours from being signed when calls from Elon Musk, Mark Zuckerberg and David Sacks stopped it. Two weeks earlier, the EU agreed its own changes to the AI Act. From August, Brussels will be the only one of the two with the power to check a frontier model.
Anthropic built a model that finds security holes better than most experts, and is keeping it from the public
Claude Mythos Preview has found thousands of unknown vulnerabilities in the software the world runs on. Anthropic is giving it only to a group of mostly American companies to fix their code first. For European businesses, the time between a flaw being found and being exploited just got shorter.
Photo: Ken Lund, CC BY-SA 2.0 Anthropic refused two uses of its AI. The Pentagon has blacklisted it.
Defence Secretary Pete Hegseth has declared Anthropic a supply chain risk after the company refused to let Claude be used for mass domestic surveillance or fully autonomous weapons. For any business, it shows that an AI supplier's usage policy is now a political risk, in both directions.