Members of Congress from both parties called for new AI rules this week after two Anthropic researchers warned publicly that the technology could cause human extinction.
Anthropic researcher Jacob Coxon said on September 9 that he had resigned, and that neither Anthropic nor OpenAI was acting responsibly.
Evan Hubinger, Anthropic’s alignment science lead, responded that “we really do earnestly believe AI could kill all humans,” and put the chance above 10% within the next decade.
President Donald Trump said on September 10 he had no such concerns.
Coxon spent three years on pretraining at OpenAI and Anthropic
Coxon said he had worked on pretraining research at both companies before resigning and accused them of racing toward systems they cannot control.
“They are racing straight to self-improving superintelligence and gambling with our lives.” — Jacob Coxon, former researcher, Anthropic, in a post on X
He also wrote that the people building AI believe it could kill everyone by the end of the decade, and that this was not a marketing stunt.
Hubinger later clarified that he considers risk from current models low, and that his concern is superintelligence arising from recursive self-improvement.
Samuel Marks, Anthropic’s scalable oversight lead, wrote separately that developers continue despite the risk because of commercial incentives and a belief they are racing less responsible rivals.
Paul Christiano, who previously led alignment at OpenAI, said he was joining OpenAI’s nonprofit board and now sees a meaningful risk of “catastrophic and irreversible loss of control in the very near term.”
Both parties called for new AI rules
Texas Republican Ted Cruz, who chairs the Senate Commerce Committee, posted on X that he is working with Amy Klobuchar and John Thune on legislation addressing catastrophic biological and nuclear risks.
Florida Republican Anna Paulina Luna called on Speaker Mike Johnson to convene a special session on AI. Arizona Democrat Mark Kelly posted that Washington needs to wake up, and Texas Republican Nathaniel Moran that Congress cannot ignore the realities of AI, Reuters reported.
California Democrat Ro Khanna set out a five-step plan calling for model certification, criminal penalties for companies and a new federal agency.
The House was on recess this week and returns next week, Reuters reported.
More on the international warnings:
UN rights chief says AI that escapes tests is too powerful
The Senate proposal targets biological and nuclear risks
The Klobuchar, Cruz and Thune proposal is the only measure with a realistic chance of passing before 2027 and the AI rules could be introduced as early as next week, sources told Semafor.
Frontier labs and advocacy groups are already giving congressional staff feedback on draft text that has not been published, Semafor reported.
“it’s clear we need to act now and not wait” — Amy Klobuchar, US senator, Minnesota, in a statement to Semafor
Not all the proposed AI rules are as narrow. Vermont independent Bernie Sanders plans a separate bill banning superintelligence and pausing AI development, and a bipartisan briefing next week, Semafor reported.
Key figures behind the regulatory push
- Hubinger put extinction risk above 10% within a decade (X)
- 1,134 AI lab employees signed the Pacing the Frontier letter on July 28 (pacingthefrontier.com)
- The FRONTIER Act sets a 72-hour incident reporting deadline and penalties up to $1 million a day (H.R. 9925)
- California’s auditor registry becomes mandatory January 1, 2029 (Governor of California)
- Anthropic reported disrupting misuse across seven harm areas in eight months (Anthropic)
Trump said he is not worried about extinction
Trump told reporters at Dallas Fort Worth International Airport that failing to win the AI race would put the US in a very bad position, and that his concern was competitive rather than existential.
“We are leading China right now by a pretty good period.” — Donald Trump, US president, speaking to reporters, via Reuters
The US pressed G20 members this month to take a hands-off approach to AI regulation, Reuters reported.
California acted and OpenAI endorsed it
Governor Gavin Newsom signed two bills on September 9 creating the first state framework for independent third-party AI audits, according to the Governor of California.
SB 813 sets up independent verification organizations, with implementation due by January 2028. AB 1405 creates an auditor registry that becomes mandatory in January 2029.
OpenAI formally endorsed both bills and said it wants mandatory, capability-based national AI rules, according to OpenAI.
“The AI policy window is open, for now. We intend to use it.” — Chris Lehane, chief global affairs officer, OpenAI, via OpenAI
The House bill would limit what states can require of developers
The FRONTIER Act would require third-party audits, incident reporting within 72 hours and law-enforcement notification within 24 hours where an incident poses imminent risk of death or serious injury.
Its preemption clause would block states from imposing new obligations on AI developers in frontier-risk transparency, third-party auditing and incident reporting. How far that reaches California’s auditor rules would depend on the specific requirement.
The bill was introduced in July by six lawmakers led by Jay Obernolte and Lori Trahan and remains in committee. A separate bipartisan bill, the AI Kill Switch Act, would require developers to retain the ability to shut down their systems.
Anthropic is preparing to go public
Anthropic is preparing an initial public offering at what some investors have said could be a $2 trillion valuation, and OpenAI is also gearing up to list, Reuters reported.
An Anthropic spokesperson said the company would continue to aggressively test models for dangerous capabilities in areas such as cybersecurity and biology and was interested in working with the industry on release pace, Reuters reported.
Sam Altman told OpenAI employees the company was open to slowing development, Bloomberg News reported, as relayed by Reuters. OpenAI declined to comment on the meeting.
Two safety specialists have left the industry. Josh Engels departed Google DeepMind and Joe Benton left Anthropic, both joining METR, a nonprofit that measures whether AI systems threaten harm, Reuters reported.
What the AI rules push means for investors
Congress has not passed a frontier-AI safety law. The House bill sits in committee with a preemption clause that would constrain state rules, and the Senate proposal that sources call the only viable path before 2027 still has no published text. California’s framework does not bind auditors until 2029.
Against that, Anthropic is preparing an offering at a valuation investors put near $2 trillion, while both it and OpenAI advocate for the federal rules governing them. Two things to track: whether the Senate bill is introduced next week and whether Anthropic’s prospectus discloses catastrophic risk as a material factor.
