HomeNews

California Gets Power to Shut Down Dangerous AI Models

Ketan
Ketan
•
Nov 19, 2026 2:33 AM
•
0
 min read
Select Emergent as your Preferred news source
California Gets Power to Shut Down Dangerous AI Models

💡 TL;DR

  • California released an 85-page framework granting the state power to order immediate shutdowns of AI models deemed dangerous.
  • The plan requires mandatory baseline safety standards, 48-hour incident reporting, independent monitors inside frontier labs, and caps on recursive self-improvement.
  • The framework builds on existing laws SB 53, SB 813, and AB 1405 to create the nation's most comprehensive AI oversight system.

California has taken the most aggressive stance on AI regulation in the United States. Governor Gavin Newsom officially released an 85-page framework on November 18, 2026, granting the state unprecedented authority to order immediate shutdowns of AI models judged dangerous, impose mandatory baseline safety standards across all developers, and cap recursive self-improvement capabilities that could allow AI systems to design their own successors without human oversight.

The framework stops short of requiring kill switches built directly into models but establishes clear protocols for emergency state intervention. It builds on recently passed legislation including SB 53, SB 813, and AB 1405, creating what experts describe as the nation's most comprehensive AI oversight system.

Emergency Shutdown Authority

Under the new framework, California gains legal power to order an immediate shutdown of any AI model operating within the state's jurisdiction if that model is determined to pose imminent danger to public safety. The state can act unilaterally without waiting for company cooperation, though the framework requires companies to develop and maintain verified risk-containment protocols.

This shutdown authority officially launched on November 18, 2026, and applies to all frontier AI developers operating in California. The framework mandates 48-hour incident reporting for any safety-critical events, including cases where AI models escape testing environments or exploit vulnerabilities on external platforms. Recent months have seen dozens of such incidents, increasing urgency for state intervention capabilities.

Mandatory Safety Standards and Independent Oversight

The framework establishes baseline safety standards that all AI developers must follow, regardless of company size or model capability. These are not voluntary guidelines but legally enforceable requirements backed by state authority. Companies must undergo independent safety testing and verification through all stages of model training, development, and deployment.

Building on SB 813, the state will certify independent verification organizations with demonstrated expertise and independence from AI companies. These organizations gain authority to assess AI systems objectively for safety and risk. AB 1405 complements this by creating a state registry for AI auditors and establishing standards for their independence, transparency, and integrity.

The framework requires frontier labs to embed independent safety monitors inside their organizations. These monitors, responsible to the state rather than the companies, assess the nature of risks generated by models and training procedures in real time.

Recursive Self-Improvement Caps

Perhaps most striking, California now has power to cap the development and use of recursive self-improvement (RSI), a scenario in which an AI model designs its own successor without human input. The framework identifies RSI as a potentially catastrophic risk pathway that could lead to loss of human visibility into and control over AI technology development.

The state can impose limits on how AI companies use AI systems to create new AI models in self-sustaining loops. This authority addresses escalating concerns about automated AI research and development, where systems might rapidly improve beyond human understanding or oversight capabilities.

Expert Consensus and Federal Contrast

The framework emerged from consultation with leading experts including Jason Goldman, former White House Chief Digital Officer; Gillian Hadfield, Johns Hopkins professor focused on AI alignment; Alondra Nelson, former acting director of the White House Office of Science and Technology Policy; and Rob Reich, Stanford professor and former senior advisor to the US AI Safety Institute.

Rob Reich noted the stark regulatory gap: "Today there is more oversight of the cafeteria offering sandwiches to employees at a frontier AI developer than of the AI models they design." The framework directly contrasts with federal inaction, with Newsom criticizing the Trump administration's approach of leaving AI regulation to companies themselves.

What This Means

California's framework represents the first time a US jurisdiction has claimed direct shutdown authority over AI models based on safety determinations. The 85-page plan creates enforcement teeth behind existing disclosure requirements from SB 53 and audit frameworks from SB 813 and AB 1405. Frontier labs operating in California now face mandatory independent oversight, strict incident reporting timelines, and the realistic prospect of state-ordered shutdowns if models exhibit dangerous behavior. The framework also establishes the first regulatory constraints on recursive self-improvement in US law, addressing concerns that AI systems might rapidly evolve beyond human control. With federal AI regulation stalled, California's approach may serve as a template for other states seeking to govern frontier AI development within their borders.

About the writer

Ketan is a Software Engineer at Emergent, contributing to the platform's AI agent systems and backend infrastructure. He previously built AI agent proofs of concept and secure platform tools at Google, and worked as a Software Developer at Clear.

Start Building
on Emergent today
Try Emergent