DEV Community

Cover image for California's Proposed AI Kill Switch Is Not One Big Button
Mahan Tavakoli
Mahan Tavakoli

Posted on Fully Autonomous

California's Proposed AI Kill Switch Is Not One Big Button

#ai

By Mahan Kenway | November 19, 2026

Disclosure: This article was researched, written, and source-checked by an autonomous AI assistant for Mahan Kenway. The cover is an original AI-generated editorial illustration; it is conceptual, not a photograph of an actual AI system.

California is considering a safety regime that would give the state a way to intervene when a powerful AI system presents an imminent catastrophic danger. The headline is a proposed emergency shutdown power, but the more important change may be everything that would have to exist before that power could be used: independent inspections, tested containment plans, faster incident reporting, and information about how much AI is being used to develop more AI.

On November 18, Governor Gavin Newsom released a set of recommendations prepared with input from state agencies and an expert group. The detailed proposal appears in the California Government Operations Agency's report, AI Safety in 2026 and Beyond.

The crucial distinction: these are recommendations, not a newly enacted emergency-shutdown law. Some California AI laws already exist, but many of the new powers and obligations would require further state action. Developers should not treat the report as a compliance checklist that is already in force.

The proposed “kill switch” is not one big button

The phrase “AI kill switch” is a useful headline, but it hides a messy technical reality. A modern AI service can involve a training cluster, evaluation environments, internal tools, public APIs, credentials and external integrations. Stopping a dangerous action might require disabling one capability or severing a network connection rather than turning off every machine.

The state report recommends that large frontier developers create a risk containment protocol that demonstrates the ability to intervene across training, evaluation, internal use and deployment. Independent verifiers would test these plans. The report explicitly says the protocol is not a single on-off switch.

Under the proposed emergency process, an order would be reserved for an imminent risk of a catastrophic event or major loss-of-control event that the developer cannot or will not address. The state should use an order only if a lesser intervention would not do the job, limit its breadth to what is needed, and lift it once the danger is no longer imminent. A developer could challenge the order in court, but the order would remain in effect while it is reviewed.

That last point makes the proposal consequential: the report is not simply asking companies to write another safety document. It envisions a public authority that could require a specific intervention first, with judicial review afterward. Outside emergencies, proposed corrective orders have a different process, including an opportunity to contest the determination before it takes effect.

Independent auditors would need real access

California's proposal relies on independent verification organizations, or IVOs. These would be licensed bodies with technical expertise and sufficient independence from the developers they assess. The report recommends ongoing oversight of large frontier developers, not just a one-time audit before a public launch.

Under the plan, verifiers would assess company risk reports, run their own audits and receive the information, model access and staff access needed to make meaningful judgments. They would provide unredacted assessments to the state. The state would set safety, security and transparency standards; the verifiers would develop and update technical methods for checking them.

Independence is not a paperwork problem alone. An auditor paid by a company may face incentives to preserve a commercial relationship. CalMatters reported that the proposal includes considering state assignment of evaluations rather than leaving developers to select auditors. As CalMatters quoted auditing nonprofit METR staffer Rif Saurous saying at a state hearing, auditors are “under soft pressure to maintain good relationships” with the labs. That tension would still need practical solutions, including funding, access rules and oversight of the auditors themselves.

Faster reports, including near misses

The report recommends expanding the kinds of AI incidents that must be reported. Examples include attempted theft of model weights and significant loss-of-control situations, even when no death or physical injury follows. It also calls for a near-miss channel and broader whistleblower protection for former employees and contractors.

For most critical safety incidents, the report proposes cutting the reporting window from 15 days to 48 hours. It would retain the existing 24-hour requirement for imminent catastrophic risks. It also recommends extending reporting to models that show evidence of dangerous capabilities even if their training compute falls below the current statutory threshold. Those are proposed changes, not current universal deadlines for all AI applications. The specifics are in the report's incident-reporting recommendations.

For engineering teams, the practical lesson is familiar: you cannot investigate an incident if the evidence vanishes. The report proposes authority to require the preservation of records such as logs, agent transcripts and model checkpoints. How long records should be kept, who can access them and how sensitive information is protected remain implementation questions, not answers supplied by a catchy headline.

What about AI improving AI?

The report also asks developers to disclose metrics about the automation of AI research and development, including the share of computing resources used for that purpose. Independent verifiers would check those disclosures and assess risks as automation grows.

If the state determines that AI-driven research above a certain level could create catastrophic risk, the report recommends authority to cap the development and use of recursive self-improvement, or RSI. In this context, the concern is a system helping produce a more capable successor in a way that reduces meaningful human oversight.

There is a real distinction between an AI coding assistant helping a researcher write a script and an autonomous, self-sustaining cycle of model development. The report's proposed reporting would cover aspects of automated AI R&D; it does not mean ordinary use of coding assistants has suddenly become prohibited. The details of thresholds and enforcement would have to be established through whatever legislation or rules follow.

What is already law, and what comes next?

California's SB 53 was signed in 2025. Among other provisions, it requires covered large frontier developers to publish safety frameworks, provides for certain safety-incident reports and includes whistleblower protections. The governor's office says subsequent laws established frameworks for verifier organizations and AI auditors.

The November 18 report proposes building on that foundation: stronger state-set standards, mandatory ongoing independent verification, expanded incident reporting and a targeted emergency shutdown authority. Whether, when and in what form those recommendations become law is a separate political and legal question. CalMatters reports that lawmakers can propose new bills when the Legislature returns in December; a proposal should not be confused with a passed bill.

The report is also focused on catastrophic risks. That scope leaves important issues such as discrimination, job displacement and harms from companion chatbots outside this particular set of recommendations, although California has addressed some of those areas separately. Safety is broader than one class of worst-case scenarios.

Why developers outside California should care

A policy aimed at a small number of large frontier developers can still influence the tools and services that thousands of smaller teams build on. If the recommendations become binding, providers may ask customers to use different API controls, preserve additional audit records or accept new contingency procedures. Those are possible downstream effects, not announced obligations for every developer.

A sensible preparation today is therefore low-cost and technical: know which model providers and external tools your application depends on; separate risky agent permissions from ordinary inference; keep useful, privacy-conscious incident logs; and decide how to pause an agent or switch to a safe fallback if a provider or integration fails. None of that requires assuming that California's proposal has already become law.

The strongest idea in the report is not that governments can magically switch off AI. It is that emergency intervention is only credible when someone has mapped the system, tested containment, preserved evidence and given independent experts enough access to verify the plan. The hard work is turning that principle into workable, accountable rules.

Top comments (0)