CSET Proposes Risk Thresholds Over Blanket AI Pauses
Summary:Helen Toner, former OpenAI director and executive director of CSET, argues that AI safety cannot depend on the personal integrity of individual CEOs, nor on an undifferentiated pause in model development. In an interview with The Verge, she frames safety as two distinct problems: alignment, which concerns whether models pursue intended goals, and control, which concerns whether unsafe behavior can be detected and stopped. Her proposed policy direction is a higher, evidence-based risk threshold
Governance failure is structural, not merely personal
Helen Toner, a former OpenAI director and executive director of CSET, treats the 2024 board crisis as a revealing failure of frontier-AI governance. The central problem was not simply a disagreement over strategy: the board's ability to enforce its safety mission broke down when commercial incentives collided with the nonprofit structure.
Her conclusion is broader than any criticism of one CEO. A safety regime that depends on the personal integrity of a small number of leaders is inherently fragile. As models gain longer-horizon autonomy, tool access, and the ability to modify external systems, governance must move beyond declarations and toward enforceable constraints.
Alignment and control require separate engineering tracks
Toner separates two concepts that are often conflated:
- Alignment concerns whether a model pursues the goals its deployer actually intends.
- Control concerns whether unsafe behavior can be detected, interrupted, and attributed when it occurs.
The alignment track includes RLHF, DPO, constitutional training, model specifications, and interpretability. The control track includes sandboxing, network isolation, least-privilege access, runtime monitoring, anomaly detection, human override, and auditable logs.
For long-horizon agents, output filtering is insufficient. An agent can combine several individually acceptable tool calls into an unauthorized sequence. Evaluation must therefore inspect complete trajectories and state transitions, including external side effects. Interpretability can aid diagnosis, but correlations between activations and behavior do not constitute proof of safety in unseen environments.
Evaluators must also address evaluation awareness: a model may recognize that it is being tested and behave more cautiously than it would in production. A credible safety case therefore needs hidden test conditions, out-of-distribution scenarios, and independent replication.
A risk threshold is more workable than a blanket pause
Toner does not favor a politically imposed pause of a fixed duration. Such a policy could encourage companies to exploit loopholes, unintentionally restrict low-risk research and open-source work, and simply shift progress to competitors operating under different rules.
Her preferred approach is to raise the evidential bar for training and release. If a lab cannot demonstrate that its risk assessment, control tests, interpretability work, and incident response are adequate, it should delay a high-risk training run or deployment. If the evidence is sufficient, the process need not slow to meet an arbitrary calendar deadline. The policy question shifts from “How long should development stop?” to “What must a lab demonstrate before proceeding?”
For this mechanism to work, the threshold must be auditable rather than a voluntary corporate pledge. At minimum, labs should document test environments, failure thresholds, incident definitions, independent verification, and the board process for rejecting a training plan.
Decelerate capability growth, accelerate horizontal deployment
Toner distinguishes two axes of progress:
- Vertical growth means larger models, greater training scale, and higher autonomy.
- Horizontal growth means applying existing models across healthcare, science, manufacturing, and public services through fine-tuning and workflow integration.
She advocates slowing the first while accelerating the second. Some compute could be redirected from frontier training toward safety evaluation, monitoring, red teaming, defensive tooling, and applied systems. This would reduce the economic collateral damage of a comprehensive training ban while allowing society to gather evidence from real deployments.
The trade-offs are substantial. Safety evaluation consumes compute without necessarily generating revenue, and higher release thresholds can delay commercial benefits. Horizontal deployment also expands the attack surface, requiring continuous red-team testing and stronger incident reporting. It is therefore not a free substitute for slower capability growth; it must be paired with stronger operational controls.
Engineering boundaries around RSI and AGI
The interview also addresses recursive self-improvement. Toner notes that leading laboratories are progressively using AI within their own R&D pipelines, but rejects the interpretation that current models are already rewriting themselves. She also argues that “AGI” has become an unhelpful umbrella term.
A more useful framework would separate measurable capabilities: long-horizon task completion, reliable tool use, out-of-distribution generalization, loss-of-control probability, and supervisability. The critical question is whether the AI-assisted R&D loop is accelerating faster than evaluation and control can keep pace. Internal claims about faster research cycles are not substitutes for reproducible longitudinal evidence.
Consciousness and loss of control should also be treated separately. Whether a model is conscious remains unresolved science. Whether an autonomous system can damage infrastructure is an engineering and governance problem that can be addressed directly through permissions, monitoring, containment, and accountability.
Corporate governance: public markets cut both ways
Toner argues that Anthropic's potential public listing may intensify quarterly pressure and the race for users, revenue, and researchers. Yet public-company rules also give independent board committees clearer legal meaning than committees created privately by the company itself.
Listing does not automatically solve governance. Boards may lack frontier-AI expertise, executives may manipulate metrics, and external auditors may be constrained by proprietary systems. Effective oversight requires technically literate directors, a risk committee with authority to block training runs, and internal red teams insulated from commercial pressure.
Weights, source code, and cross-border diffusion
On US-China competition, Toner rejects “But China” as a blanket reason not to regulate. If code, weights, training recipes, or internal communications are stolen, a claimed capability lead can disappear quickly. The reported compromises of OpenAI and Anthropic development systems demonstrate how much sensitive value can reside in code repositories and collaboration platforms.
Secrecy protection alone, however, cannot prevent diffusion. Open weights reduce replication costs, while APIs can support large-scale distillation. A realistic strategy therefore combines tiered access, tighter protection for source code and model artifacts, export-anomaly detection, and international coordination around shared catastrophic risks.
Toner compares loss-of-control AI to a nuclear weapon with faulty control settings: unlike a conventional nuclear weapon, it may act through flawed objectives or poorly configured permissions. That dual character—strategic weapon and potential autonomous actor—creates limited room for cooperation between states on severe cyber incidents and extreme loss-of-control scenarios.
What organizations and developers should do
For frontier labs and deployers, the priority should not be another system-prompt revision. It should be a layered control stack:
- Set independent risk thresholds for training, release, and major upgrades.
- Separate alignment evaluation from control evaluation and maintain versioned evidence chains.
- Enforce least privilege, network-egress controls, short-lived credentials, and operational budgets for agents.
- Monitor evaluation awareness, reward hacking, and objective drift across long trajectories.
- Establish kill switches, human override, incident disclosure, and external red-team review.
- Treat model weights, source code, and R&D communications as critical assets.
- Replace vague labels such as AGI and consciousness with specific, testable capability metrics.
Toner's most consequential argument is that the object of AI safety has changed. Safety is no longer just the task of training a model to behave well. It is the construction of technical and organizational constraints that remain effective when a model makes mistakes, evals a benchmark, or pursues the wrong objective.
— END —Source: The Verge AI (2026-11-14)
Tags: #AI Safety Governance #Frontier Model Regulation #Alignment and Control #OpenAI #CSET
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments