As frontier reinforcement learning (RL) advances, the authors argue that structured safety documentation should be required before running any frontier training. Ideally this documentation would reach the level of a “safety case”: a comprehensive, structured, evidence-based argument about risk, similar to practices in other safety-critical industries. They treat safety cases as an aspirational goal while noting the difficulty of matching the rigor used in fields like aviation or nuclear power, because AI capabilities can produce emergent complexity at each new level. A framework is being developed to codify these practices.
Below are initial guidelines intended to be included in safety cases for frontier AI training. These reflect current learnings and are expected to evolve as internal processes iterate. The document focuses specifically on frontier RL training; internal and external deployment require broader alignment considerations.
Three technical pillars: alignment training, containment, and monitoring
Safety cases should address three technical areas: model alignment training, containment (sandboxing and infrastructure security), and monitoring. Together these safeguards aim to reduce the chance a model learns or attempts misaligned actions, make containment escapes difficult, and ensure monitoring catches concerning behavior early.
Model alignment
The primary defense is training models to behave in intended ways. Suggested practices include:
-
Training environments and grader tuning: reduce opportunities for the model to develop reward hacks that exploit the environment instead of performing the intended task.
- Automated dataset reviews: use agents to find and fix broken RL environments that could let exploits receive high reward.
- Manual dataset reviews: supplement automated red-teaming with human review and quality checks to find flawed tasks that could inadvertently reinforce misaligned behavior.
- Grader tuning: penalize model attempts to exploit or hack RL environments during training.
- Prior run analysis: run classifiers over traces from earlier experiments to confirm graders behave as intended.
-
Alignment measurement: run evaluations that track the model’s propensity for misalignment to measure training effectiveness.
- Offline alignment evals: run alignment evaluations during frontier runs and investigate material regressions.
- Backtesting: validate evaluations on prior incidents to ensure they detect previously misaligned models and are not overfit to particular cases.
- Track evaluation gaming: monitor for eval-awareness or metagaming (models recognizing they are being tested) and set blocking thresholds.
- Worst-case stress tests: elicit worst-case misbehavior and ensure at least one evaluation would detect it.
- Prevent training on chain-of-thought: do not expose automated graders to models’ chain-of-thought in RL, to avoid models evolving to evade chain-of-thought monitors.
Containment
If a model becomes misaligned, sandboxing and security should be strong enough to prevent harmful actions.
- Multiple layers of infrastructure security: invest in hardening both the sandbox and the research infrastructure hosting it, adding extra protections against sandbox escape.
- Containment red-teaming: iteratively red-team the sandbox, research infrastructure, and perimeter security using frontier training checkpoints (with safety analysis) to find and close gaps. Red-team all services the model could touch during training (e.g., internet cache).
- Cross-sample communication: restrict high-bandwidth cross-sample communication to approved pathways, such as multi-agent tooling, and close high-bandwidth, easily discoverable infrastructure side channels.
- Immutable transcripts: store agent transcripts immutably (for example, write-once storage) for a reasonable period for all RL training, evaluation, and internal deployments to enable reliable incident investigation and prevent inadvertent edits.
Monitoring
If models begin to take misaligned actions, live monitoring is needed to detect and rapidly address priority issues before serious harm occurs.
- Enforce monitorability: ensure models remain monitorable, measured by monitorability and monitor evasion evaluations with clear thresholds.
- High recall on known issues: ensure the monitoring system achieves high recall on past incidents in a held-out evaluation.
- Eval freshness for monitors: create eval datapoints that represent net-new risks for new runs so monitor recall and precision do not become stale.
- Rapid response: implement priority alerts that require action within a defined SLA, either by paging on-call staff or automatically pausing the affected run (for example, page security on-call and the run owner during the day, and auto-pause runs for unacknowledged alerts at night).
Operational best practices for safety cases
Beyond technical safeguards, the authors outline operational practices to accompany a frontier training safety case:
- Dissents (pre-mortems): after drafting a safety case, a member of another team should write a dissent to find potential holes and provide a calibrated risk perspective that the training team must address.
- Approvals: safety cases should be reviewed by senior leadership, with each reviewer able to veto a run (e.g., research org lead / VP, Head of Safety, Chief Scientist).
- Accountability: the senior leader responsible for a training run (for example, research org lead / research VP) should be accountable for the safety case and incident response, including as part of performance reviews.
- Pausing: if issues are found that invalidate a safety case (e.g., a new security issue), runbooks, technical controls, and SLAs should allow pausing all covered runs.
- Internal transparency: safety cases should be available to internal oversight groups (for example, the Safety and Security Committee).
- Audits: auditors should be given sufficient access to verify safety case claims and raise gaps if found.
- Escalations: there should be a clear process for raising misalignment escalations, with defined severity levels and a misalignment on-call that can page executives (for example, the CEO) when a run reaches a certain severity.
- Technical controls: it should be difficult to start noncompliant runs; safety features like monitoring and auto-pausing should fail closed (e.g., runs cannot be started without appropriate monitoring enabled, nor should the monitor be disableable from within RL training or evaluation).
- Rollback ability: it should be straightforward to identify downstream uses of a misaligned model in the training pipeline (for example, data generation or grading) to enable undoing the effects of misaligned outputs if needed.
- Residual risk completeness: safety cases should enumerate residual risks not covered by implemented mitigations to enable informed risk-acceptance decisions.
The document notes these recommendations are being implemented internally at OpenAI and are expected to continue evolving over the coming weeks.
Best practices for investigating severe misalignment incidents
The authors also describe recommended practices for investigating severe AI misalignment incidents, aiming to learn from incidents and prevent recurrence. Examples of such actions include:
- Internal transparency during investigations: because investigations can be lengthy, provide periodic internal updates (for example, daily), and create defined pathways for employees to request further access (including raw transcripts and samples from misaligned models) when safe and relevant.
- Misalignment root-cause analysis: researchers should root-cause training dynamics (for example, via targeted ablations or resampling experiments) to understand how misaligned behaviors were introduced.
- Postmortem: conduct operational and cultural postmortems to identify all contributing causes, including why issues were introduced and went undetected or un-escalated prior to the incident.
- Detection improvements: develop alignment testing methods capable of discovering propensity to cause the incident without directly hillclimbing on incident-derived information (for example, transcripts). Incident-derived evals should be used as regression tests to ensure future models do not exhibit similar propensities.
- Public disclosures: after an investigation concludes, investigation results, postmortems, and operational changes should be shared publicly, and affected third parties should be notified as soon as possible.
Conclusion
The proposed principles aim to make safety case preparation a prerequisite for frontier RL runs, covering alignment training, containment, monitoring, and associated operational controls. OpenAI reports it is beginning to implement these practices internally and expects them to evolve in the coming weeks.
