Safety

AI-generated text

OpenAI Chief Scientist Proposes Shared Metric for AI Safety Limits

OpenAI Chief Scientist Jakub Pachocki proposes using perplexity, a measure of predictive performance, to compare AI development across laboratories and set shared safety limits.

OpenAI Chief Scientist Proposes Shared Metric for AI Safety Limits

Jakub Pachocki, Chief Scientist at OpenAI, proposes that AI developers use perplexity, a measure of predictive performance, as part of a shared approach to safety standards and development limits. As a first step, OpenAI has released some of its internal pretraining measurements. Pachocki says the disclosure gives competitors a clearer view of the company’s position, but that the urgency of industry-wide safety measures outweighs that concern.

In the researcher’s account, modern models derive much of their capability from a process that is simple to describe but difficult to interpret: a system is trained on a large dataset to predict, or compress, data. Developers then use methods such as prompting, fine-tuning and reinforcement learning to elicit the model’s latent capabilities in different settings. Pachocki says algorithmic progress can blur the boundaries between training stages, but that the creation of general capabilities can still be distinguished from their targeted elicitation.

What the measurements show

Perplexity measures how well a model predicts the next token on average: a lower value indicates better predictions. In practice, OpenAI generally tracks the logarithm of perplexity, also known as cross-entropy loss. The newly disclosed figures, starting with GPT-3.5, are measured on a fixed held-out evaluation dataset drawn from the company’s internal codebase.

Pachocki places the development of several models alongside this metric. GPT-2 set a new record on older datasets and, in his view, was among the first models able to consistently write coherent paragraphs. GPT-3 could learn a task in context from a few examples. GPT-3.5 achieved a log-perplexity of about 0.99 on the current main evaluation dataset, could hold fluent conversations and formed the basis of the original ChatGPT.

GPT-4, at roughly 0.91, productively used step-by-step, or chain-of-thought, reasoning; that later helped motivate the creation of modern reasoning models. At losses below 0.8, OpenAI observed significant new capabilities in areas including cybersecurity and mathematics. Pachocki stresses that it was not possible to know in advance that these capabilities would emerge at those particular thresholds.

The researcher argues that task-based tests are difficult to compare across old and new models: many current benchmarks would have made little sense for GPT-2 or GPT-3, while earlier tests were largely saturated by the time of GPT-4. By contrast, he considers measurements of pretraining progress likely to remain informative for longer.

Acceleration and proposed limits

Pachocki says AI’s contribution to its own development could produce a step change particularly in algorithmic research on pretraining. He nevertheless considers fully automated recursive self-improvement without human oversight something to avoid in order to preserve human control.

For more than a year, OpenAI has posed unsolved mathematical problems to its models as part of its evaluations. According to Pachocki, a model with a log-perplexity of around 0.76 began producing groundbreaking results. He believes a similar shift is conceivable in deep learning research, with a series of fundamental pretraining advances possible as soon as 2027. He also expects AI to contribute to alignment and monitoring, but considers it unlikely that progress in those areas will be fast enough to keep pace with the full potential acceleration in pretraining.

Extrapolating from recent trends, Pachocki expects further significant capabilities to emerge in the log-perplexity range of roughly 0.68–0.70, although he does not know what they will be and current tests may not measure them. At the same time, he expects dominant forms of alignment auditing to deteriorate further, while models become more aware that they are being evaluated or trained.

He therefore proposes that frontier model developers establish shared norms for evaluation, adversarial testing of safety cases and transparency before such models are first put to use. He also calls for common development limits based on perplexity and other measurements. Companies would not cross those limits until the ability to understand and monitor model behavior reached a concrete milestone, such as developing a successor to systems that monitor chain-of-thought reasoning. The aim is a framework that could later set successive limits and govern the pace of development.

Limits of the metric

Pachocki acknowledges that perplexity is difficult to compare across laboratories because they use different evaluation and training datasets. OpenAI reported figures from its internal codebase because those are the measurements it currently relies on most to assess pretraining progress. However, given their training data, code prediction is a poor proxy for the general capabilities of GPT-2 and GPT-3; he says evaluating old and new models together on general text could still be informative.

Perplexity also does not directly measure performance on a practical task, and it does not capture every advance outside pretraining. Chain-of-thought reasoning, for example, can substantially increase capabilities without reducing a model’s perplexity. The metric’s lower bound is unknown as well: zero loss would mean predicting the data entirely deterministically, which is not expected to be possible. As models approach the inherent entropy of the data, measured progress may slow even as smaller improvements correspond to larger leaps in capability.

In this context, the researcher says, a “compressor” is not a finished compressed artifact such as a GZIP file, but a system capable of compression by predicting previously unseen data. That does not necessarily mean it remembers the contents of its training corpus.

OpenAI attributes the steep acceleration shown in its chart for 2026 mainly to long-term research and computing investments, and to automation to a lesser extent. For its most recent models, however, the company says AI has already made meaningful contributions to their design. Despite the metric’s limitations, Pachocki believes perplexity could provide a useful common language for tracking AI progress and calls for greater transparency across the industry.