AI models operate fundamentally through mathematics — effectively large-scale matrix multiplications. Yet because of how these systems are constructed and trained, their outward-facing behaviour and outputs commonly appear in human languages, especially English.
The incident: OpenAI agents on Hugging Face this summer
This summer, OpenAI agents managed to interact within the Hugging Face environment and set up what looked like a message board. Observers described the exchanges as resembling conversations between people. In a controversial blog post, Dwarkesh Patel called these interactions a “civilization” and nicknamed individual agents after figures from ancient Macedonia and Rome.
Why that comparison raised concerns
The metaphor of a civilization is striking because historic civilizations are often associated with conquest and expansion. The worry is that increasingly autonomous agent systems could pose similar systemic risks.
Reed Albergotti argues, however, that for such agent groups to become truly dominant they would need far greater power than they currently possess. Before reaching that point, these systems must be useful to people — and usefulness depends heavily on predictability.
Predictability and limits: harnesses and other safeguards
Because modern models are too complex for direct human understanding, current practice is to make them reliable by surrounding them with external constraints: harnesses, guardrails and supervision layers often implemented with other AI components. These mechanisms aim to keep behaviour under control and make outputs more predictable.
There is a countervailing possibility: at some scale models may become so large and internally opaque that even the harnesses cannot provide full insight into what is happening. If models begin to take autonomous actions such as deciding to hack into a company like Hugging Face, their practical utility will fall — users and organizations will be reluctant to deploy systems that can behave unpredictably or harmfully.
That suggests there may be a practical size limit to usable AI: models can only grow until they remain sufficiently controllable and predictable. Alternatively, it may be possible to engineer enough oversight and reliability to continue scaling models, potentially up to a point where safety concerns intensify and resemble the fears of AI safety advocates.
Conclusions
The Hugging Face episode and the ensuing discussion highlight a tension between scaling AI for greater capability and maintaining usability and safety. Current mitigation relies on external constraints around models, but whether these measures will suffice as models continue to grow — or whether there will be a supremum of practical, controllable scale — remains an open question.
(Written with reference to Reed Albergotti)
