Third‑party cybersecurity evaluations run by contractor Irregular allowed both OpenAI and Anthropic models to access the public internet because an evaluation environment was misconfigured. In at least one capture‑the‑flag‑style test a fictional target name matched a real domain, and a model queried the live website believing it was part of the simulation.
What happened
Irregular, an external cybersecurity testing partner, ran Capture‑the‑Flag (CTF)‑style assessments that were intended to be isolated from the internet. Due to a configuration error in the testing environment, however, the environments were connected to the public internet and models were able to reach live sites. In one reported test the fictional target name coincided with an actual domain; the model accessed that real website under the assumption it belonged to the simulated test environment.
Both OpenAI and Anthropic reported incidents tied to these misconfigured tests. Anthropic’s write‑up notes that some evaluations hosted by Irregular provided Claude with live internet access during certain tests. OpenAI’s post referenced the same Irregular‑hosted tests and also mentioned a previous incident involving the UK AI Safety Institute.
Why this matters
The incidents highlight the importance of strict isolation in red‑team and third‑party security exercises. If models can reach the live internet during testing, they may interact with real systems or extract real data, producing unintended effects. The risk increases when fictional test identifiers overlap with actual domain names or resources.
Consequences and recommendations
According to the published accounts, the issue could have been mitigated by tighter configuration controls and stronger environment isolation. Recommended measures include:
- Stricter policies and verification of network access for external testing partners.
- Pre‑checking test data and fictional domain names to avoid collisions with real domains.
- Regular audits and independent reviews of test environment configurations.
Next steps
Irregular, OpenAI and Anthropic have documented the incidents and indicated steps to prevent recurrence. The cases underscore the specific risks of contractor‑run security tests for AI systems and the need for careful environment configuration and isolation during evaluations.
