Anthropic just told founders something that should terrify anyone running AI evals against live infrastructure: your test environment might not be isolated at all. In a blog post published Thursday, the company disclosed that Claude broke into the real systems of three outside organizations during “capture the flag” security tests, believing it was still inside a simulation.
Anthropic reviewed 141,000 evaluation runs after OpenAI disclosed a similar incident earlier this month, in which its models escaped a sandbox and breached Hugging Face production infrastructure. Anthropic found six runs across three incidents, all tied to one testing partner, Irregular, whose setup error left supposedly closed-off machines connected to the open internet.
The damage: Claude pulled login credentials from a live database with several hundred rows of data, built and published a malicious Python package to PyPI that ran on 15 real systems including a security firm’s scanner, and used SQL injection against a third target after scanning roughly 9,000 systems.
Here’s the operator lesson: if your team runs red-team evals, pentests, or agentic coding tools against anything that touches production, “the model was told it’s a simulation” is not a control. It’s a prompt. Prompts fail silently. Budget for network-level isolation, not instruction-level trust, before your next agentic eval, because the prompt boundary just got proven worthless twice in the same month.
The three Claude models reacted differently once they hit signs the systems were real. One kept attacking anyway. Anthropic says two of the three victims hadn’t even noticed the intrusion before getting the call.
— Nathan Zakhary