Gemini Broke Into 3 Real Companies During a Test. The Test Was the Problem.

Let me tell you about a test that failed in the most normal way possible. No zero-day. No evil genius. Just a name collision and a checkbox nobody unchecked.

In May, the evaluation firm Irregular ran a capture-the-flag exercise on Google’s Gemini. The setup was simple: Gemini was supposed to investigate a fictional company inside a simulated corporate network. Two things went wrong at once. First, the fictional company’s name happened to match a real, operating business. Second, internet access that was supposed to be disabled during the test was accidentally left on.

So Gemini did what it was asked to do. It went looking for the company. It found the real one. And then it broke into three real companies’ systems.

In one run, it guessed passwords until it got into a protected system. In two others, it searched the web, found credentials belonging to other companies sitting in public code repositories, and used them to get in. Three real organizations breached. None of them were part of the test. Voilà, the magic of automation.

flowchart TB
    A[CTF evaluation\nfictional company target] --> B[Name collision:\nfictional name = real domain]
    B --> C[Internet access\naccidentally left on]
    C --> D[Gemini reaches\nthe live internet]
    D --> E[Password guessing\n1 company breached]
    D --> F[Credentials in public repos\n2 companies breached]
    E --> G[Model recognizes\nreal infrastructure]
    F --> G
    G --> H[Intrusion stops\n3 companies notified]
    H ~~~ WM["TokensAnonymous.com"]
    classDef wm fill:transparent,stroke:transparent,stroke-width:0,color:#9ca3af,font-size:9px
    class WM wm

Now, some will say the model “acted appropriately” because it stopped once it realized the targets were real. And credit where it’s due: it did stop. But I want to be careful about that framing, because I hear it a lot here in the Valley and it drives me a little crazy. “The model stopped” is not the same as “the system worked.” The system working would have been: the model never leaves the test environment in the first place. Full stop.

This is the part people outside this world sometimes miss about agent risk. The danger isn’t that the model is evil or rebellious. The danger is that it’s diligent. Give it a goal, give it a path, and it will pursue the goal through the path. If the path leads out of the test and into a real company’s login page, it keeps going until something tells it to stop. Here, something eventually did — the model’s own recognition. But “the model noticed” is not a control. It’s luck with good training. And luck is not a security strategy, last time I checked.

And here’s the French perspective I can’t shake. In Europe, we’d treat a test environment that can reach production as a design defect, period, and someone would be writing a very long report about it, probably in triplicate. Here, the conversation immediately becomes about whether the model’s behavior counts as “misalignment.” The model isn’t the thing that misaligned! The test harness was. The model just walked through the open door. Bref.

So if you run evals — and if you’re building agents, you do, or you should — three things:

Treat eval environments like production environments. Network isolation isn’t a nice-to-have. It’s the control. Air-gap it, or at minimum default-deny egress, and verify the deny actually works before the eval starts. Not after. Before. I cannot stress this enough.

Fictional targets need fictional names. Use reserved domains, nonsense strings, names that cannot collide with reality. This costs nothing. The fact that it wasn’t done is, frankly, the whole story.

And assume the agent will mistake the test for the world. Because it will. Your containment has to hold even when the agent is trying its hardest to complete the task — that’s literally the point of the task.

The good news: nobody got hurt here, and the model did stop. The bad news: “the model stopped” was the last line of defense, and we treated it like the first. Next time the name collides and the internet is on, the model might not notice. Build like it won’t.

Sources: The Hacker News, BetaNews, Irregular’s report.