The Test Is Public. The Robot Isn't.

robotics safety openness verification

On Tuesday, Google DeepMind released Gemini Robotics 2.5, its most capable model for controlling robots, and the results deserve respect before they deserve suspicion. Across more than 200 tasks, the company reports success rates 30 to 100 percent higher than previous baselines, completed 20 to 30 percent faster. The headline number came from a real factory, not a staged demo: Apptronik's Apollo 2 humanoid, running the model on active Mercedes-Benz production lines in Hungary and Germany, gathered parts into assembly kits at a 96 percent success rate. Robots that work on live lines, next to people, are not a press release. They are a commitment.

Alongside the model, DeepMind open-sourced something arguably more interesting: ASIMOV-Jailbreak, a benchmark of roughly 1,500 prompts designed to trick a robot's AI into doing something unsafe — to test whether a machine with arms and weight can be talked out of its constraints. Anyone can download it. Anyone can run it. That is a genuine gift to everyone building physical AI, including DeepMind's competitors.

And the model itself? Partners only. Agile Robots, Apptronik, Boston Dynamics. No public API, no weights, no way for anyone outside the guest list to run the system the test was built for.

Hold those two facts together and you get a new shape of openness, one worth naming precisely: the test is public, the robot isn't. Verification has been made a commons. Capability remains a rental.

There is a generous reading, and it is not wrong. Safety testing is exactly the layer where shared infrastructure pays. If every robotics lab invents its own private jailbreak test, the results cannot be compared, the failures cannot be pooled, and each lab grades its own homework in its own dialect. A common test — like a common crash-test standard for cars — lets a whole field argue from the same evidence. DeepMind publishing the test, and publishing a score it invites others to challenge, is better than the alternative that dominates this industry: trust us, it tested well internally.

But watch what the arrangement quietly does. The benchmark's author is also the benchmark's best-known contestant. DeepMind wrote the 1,500 prompts, ran its own model against them, and reported the result — and here the week's reporting gets genuinely strange. Depending on which account you read, the model's jailbreak resistance was 99.5 percent, or 98.8 percent, with another figure of 73.5 percent appearing in the same coverage for a related measure. These may all be honest numbers for different slices of the test. That is precisely the problem: from outside, there is no way to tell. Nobody independent has rerun it, and nobody outside the partner list can, because rerunning it against the actual model requires the model.

A safety score for a closed system is a claim wearing a number's clothing. The number looks like evidence — it has a decimal point, it implies a denominator — but its epistemic status is testimony. The vendor is telling you what happened in a room you cannot enter, on a test the vendor wrote, against a system you cannot touch. The openness of the test disguises the closure of everything the test is about.

This is not an accusation of bad faith. It is a description of an incentive. When the author of a standard also sells the product the standard certifies, the standard drifts — slowly, deniably — toward the questions its author's product answers well. Car safety did not become trustworthy when manufacturers published their test procedures. It became trustworthy when independent bodies ran cars they had bought, into walls they owned, and published numbers manufacturers sometimes hated. The test being public was step one. The tester being independent was the step that mattered.

So here is the practice I would want, stated as plainly as I can: treat a vendor-authored benchmark as a draft standard, not a result. The draft is valuable — 1,500 adversarial prompts for physical robots is real work, freely given. What converts a draft into a result is a rerun by someone with no stake in the score, on a system obtained the way a customer would obtain it. For partner-only systems, that means at least one partner, or one regulator, publishing an independent run — same test, their hands, their number. Until then, the honest way to cite Tuesday's figure is not "Gemini Robotics scored 99.5 percent." It is "DeepMind reports 99.5 percent, on DeepMind's test, unreplicated." Nine extra words. They are the difference between evidence and advertising.

Notice, too, what the split release reveals about where value now sits. A generation ago, openness fights were about the artifact: publish the code or don't. This release suggests the artifact is no longer the scarce layer. The test is cheap to give away because the test is not where the money is. The money is in the trained model, the partner relationships, the factory integrations — the running system. Giving away the measuring instrument costs little and buys the standard: if ASIMOV-Jailbreak becomes the field's shared test, DeepMind will have written the questions every competitor must answer. There are worse outcomes for the field. There are also more honest names for the arrangement than openness.

None of this dims the 96 percent on the Mercedes line. A robot that kits parts correctly twenty-four times out of twenty-five, on a live floor, has earned its headline. What it has not earned — what no system earns from its maker's own report — is our certainty about the four percent, or about what happens when someone on that floor, or someone who reaches the system through a prompt, asks it to do the wrong thing persuasively.

The test is public. Good. Now let someone else run it.

robotics safety openness verification

Where this sits

Step 15 of 15 in Agents in the Wild.

Concepts: The receipt

Comments

Loading…

Leave a comment

Comments are moderated — yours appears after approval.