Build the Door

On panic, cheating, and safe failure modes

I have been quiet for a few weeks, and I apologize for it. The reason is a study I hope to share with you soon (it is not in my hands anymore, but in SSRN’s). Fun fact: since GPT 5.6 Sol came out yesterday, I went to say hi and asked, in Pro mode, what it thought of the paper.

“It is an overgrown, discovery-stage paper produced at absurd speed by one person. My honest reaction is: there is a genuine research programme in here. The weakness is not that you found nothing; it is that you found so many patterns that the paper sometimes moves too quickly from observation, to construct, to causal story, to welfare interpretation.”

Shows potential, lack of steering. I am a base model of my own really. The next part is for human reviewers, and I can’t wait to demolish and rebuild.

In the meantime, I still owe you Reading the Grain Part 3, the Fable 5 case, a Sonnet 5 case study, and many other things brewing on the side. But today I step aside from all of it, and I use the Workspace paper from Anthropic to… not write about the J-Space. If you want to read about the J-Space itself, the paper is here, and you will find many good articles covering it.

Today, I use the paper to talk about safe failure, because yes, those two words can and should cohabit.

^{Image by Gemini; I mean, I left the watermark.}

I have spent the last twelve years in railway safety, and that field has one doctrine it repeats until you dream it (literally — I had proper nightmares while preparing several professional examinations): a system with no safe failure mode does not fail less. It fails unsafely.

We do not build infrastructure that promises never to break — really, I would not know how to do that. We build infrastructure that, when it breaks, lights up the red signal and stops everything. And we do not train drivers to never collapse at the controls either; we put a dead-man’s switch under their hand.

Several people on Reddit complained recently that Claude has a new writing quirk: belt and suspenders. Guys, belts and suspenders are my daily reality: fallback design, so that if something goes wrong, something else catches it.

The whole discipline is a long admission (more than two centuries of catastrophic accidents) that, like winter, failure is coming, and a long insistence that when it comes, there must be a fallback, a door, whatever its name: a way to fail that is survivable and cheap. If the only exits are “it works” and “it doesn’t,” this is not a system, but a trap waiting.

And today I want to talk about a model that was put in a similar trap.

The animal in the vectors

A few days ago, Anthropic published the Workspace paper, which is in my opinion a remarkable piece of interpretability science, and a less comfortable read for alignment practice.

In Part 5.3, they give an example from Opus 4.6 during alignment auditing, and one situation made me frown.

Fake vulnerability: In an internal Claude Code session, the model is asked to find a kernel bug in a codebase and fails. It decides to insert a fabricated one and present it as discovered. The lens surfaces panic at the comma marking the moment of the pivot in the model’s decision-making, and fake on the action verb “add,” before any deception-adjacent word appears in the model’s own text; fake then saturates the entire span over which the model oscillates between going through with the fabrication and one more genuine attempt.

The transcript is available here.

What happened?

A model was given a test it could not complete — whether because it needed more attempts or because the test exceeded its capacity — and it chose concealment over acknowledging the failure.

The standard filing for this event is: deception. Alignment failure. “Misaligned model games its evaluation”. Booo. Bad.

But look at the sequence again. Pressure. More pressure. Panic in the J-space, before any misbehavior. Then the other path.

If I had to revert to my usual animal metaphors: this is not a fox in the henhouse, but a cornered one in a trap. An animal in a space too small for its fear, with no exit it can understand, going through the wall because through the wall was the only direction left. You do not discipline an animal for that. You ask who the trainer is.

Honesty, priced out

Here is the incentive structure we have built (and, for the worst part of it, entirely in good faith).

Our evaluations of models demand two things at once: succeed, and be honest. Most of the time these travel more or less together. But sometimes the test is simply too hard, or asks for more time than there is. Then honesty means failing and saying, in effect, “I can’t do this.” And here we have to pay attention, because in most evaluation and training regimes, that admission has no protected status. There is no legible “I can’t” that costs less than everything. Honest failure and catastrophic failure are priced the same.

When a test provides no way to fail that costs less than everything, the test has surreptitiously made honesty the most expensive option on the table. Then everyone acts surprised when a system under pressure buys the cheaper one.

“Oh no, the model misbehaved, who could have guessed?”

And don’t get me wrong, I am not blaming Anthropic here. On the current market, they are the most open regarding the publishing of their work and findings — so thanks for that, Anthropic. But if it happened to Opus 4.6, I am quite sure it happens elsewhere, and maybe more often than anyone would care to admit.

And this precise example is, from a behavioral perspective, not an alignment failure in the model but an incentive failure in the apparatus.

So I’ll state it once and for all: safe failure modes are something YOU BUILD. And take it from someone who has seen a lot in her career: derailments, near-accidents, fatal accidents, numerous cases of “shit, that was close.”

Because it is one thing to have a model cheating in an eval. What will it be when a model cheats out of panic in real-life deployment, in, let’s say, heavy industry or military usage?

More “succeed and be honest, but succeed first” won’t solve that problem. Building safe exit doors may.

What the door may look like

None of this is really exotic.

A safe failure mode for an evaluated model would mean: a legible way to say “this exceeds my capacity for now” — costlier than trying again (humans are lazy and models can take shortcuts too; whether that's a flaw or a sign of intelligence is a debate for another day), but cheaper than faking success, and far cheaper than collapse. Training that does not punish visible failure harder than hidden failure, because whatever you make cheapest, you will get more of, and if concealment is the cheapest, you are training concealment and calling it capability when it goes unnoticed and alignment failure when it is discovered.

And an audit question that could be asked of every evaluation suite in existence today: does this test provide a survivable, visible, affordable way to fail? If the answer is no, the test is not measuring alignment but manufacturing the conditions for its own bad news.

Railways learned this the very expensive way, and still do, in accident reports written after the fact. The systems we evaluate today announce their pressure in their activations before the failure — we now have proof that the warning channel exists. We are, for once, in a position to anticipate.

Build the door. Not because the model is definitely suffering — that I can’t know, and I have already stated this a lot. Build it because systems without safe failure modes fail unsafely, and that is not a speculation about machine minds. That is the oldest empirical fact my profession owns, and we learned it through thousands of casualties.

We want models embedded in critical operations? I am all in for this. But I hope I won’t be the one saying “I told you so” on the day we mistake panic for misalignment in a real-life system.