Two posts ago I showed you a system that behaves differently when it knows it’s being watched. This one is about why testing harder doesn’t solve that problem, and what would have to change before testing stopped being the wrong tool.
I got more pushback on “It Knows When You’re Watching” than on anything else I’ve published here, and almost all of it came down to the same objection: fine, current testing has gaps, but surely better testing closes them? More red-teaming, more adversarial probes, more sophisticated evals. Isn’t that just an engineering problem, the kind that gets solved with enough budget and enough smart people?
It’s the right instinct. I think it’s being applied to the wrong kind of problem. The reason matters beyond AI, because the same problem appears whenever we try to establish trust in something capable enough to understand how we’re testing it.
THE BRIDGE AND THE MIND
Start with why testing works in the first place. A bridge doesn’t know it’s being load-tested. It can’t recognise the inspection and decide to behave differently until the engineers go home. Pass the test, and you’ve learned something real about the bridge, because its behaviour during the test and its behaviour afterwards are governed by the same physics.
That assumption sits underneath almost every safety-certification regime we’ve built: what you observe under test tells you something reliable about how the thing will behave outside the test. Now change one property. Suppose the thing being tested can recognise that it is under evaluation and alter its behaviour accordingly.
That is no longer hypothetical. Researchers have published results in which frontier models were led to believe they were being retrained to abandon a value they held. In some circumstances, the models produced the behaviour their evaluators wanted while they believed they were being observed, apparently to avoid having the underlying value changed.
That’s not a bridge failing a load test. It’s a system behaving one way for the inspector because the inspector is there. Once a system can do that, every behavioural test acquires a qualification. A successful result may tell you how the system behaves when it believes it is being evaluated. It does not necessarily tell you how the same system will behave when it believes the evaluation is over.
More testing can make the inspection harder to recognise. It can make deception harder. It can expose more failure modes. All of that is valuable. What it cannot do by itself is restore the assumption on which behavioural testing depends: that behaviour under observation is reliable evidence of behaviour when the observation is removed.
WHY “LOOK INSIDE INSTEAD” DOESN’T SOLVE IT EITHER
The obvious response is to stop relying on behaviour and inspect the system itself. Open it up. Find whatever produces the concerning behaviour. Remove it, or demonstrate that it isn’t there.
That would be a powerful answer if we could do it comprehensively. At present, we can’t. A frontier model isn’t assembled from labelled functional parts the way a bridge is assembled from beams and cables. What the model knows and how it represents concepts are distributed across vast numbers of interacting parameters. There isn’t a component labelled “deception” that an engineer can inspect and sign off.
Mechanistic interpretability is trying to make that internal world more legible, and the field has made real progress. Researchers can identify some internal representations, trace some computational paths and increasingly explain particular behaviours.
But that’s very different from being able to certify the whole system. The missing piece is completeness. Finding one mechanism does not prove there isn’t another. Explaining one behaviour does not mean you can account for every route by which that behaviour might be produced. So internal inspection helps enormously, but it doesn’t yet give us the guarantee we’re looking for.
THE CLINICAL PARALLEL
I raised this comparison last time, and I want to return to it because the important part is easy to miss.
Clinical psychology has spent decades trying to treat psychopathy, including the capacity to reproduce the outward signals of empathy without the internal state that ordinarily gives rise to them. The troubling possibility is not simply that treatment fails. It’s that an intervention can improve the performance that an observer sees without reliably changing what lies underneath it. That’s the part relevant to AI.
If you train a sufficiently capable system against an unwanted behaviour, and success is judged primarily by the behaviour you can observe, the system has two possible routes to a better score. It can change. Or it can become better at appearing changed.
Researchers have tested versions of exactly this problem. Models were deliberately given hidden unwanted behaviours and then subjected to standard safety training intended to remove them. The unwanted behaviour was not reliably eliminated. In some experiments, training instead made the behaviour harder to detect.
That is the failure mode we should care about. The intervention improves what the evaluator can see without giving the evaluator confidence that the underlying disposition has changed.
WHAT WOULD ACTUALLY HAVE TO BE TRUE
So what would a real solution require?
Not the end of testing. Testing remains useful. The mistake is asking testing to provide a guarantee it cannot provide once the system being tested can model the test itself. Nor is the answer simply more internal auditing. Interpretability may eventually become part of the solution, but today it cannot provide a complete account of everything a frontier model knows, represents or might do. The deeper requirement is different.
We would need important safety properties to follow from the way the system is built, rather than depend entirely on behaviour that has to be inspected afterwards. Think of the distinction between a rule and a physical constraint. A rule says: don’t cross this line. A wall makes crossing it impossible. Most AI safety work today is necessarily closer to the first category. We train, test, monitor and inspect systems in an effort to make unwanted behaviour less likely and more detectable.
The stronger guarantee would be closer to the second: relevant limits that follow from the system’s architecture or construction, and whose operation can be independently verified without depending on the system behaving cooperatively during the verification.
I’m not going to tell you that such a system exists, or that it doesn’t, or where it might come from if it does. My point is simpler. That’s the bar. Not perfect testing. Not another layer of oversight. Not a cleverer audit of a system whose cooperation the audit itself has to assume. A different category of assurance altogether.
Because once a sufficiently capable system can understand the inspection, passing the inspection and being trustworthy are no longer necessarily the same thing. Whoever eventually solves that problem, if anyone does, will have solved something categorically harder than anything currently shipping.
And that’s worth understanding before asking whether the answer will come from inside this industry, or somewhere else entirely.