Asuu

A machine that knows where its knowledge ends.

The most useful thing a model can say is sometimes that it does not know.

Research direction. Not yet tested.

On this page

What we know.

A good junior doctor has one habit above all others. When a case is beyond them, they say so and fetch someone. Modern models tend to lack it. Asked something outside their experience, they answer in the same fluent, assured voice they use for everything.

There is a measure for this. A model is well calibrated if, among all the answers it gives with ninety per cent confidence, nine in ten are right. Researchers showed in 2017 that the large neural networks then coming into use were markedly more confident than they were correct.

The open question.

Accuracy is only one part of evaluating a model. Calibration, uncertainty, and the choice to abstain are also studied. Asuu asks how these properties could be measured for a defined medical use and made clear to the person responsible for a decision.

What Asuu could do.

The core protocol already treats this as a matter of safety: a model whose confidence drifts away from its accuracy is to be suspended. An independent examiner could go further and score every model on its honesty about its own limits, alongside its accuracy.

The record could keep each occasion on which a model stepped back. And the assistant would be held to the first rule of its bedside manner, which is to say plainly what it does not know.

We rank models by how often they are right. We should also ask how well they know when they are wrong.

The first experiment.

Take models that are freely available and score them twice on the same medical questions: once for being right, and once for knowing when they are probably wrong. Publish both tables side by side. If the two rankings turn out to be the same, there is nothing new to measure. We suspect they are not.

What could go wrong.

A model that declines too often is useless, and each refusal hands the work back to a tired clinician. A maker could also learn to duck the hard cases and look better for it. The score would have to punish a needless refusal as firmly as a confident mistake.

Sources and scope

The studies listed here motivate a research question. They do not establish that Asuu’s proposed method works. The experiment described here has not been run.

References may be in English.

Where this connects.

Which horizon is yours?

If you work in one of these fields, we would like to understand what it needs.