Asuu

The next milestone

The first evaluation.

One workflow, invented patients, and a comparison we could lose.

Evaluation plan. Not yet run.

On this page
A hand holding a measuring divider, engraved in cobalt

The question.

Everything on this site rests on one claim that has not yet been tested. Can someone outside a hospital reconstruct, from a record alone, what a model did, in which version, and who authorised it? And is a shared record any better at this than something far simpler?

Until that is answered, the rest is a plan. This page sets out how we intend to answer it. We are publishing the method before running it, so that the result cannot be shaped afterwards to suit us.

The setting.

One institution and one small workflow. A model flags a case. A clinician reviews the flag and authorises an action, or declines to. Partway through, the model is updated to a new version, and the new version needs fresh authorisation.

Every patient in it is invented. No real record is used. The model is a demonstration and its medical accuracy is not being tested. The only thing under examination is whether the history of what happened can be recovered and trusted.

The comparison.

The same events are fed into two systems. The first is a prototype of the Network's record. The second is the simplest thing a competent hospital could build in a month: an ordinary log that the hospital signs and stores itself.

An independent auditor, who had no part in building either, is then asked to reconstruct what happened from each. We also interfere. After the events, we quietly alter the history in both systems and see which alteration the auditor catches.

If the simple log does the job as well, we will say so and build something else.

What we would measure.

Five things. Whether the complete history can be rebuilt. How many hours the auditor needs. How much work it takes to connect the system to what the institution already runs, and what it costs to keep running. How many minutes it adds to the day of the people using it.

The fifth is the one the Network exists for: what a shared record offers that a private one cannot. That means evidence that nothing was rewritten after the fact, and the ability to compare one institution's history with another's.

What would count as failure.

If the signed local log lets the auditor rebuild the history as fully, and catch the tampering as reliably, then for this purpose the Network is not needed. If the prototype costs clinicians more minutes than the institution is willing to spend, it will not be used, however elegant it is.

We will publish the outcome in either case. A result that tells us to stop is worth more than a year of building the wrong thing.

What it would not show.

It would say nothing about whether a model's advice is medically right, whether patients do better, or whether hospitals would ever adopt such a system. It is one institution, with invented data. A good result earns the next test and nothing more.

Who we need.

The people in one institution who are responsible for governing clinical AI, for its information systems, and for audit. An auditor with no stake in the answer. And critics willing to read this plan and tell us what it fails to test, before we run it.

Where this connects.

Help us earn the next stage.

If you can tell us where the work is wrong, or where it could be tested, that is the most useful thing we could hear.