Silent validation means the model scores the unit's own data every hour and logs each prediction, but no clinician sees an alert and no care process changes.
This is the step between retrospective evidence and any bedside use. It tests what public data can't tell us: whether the unit's own feeds arrive on time and in the right units, how often the model would flag, and whether its risk estimates hold on local patients.
Scope
Adult ICUs, for one to three months, with a prediction logged every hour for each eligible stay.
Fixed before the first prediction
The model version and the threshold policy will be frozen, and the metrics will be pre-registered before the first prediction is logged. The metrics are AUROC and AUPRC for each endpoint; sensitivity, specificity, PPV and NPV at the frozen thresholds; flags per patient-day; median lead time from first flag to event; and calibration, measured by Brier score and observed against predicted risk by decile and tracked for drift over the pilot.
Secondary measures will include performance by unit, drift in data availability, model uptime, and chart review of false positives and missed events.
Governance
The protocol requires prediction logs to stay inside the hospital's approved environment. The unit agrees who can see patient-level logs, how long they are kept, and obtains ethics and data-use approval before logging begins.
Before any alert is shown
Alerting follows only when silent validation shows stable calibration, an acceptable alert burden and clinician agreement on a threshold policy. Liver failure, kidney failure and hyperglycaemia are the recommended first endpoints. Liver and kidney failure combine strong discrimination with a long lead time; hyperglycaemia discriminates well, but its alert frequency needs local workflow review. Sepsis and circulatory failure are to be used with caution because of short lead times. Kidney and liver thresholds need review: at the conservative setting neither caught a true event in retrospective scoring.
