Using workload capacity indicators to evaluate rule-based early warning tools and their relationship to escalation events

Objective: To compare prediction accuracy of rule-based early warning tools (EWTs) using a large healthcare electronic medical record (EMR) dataset and to re-evaluate using a novel hospital workload capacity evaluation method. Materials and methods: Adult inpatient admissions to 11 Australian hospitals were included in a retrospective analysis of four EWTs: National Early Warning Score (NEWS), Between the Flags (BTF), Modified Early Warning Score (MEWS) and Queensland Adult Deterioration Detection Systems (Q-ADDS). Using death and unplanned transfer to the intensive care unit (UICU) as composite outcome, each EWT was evaluated with area under the receiver operating curve (AUROC), sensitivity and positive predictive value (PPV). A second analysis was performed with clinician workload capacity indicators. Results: A total of 683,617 admissions were analysed, including 4954 deaths and 3400 UICU. NEWS2 AUROC was superior to Q-ADDS (1.6%, p < .001), MEWS (3.1%, p < .001) and BTF (28%, p < .001). At each alert threshold, Q-ADDS had superior PPV. Q-ADDS and MEWS operated at the lowest alert burden (1.0–3.8 alerts per 100 patient days) across all alert thresholds [low, moderate and Medical Emergency Team (MET)], followed by NEWS2 (1.9–5.5) and BTF (4.1–18). Conclusion: Precision-recall workload capacity analysis provides a visual means of displaying the operational characteristics of EWTs in terms of EWT alert thresholds, resultant alert rates and traditional EWT accuracy (PPV and sensitivity). It may be helpful for healthcare organisations to consider clinician workload capacity, in addition to traditional evaluation metrics such as sensitivity and PPV, when selecting EWTs or setting escalation thresholds.