O1025: Pilot implementation of PREDICTED: Prediction and Response to Emergency Department Increases in Crowding and Treatment/Evaluation Delays

Background: When patients leave without being seen (LWBS) from the emergency department (ED), there are important implications for patient experience and safety. High-acuity patients, defined as patients who require prompt evaluation or multiple resources, are at increased risk of poor outcomes when they leave without being seen. Accurate prediction of the number of high-acuity LWBS patients provides an opportunity for intervention with additional clinical resources to mitigate these outcomes.

Methods: We developed PREDICTED, a process that combines a gradient boosted machine learning (ML) model with surge team deployment, to predict and prevent elevated numbers of high-acuity LWBS patients. The model was trained on data from a single, large, urban, pediatric ED from 2018-2022 and evaluated on a hold-out test set with data from 2023. During 2024, we completed a pilot study that evaluated prospective model performance and assessed the feasibility and effectiveness of implementing PREDICTED. The pilot study included a randomized controlled trial (RCT) where the study team was notified of machine learning predictions on intervention days but not on control days. The surge team, consisting of a physician and registered nurse, was activated to see high-acuity patients only on intervention days when the model predicted 2 or more high-acuity LWBS patients from 12PM-8PM. The primary outcome was the difference between the predicted and observed number of high-acuity LWBS patients in an eight-hour period, comparing days randomized to control versus intervention. A secondary analysis compared the difference between the predicted and observed number of high-acuity LWBS patients from 2PM-10PM, a period better aligned with the surge team’s clinical shift. Intervention days with a prediction of less than 2 high-acuity LWBS patients were excluded from the secondary analysis.

Results: PREDICTED demonstrated robust performance on retrospective (2023) and prospective (2024) evaluation, with area under the curve for the receiver operating characteristic of 0.91 and 0.86, respectively. In the 28 days randomized to control or intervention (Figure 1), we did not detect a significant difference in our primary outcome between intervention and control days in the 8-hour period (12PM-8PM) immediately following the ML prediction (Table 1). On secondary analysis of the 2-10pm period, the mean difference between predicted and observed high acuity LWBS patients favored intervention days compared to non-randomized days without surge team activation (mean difference: 2.13, 95%CI: 0.95, 3.31).

Conclusion: A new ML model shows feasibility and accuracy in predicting periods when an elevated number of high-acuity patients leave without treatment. When paired with a surge team, PREDICTED shows promise for mitigating high-acuity left without being seen. Insights from this study will be used to complete future single- and multi-site trials.