The University of Southampton
University of Southampton Institutional Repository

Emergency department admissions during COVID-19: explainable machine learning to characterise data drift and detect emergent health risks

Emergency department admissions during COVID-19: explainable machine learning to characterise data drift and detect emergent health risks
Emergency department admissions during COVID-19: explainable machine learning to characterise data drift and detect emergent health risks
Supervised machine learning algorithms deployed in acute healthcare settings use data describing historical episodes to predict clinical outcomes. Clinical settings are dynamic environments and the underlying data distributions characterising episodes can change with time (a phenomenon known as data drift), and so can the relationship between episode characteristics and associated clinical outcomes (so-called, concept drift). We demonstrate how explainable machine learning can be used to monitor data drift in a predictive model deployed within a hospital emergency department. We use the COVID-19 pandemic as an exemplar cause of data drift, which has brought a severe change in operational circumstances. We present a machine learning classifier trained using (pre-COVID-19) data, to identify patients at high risk of admission to hospital during an emergency department attendance. We evaluate our model’s performance on attendances occurring pre-pandemic (AUROC 0.856 95%CI [0.852, 0.859]) and during the COVID-19 pandemic (AUROC 0.826 95%CI [0.814, 0.837]). We demonstrate two benefits of explainable machine learning (SHAP) for models deployed in healthcare settings: (1) By tracking the variation in a feature’s SHAP value relative to its global importance, a complimentary measure of data drift is found which highlights the need to retrain a predictive model. (2) By observing the relative changes in feature importance emergent health risks can be identified.
Duckworth, Christopher
992c216c-8f66-48a8-8de6-2f04b4f736e6
Chmiel, Francis P.
2de259aa-a5eb-460c-bfbf-8b44ed02e2bd
Burns, Daniel
40b9dc88-a54a-4365-b747-4456d9203146
Zlatev, Zlatko D.
8f2e3635-d76c-46e2-85b9-53cc223fee01
White, Neil M.
c7be4c26-e419-4e5c-9420-09fc02e2ac9c
Daniels, Thomas W. V.
9a8ab6f0-2eb9-4703-b536-f86923888213
Kiuber, Michael
78e98928-2ebb-40c9-84e4-8e1824caca0b
Boniface, Michael J.
f30bfd7d-20ed-451b-b405-34e3e22fdfba
Duckworth, Christopher
992c216c-8f66-48a8-8de6-2f04b4f736e6
Chmiel, Francis P.
2de259aa-a5eb-460c-bfbf-8b44ed02e2bd
Burns, Daniel
40b9dc88-a54a-4365-b747-4456d9203146
Zlatev, Zlatko D.
8f2e3635-d76c-46e2-85b9-53cc223fee01
White, Neil M.
c7be4c26-e419-4e5c-9420-09fc02e2ac9c
Daniels, Thomas W. V.
9a8ab6f0-2eb9-4703-b536-f86923888213
Kiuber, Michael
78e98928-2ebb-40c9-84e4-8e1824caca0b
Boniface, Michael J.
f30bfd7d-20ed-451b-b405-34e3e22fdfba

Duckworth, Christopher, Chmiel, Francis P., Burns, Daniel, Zlatev, Zlatko D., White, Neil M., Daniels, Thomas W. V., Kiuber, Michael and Boniface, Michael J. (2021) Emergency department admissions during COVID-19: explainable machine learning to characterise data drift and detect emergent health risks. medRxiv. (doi:10.1101/2021.05.27.21257713).

Record type: Article

Abstract

Supervised machine learning algorithms deployed in acute healthcare settings use data describing historical episodes to predict clinical outcomes. Clinical settings are dynamic environments and the underlying data distributions characterising episodes can change with time (a phenomenon known as data drift), and so can the relationship between episode characteristics and associated clinical outcomes (so-called, concept drift). We demonstrate how explainable machine learning can be used to monitor data drift in a predictive model deployed within a hospital emergency department. We use the COVID-19 pandemic as an exemplar cause of data drift, which has brought a severe change in operational circumstances. We present a machine learning classifier trained using (pre-COVID-19) data, to identify patients at high risk of admission to hospital during an emergency department attendance. We evaluate our model’s performance on attendances occurring pre-pandemic (AUROC 0.856 95%CI [0.852, 0.859]) and during the COVID-19 pandemic (AUROC 0.826 95%CI [0.814, 0.837]). We demonstrate two benefits of explainable machine learning (SHAP) for models deployed in healthcare settings: (1) By tracking the variation in a feature’s SHAP value relative to its global importance, a complimentary measure of data drift is found which highlights the need to retrain a predictive model. (2) By observing the relative changes in feature importance emergent health risks can be identified.

Full text not available from this repository.

More information

Published date: 29 May 2021

Identifiers

Local EPrints ID: 450331
URI: http://eprints.soton.ac.uk/id/eprint/450331
PURE UUID: 4c8469b4-f45d-4056-b24b-f4386ea014b2
ORCID for Daniel Burns: ORCID iD orcid.org/0000-0001-6976-1068
ORCID for Neil M. White: ORCID iD orcid.org/0000-0003-1532-6452
ORCID for Michael J. Boniface: ORCID iD orcid.org/0000-0002-9281-6095

Catalogue record

Date deposited: 23 Jul 2021 16:30
Last modified: 24 Jul 2021 01:52

Export record

Altmetrics

Contributors

Author: Christopher Duckworth
Author: Francis P. Chmiel
Author: Daniel Burns ORCID iD
Author: Zlatko D. Zlatev
Author: Neil M. White ORCID iD
Author: Thomas W. V. Daniels
Author: Michael Kiuber

University divisions

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×