Predicting outcomes for severe COVID-19 patients
22 March 2021
Recent research using nationwide patient data and building on existing informatic tools, such as CogStack, examined the use of early warning scores for COVID-19.
Overview
The speed and scale of the coronavirus pandemic has highlighted the need for rapid research to help shape treatment and save lives. When COVID-19 patients arrive in hospital clinicians have been advised to use the NEWS2 (National Early Warning Risk Score) to predict who are at most at risk of dying or needing ICU care over the ensuing 14 days. But is it effective? Researchers needed nationwide patient data to find out. Thanks to a collaboration with HDR UK they were able to link up with hospitals across the country, supply them with computer code to interrogate their health records, and start getting answers in days. The resulting BMC Medicine paper alerted the NHS that, for 14-day outcomes, NEWS2 was of poor to moderate use on its own (though it performed better over shorter periods) but there were straightforward ways to increase its value.
The challenge
In 2020 COVID-19 was a new threat which was little understood, and the sheer volume of hospital admissions put huge pressure on NHS staff and resources. Clinicians were advised to use NEWS2, NHS England’s standard tool for the assessment and response to acute illness, to help predict which patients were at highest risk in the fortnight after admission. However NEWS2 was last updated in 2017 and was generally used to anticipate the next 72 hours.
For patients with severe COVID-19 it is vital to know who might need organ support or is at risk of dying in the medium term. If NEWS2 was ineffective, the NHS needed to know.
Given the urgency the research team — led by a team including Dr Ewan Carr, Dr Rebecca Bendayan, Dr James Teo and Prof. Richard Dobson of King’s College London — wanted to get results quickly. The fact that so many people were working from home was an additional obstacle. A conventional approach, involving the creation and negotiation of a large repository of centrally held data, would have been slow and cumbersome.
The solution
Initially the team analysed data from 1,276 COVID-19 patients admitted to King’s College Hospital NHS Foundation Trust from March to April 2020. Collaboration with HDR UK then gave them a fast, effective and secure way to gather data from other trusts in England to validate their models. This was done by sharing code, via GitHub, which trusts could run on their own systems to extract the relevant data from their existing electronic health records (EHRs).
The trusts that took part were Guy’s and St Thomas’ NHS Foundation Trust Hospitals, University Hospitals Southampton, University Hospitals Bristol and Weston NHS Foundation Trust, University College London Hospitals and University Hospitals Birmingham. Data was also provided by one hospital in Norway and two in China.
Overall the study was able to analyse evidence from 6,000 patients and nine hospitals. This was used to evaluate how well patients’ NEWS2 scores measured on admission anticipated severe COVID-19 outcomes – defined as either ICU care or death.
Much of the work took place in the late summer of 2020 and the team was soon able to start gauging the effectiveness of NEWS2 and how it could be improved. After peer review a paper was published in January 2021.
What was learned
The main finding was that NEWS2 was poor to moderate for predicting the most severe cases of COVID-19 over a 14 day period and tended to underestimate who was at most risk. But, the accuracy could be increased by adding in other routinely collected information including age, oxygen saturation and neutrophil count.
The researchers also found that NEWS2 plus age provided moderate accuracy over a three-day period.
The study was the largest to date to evaluate the accuracy of NEWS2 for predicting medium-term COVID outcomes. It demonstrated the speed and effectiveness of a federated rather than a centralised approach. This is something being championed by HDR UK – which provided researchers with the means to swiftly gather large-scale data from multiple sources, across the country.
Impact and outcomes
Healthcare providers are now better placed to help patients with severe cases of COVID-19 and identify the resources they will need.
The team identified HDR UK’s contribution as being highly significant. It facilitated their research, allowing them to take a federated approach and to perform what they described as “rapid and agile analysis”.
The project also raised questions about healthcare scoring systems in general – with concerns that their value has often not been subject to proper research.
HDR UK aims and priorities
HDR UK has a series of priorities designed to maximise the impact of health data research in addressing the most pressing healthcare needs.
The NEWS2 evaluation fitted with the priority it puts on applied analytics and better care. The project reflected HDR UK’s commitment to team science and to open science (the code and pre-trained models have been openly shared for testing in other Covid-19 datasets).
The research also fitted with HDR UK’s work to support healthcare decision-making.
Team and authors
The study was a collaboration between the National Institute for Health Research (NIHR) Maudsley Biomedical Research Centre (BRC) and the Medical Research Council (MRC). The research was carried out by the King’s College London Institute of Psychiatry, Psychology & Neuroscience.
The BMC paper joint first authors were Dr Ewan Carr and Dr Rebecca Bendayan. The team and authors also thanked the public and other partners for supporting their work.
Abstract
Background: The National Early Warning Score (NEWS2) is currently recommended in the UK for the risk stratification of COVID-19 patients, but little is known about its ability to detect severe cases. We aimed to evaluate NEWS2 for the prediction of severe COVID-19 outcome and identify and validate a set of blood and physiological parameters routinely collected at hospital admission to improve upon the use of NEWS2 alone for medium-term risk stratification.
Methods: Training cohorts comprised 1276 patients admitted to King’s College Hospital National Health Service (NHS) Foundation Trust with COVID-19 disease from 1 March to 30 April 2020. External validation cohorts included 6237 patients from five UK NHS Trusts (Guy’s and St Thomas’ Hospitals, University Hospitals Southampton, University Hospitals Bristol and Weston NHS Foundation Trust, University College London Hospitals, University Hospitals Birmingham), one hospital in Norway (Oslo University Hospital), and two hospitals in Wuhan, China (Wuhan Sixth Hospital and Taikang Tongji Hospital). The outcome was severe COVID-19 disease (transfer to intensive care unit (ICU) or death) at 14 days after hospital admission. Age, physiological measures, blood biomarkers, sex, ethnicity, and comorbidities (hypertension, diabetes, cardiovascular, respiratory and kidney diseases) measured at hospital admission were considered in the models.
Results: A baseline model of ‘NEWS2 + age’ had poor-to-moderate discrimination for severe COVID-19 infection at 14 days (area under receiver operating characteristic curve (AUC) in training cohort = 0.700, 95% confidence interval (CI) 0.680, 0.722; Brier score = 0.192, 95% CI 0.186, 0.197). A supplemented model adding eight routinely collected blood and physiological parameters (supplemental oxygen flow rate, urea, age, oxygen saturation, C-reactive protein, estimated glomerular filtration rate, neutrophil count, neutrophil/lymphocyte ratio) improved discrimination (AUC = 0.735; 95% CI 0.715, 0.757), and these improvements were replicated across seven UK and non-UK sites. However, there was evidence of miscalibration with the model tending to underestimate risks in most sites.
Conclusions: NEWS2 score had poor-to-moderate discrimination for medium-term COVID-19 outcome which raises questions about its use as a screening tool at hospital admission. Risk stratification was improved by including readily available blood and physiological parameters measured at hospital admission, but there was evidence of miscalibration in external sites. This highlights the need for a better understanding of the use of early warning scores for COVID.