Data science to analyse the largest natural experiment of our time

Pandemics have left indelible marks on medicine. The 1918 influenza accelerated the emergence of American medicine as a scientific enterprise.¹ HIV ushered in universal precautions as the minimum requirement for the care of every patient.² One defining impact of COVID-19 might be the widespread adoption of telemedicine.³ However, though telemedicine is a change out of necessity, should we look for other opportunities to transform medicine for the better?

Consequences of the unprecedented national lockdowns include massive reductions in emergency room visits and hospitalisations, as well as cancellations of the majority of elective procedures. Some decrease in medical care would be expected during such a crisis. However, the magnitude of decrease in essential care is striking: for example, cardiac catheterisation for ST-elevation myocardial infarction was reduced by 38% in some areas during peaks in infection rates.⁴ Stroke presentations in some hospitals were halved during the height of the pandemic.⁵ Many surgical procedures were and continue to be delayed to a safer time.⁶ In some areas, endoscopic and colonoscopic procedures are only advised if absolutely necessary, weighed against the risk of contracting coronavirus.⁷

While it is clear that missing some treatments is harmful—undiagnosed or untreated heart attacks precipitate cardiac arrest, heart failure, or death—the avoidance of other treatments may be beneficial. We should take advantage of this opportunity to investigate which of the deferred interventions matter as much as we think they do. COVID-19 presents us with a natural experiment to study changes that otherwise would have been logistically impossible, seemingly unethical, or culturally incongruous. What happens when we delay particular procedures? Which procedures are more vital than others in terms of improving quality of life? Critically: how often do our tests and treatments lack clear value?

Answers to these questions may inform care decisions as nations face subsequent peaks in infection rates as well as future epidemics and pandemics. Similarly, even after the COVID-19 pandemic, studies of the consequences of omissions and delays done to minimise COVID-19 infection risk may inform ways in which systems can limit low-value care and instead focus resources on high-value care.

There is precedent for such investigations. A 2018 study in the Journal of the American Heart Association found that Medicare beneficiaries hospitalised at academic centres with acute myocardial infarctions experienced different outcomes based on whether or not they were hospitalised during dates of a major annual interventional cardiology meeting; a subgroup of patients who did not receive percutaneous coronary interventions (PCI) experienced improved outcomes if they were admitted during meeting days.⁸ Patients who presented on meeting days were pseudo-randomised to practice patterns of providers who chose not to attend the meeting, creating a natural experiment that enabled the study of non-procedural treatment of some of these high-risk patients. Today, a similar scenario is playing out on a larger scale. Patients who are not receiving treatment because of the crisis have been pseudo-randomised to the non-treatment arm.

In light of such pseudo-randomisation due to the pandemic, statistical techniques such as causal inference analysis will allow the use of what will become retrospective data, as the future unfolds, to ask questions in ways that simulate a randomised controlled trial (RCT). The last decade has seen a surge in the use of data science and artificial intelligence in clinical research. However, many studies using machine learning techniques have focused on prediction of events, trajectories and outcomes by discovering patterns and associations, leading to models that despite high accuracy, fail as soon as they are deployed as tools in the real world.⁹ Applying causal inference approaches to retrospective analyses, whether making use of artificial intelligence techniques or less complex studies, may facilitate more generalisable research whose conclusions inform the provision of value-based care after the pandemic.

Simply discovering and modelling the relationship between features to make a prediction, classification, or optimisation without acknowledging causal pathways may lead to errors when models are applied to settings that are not identical to those from which the training and test data were obtained. Furthermore, although the gold standard for inferring causality is the RCT, performing RCTs for every question, for every patient population, for every clinical context, and for every temporal change in practice patterns is simply not feasible. The causal inference methodologies were created to leverage observational data to estimate effect size when treatment decisions are not randomised.¹⁰ Many of these studies are cheaper and quicker than RCTs and may be used to answer questions for which an RCT would be unethical or would itself be low in value.

One approach, called inverse probability weighting (IPW), attempts to approximate the outcome if all subjects were assigned either treatment or non-treatment arm by weighting each subject’s data in a manner that is inversely proportional to the likelihood of assignment to each arm. By inflating the weight of subjects who are under-represented, IPW tries to mitigate the biases due to differential assignment.¹⁰

A recent study by Faridi et al used IPW to assess the outcomes of ad hoc as opposed to delayed PCI in over half a million patients with stable coronary artery disease and found that ad hoc PCI was associated with a lower risk of bleeding but no difference in risk of kidney injury or mortality.¹¹ Unlike RCTs, treatment decisions in reality are far from random; effects attributed to ad hoc as opposed to delayed PCI are confounded by treatment selection bias. Therefore, weights were assigned based on the inverse probability of ad hoc versus delayed intervention, taking into consideration available demographic, procedural, clinical and hospital data. IPW allows minimisation of observed confounding and, despite the potential of unmeasured confounding, may approximate an RCT. Notably, the findings of Faridi et al were in line with findings from RCTs that asked comparable questions.^{12 13} Other techniques also allow the study of potentially causal associations using retrospective data, such as hierarchical regression and more complex techniques such as Bayesian networks and neural networks.

Appropriate employment of causal inference techniques, which have been in existence since the 1990s, is incumbent on the availability of high-resolution data. The pandemic may grant us a large enough sample size with less patient exclusion and greater intensity of exposure over a sufficiently long observation period to conduct such analyses. Faridi et al made use of large amounts of high-dimensional multicentre data that allowed the authors to control for a great number of potentially confounding covariates.^{11 14} We and others have used causal inference approaches with single-centre data¹⁵; however, high-quality data from multiple institutions may allow researchers to draw even stronger conclusions. The global nature of the current pandemic will allow even broader use of such statistical techniques in asking questions about common procedures.

Like all retrospective studies, optimising control for confounding remains a challenge. Researchers will need to identify endpoints that correspond to the particular intervention that was delayed and to tease apart outcomes that may be due to delays or deferrals in other aspects of the patient’s care. The utility of causal inference techniques such as IPW will rely on the validity of assumptions and the inclusion of confounders that are based on sound clinical judgement. Careful (and critical) selection of analytical techniques should also inform studies.¹⁶ Appropriate measures to curate the quality of the data will be necessary: quality in, quality out. It is important as well for researchers, clinicians, and health service executives to carefully define the questions whose answers may best serve their individual healthcare contexts and to assess whether or not they have appropriate data to ask these questions. Furthermore, early multidisciplinary engagement with data scientists and other specialists is critical.

This work carries weight beyond an academic exercise. In the USA, COVID-19 is yet another call to action to address the healthcare disparities that disproportionately harm minority populations.¹⁷ One way to free resources for these groups is to identify and eliminate low-value care. Using the natural experiment that is the pandemic may help us construct a new normal that provides patients more of what they need and less of what they do not.