Economic evaluations of big data analytics for clinical decision-making: a scoping review

Abstract

Objective

Much has been invested in big data analytics to improve health and reduce costs. However, it is unknown whether these investments have achieved the desired goals. We performed a scoping review to determine the health and economic impact of big data analytics for clinical decision-making.

Materials and Methods

We searched Medline, Embase, Web of Science and the National Health Services Economic Evaluations Database for relevant articles. We included peer-reviewed papers that report the health economic impact of analytics that assist clinical decision-making. We extracted the economic methods and estimated impact and also assessed the quality of the methods used. In addition, we estimated how many studies assessed “big data analytics” based on a broad definition of this term.

Results

The search yielded 12 133 papers but only 71 studies fulfilled all eligibility criteria. Only a few papers were full economic evaluations; many were performed during development. Papers frequently reported savings for healthcare payers but only 20% also included costs of analytics. Twenty studies examined “big data analytics” and only 7 reported both cost-savings and better outcomes.

Discussion

The promised potential of big data is not yet reflected in the literature, partly since only a few full and properly performed economic evaluations have been published. This and the lack of a clear definition of “big data” limit policy makers and healthcare professionals from determining which big data initiatives are worth implementing.

INTRODUCTION

Extracting valuable knowledge from big healthcare data has been an important aim of many research endeavors and commercial entities. While no clear definition for big data is available, they are often described according to their complexity and the characteristics of the data such as the size of a dataset (Volume), the speed with which data is retrieved (Velocity) and the fact that the data come from many different sources (Variety).¹ Bates et al² emphasize that big data comprise both the data with their large volume, variety, and velocity, as well as the use of analytics. In this respect, analytics are the “discovery and communication of patterns in data.”

Big data’s potential to assist clinical decision-making has been expressed for a variety of clinical fields such as intensive care,^3,4 emergency department,^2,5 cardiovascular diseases,^6,7 dementia,⁸ diabetes,⁹ oncology,^10–12 and asthma.¹³ Big data analytics could also lead to economic benefits.^1,2,14–17 Annual savings for the United States (US) healthcare system of providing timely, personalized care have been estimated to exceed US$140 billion.¹⁸

Over the years, much has been invested to achieve the promised benefits of big data. For instance, the US has invested millions in their Big Data to Knowledge centers.¹⁹ While in Europe, many calls and projects in Europe’s Horizon 2020 program have focused on the use of Big Data for better healthcare (ie, AEGLE, OACTIVE, BigMedylitics). In 2018, US$290 million was allocated to The All of Us initiative which aims to personalize care using a wide variety of data sources (ie, genomic data, monitoring data, electronic health record data) from 1 million US citizens.²⁰ The investments by governments are far exceeded by the investments in “big data technologies” in the commercial sector.²¹ For example, IBM has already invested billions of dollars in “Dr. Watson” and big data analytics,²² and Roche purchased FlatIron Health for US$1.9 billion in 2018.¹¹

For optimal spending of scarce resources, economic evaluations can be used to assess the (potential) return on investment of novel technologies. Economic evaluations are comparative analyses of the costs and consequences of alternative courses of action.²³ Economic evaluations that provide evidence on the health and economic impact of a technology can assist decision-making and justify further investments required to achieve a technology’s potential. Despite the promise that big data analytics can lead to savings, it is unclear whether this promise is corroborated by good evidence. Therefore, we aimed to determine the health and economic impact of big data analytics to support clinical decision-making. Given the absence of a clear definition for big data, we first determined how analytics impacted clinical practice. We then considered which of these analytics could be classified as big data analytics.

MATERIALS AND METHODS

The study follows the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) Checklist.²⁴

Search strategy and study inclusion

Since there is no consensus on the definition of big data,¹ we widened the scope of our search to identify economic evaluations of a variety of analytics. An information specialist from the Academic Library at the Erasmus University Medical Centre was consulted when developing the search strategy (Supplementary Appendix A). In the search strategy, we included MesH and title/abstract terms related to (big data) analytics, economic evaluations, and healthcare. These terms for (big data) analytics included artificial intelligence, tools used to extract patterns from big data, such as machine learning, and generic tools that use analytics to enable decision-making, such as clinical decision support. We combined these with terms such as economic evaluations and cost-effectiveness and terms to exclude studies that had no relation to healthcare (ie, veterinary care).

All major databases were searched (Embase, Medline, Web of Science, and the NHS Economic Evaluations Database). We included all English, peer-reviewed, primary research papers and limited our search to studies of humans. The primary search was performed in March 2018 and updated in December 2019. Initial screening was performed by 1 author (LB). Hereafter, all studies about which there was uncertainty regarding their inclusion were discussed with 2 other authors (JA, WR). Studies were included if they met the following criteria: a) the study reported pattern discovery, interpretation, and communication to assist decision-making of clinical experts at the individual patient level; b) the study implemented analytics in clinical practice using computerized technology; and c) the study reported a monetary estimation of the potential impact of the analytics. Application of these 3 criteria led to the exclusion of studies that only reported time or computation savings and studies that did not assist clinical experts at the individual patient level. Thus, we did not include studies that informed guidelines or policy makers. We also excluded analytics that produced results that could be easily printed on paper for use in clinical practice (ie, Ottawa Ankle Rules) and studies that simply used data mining technologies to extract records from an electronic health record (EHR) but not to perform any analyses of the extracted data.

Data extraction

Data extraction was performed by 1 author (LB). For a random 10% of papers, data was extracted by a second author (WR) to check for concordance. In the end, there were no significant differences in the results. We extracted the following data for each study: patient population, description of the technology in which the analytics are embedded (ie, clinical decision support systems); the analytics used for discovery and communication of patterns in data; description of the data; the intervention and the comparator in the economic evaluation; the perspective, outcomes, and costs included; and results, recommendations, and conflicts of interest. Conflicts of interest included those reported in the paper (related and unrelated), commercial employment of authors, and funding by industry.

We also reported the type of economic evaluation (ie, full or partial) that was used. A full economic evaluation compares 2 or more alternatives and includes both costs and consequences. Partial economic evaluations do not contain a comparison or exclude either costs or consequences.²³ Thus, when a study reported cost estimates but no health outcomes, they were classified as partial. For full economic evaluations, we reported the ratio of costs over effects, also known as the incremental cost-effectiveness ratio. Furthermore, economic evaluations can offer valuable insights for decision-makers at many different stages in the development process (ie, during and after development).²⁵ After development, they can assist healthcare payers when choosing novel technologies in which to invest their constrained budget. During development, an “early” economic evaluation can assist developers by identifying minimal requirements of the technology, areas for further research, and viable exploitation and market access strategies.^25–27

In our results, we also distinguished in which stage of development the economic evaluation was performed. If a study provided recommendations for developing a technology that did not exist, it was categorized as “before” development. Studies were categorized as “during” development when the economic evaluation was performed and presented alongside development unless the aim of the study was to inform purchasing decisions of funding bodies (ie, perspective of the National Healthcare Services) or when the analytics were already implemented in clinical practice. All remaining studies were categorized as being performed “after” development.

We also performed an analysis to identify economic evaluations that might be classified as “big data analytics”. We used broad criteria to select the highest possible number of papers to sketch a best-case scenario. We defined these criteria based on the volume, variety, and velocity of the data. We classified papers as having big volume when they utilized next generation sequencing data, EHR records or claims data with a sample size of more than 100 000 units (ie, patients, admissions), and all imaging papers published after 2013. Papers were included because of their variety when they combined multiple data types (ie, structured and unstructured data, combining multiple data sources). All papers that used monitoring data published after 2013 were included because they might fulfil the velocity criteria.

RESULTS

The initial search yielded 12 133 records of which 71 papers were included in the final analysis after title/abstract and full-text screening (Figure 1). Important exclusion criteria for full-text papers were that no monetary estimates were included and that no analytics were used.

Figure 1.

PRISMA flowchart.

Summary of papers

We found that all papers could be classified into 4 categories according to the type of data that was used: medical history databases (ie, data from EHRs, clinical trial databases, claims databases), imaging data, monitoring data (ie, continuous data collection using sensors), and omics data (ie, proteomics, genomics, transcriptomics, metabolomics) (Table 1). Almost all papers originated from North America and Europe (87%). The US was well represented with 39 papers mainly focusing on the use of medical history and omics data. The number of papers originating from Europe was considerably lower (n = 20), while few or no papers originated from South America, Australia, and Africa. There has been a clear increase in the number of publications from 2016 onwards (Figure 2).

Table 1.

Summary of all records according to data type used

	Total	Medical history	Imaging	Monitoring	Omics
Total	71	44	8	8	11
Continent
North America	42	27	3	3	9
Europe	20	11	2	5	2
Asia	7	5	2	–	–
Africa	1	–	1	–	–
South America	1	1	–	–	–
Australia	–	–	–	–	–
Type of economic evaluation
Full	22	8	5	2	7
Partial	49	36	3	6	4
Perspective
Payer perspective	7	3	–	–	4
National healthcare system	8	3	1	2	2
Provider perspective	3	1	–	1	–
Other	2	–	2	–	–
No perspective reported	52	37	5	5	5
Stage of development
Before development	1	1	–	–	–
During development	33	31	2	–	–
After development	37	12	6	8	11
Measure of effectiveness
QALYs and Life Years	15	5	4	2	4
Model Performance	29	27	2	–	–
Other	20	10	2	3	5
Not included	7	2	–	3	2
Incremental health effects
Decrease in effects	5	2	1	–	2
No difference	5	3	–	2	–
Increase in effects	41	23	7	4	7
Not included	20	16		2	2
Incremental costs
Savings	54	39	5	5	5
No difference	5	2	–	3	–
Increase in costs	12	3	3	–	6
Include costs of implementing analytics	22	2	5	5	10
Recommendations for research & development
Focus development on improving the analytics	30	23	2	2	3
Validation and feasibility of implementation	19	13	3	2	1
Development for other clinical areas or subgroups	11	8	2	1	–
Pricing and economics of the analytics	9	2	3	–	4
Cost-effectiveness research	5	4	–	1	–
Development of the intervention that follows	3	3	–	–	–
Multidisciplinary collaboration	2	2	–	–	–
Refer to big data in the text	6	6	–	–	–
Potential to be classified as big data analytics	20	8	5	4	3

Abbreviations: QALYs, quality adjusted life years.

Figure 2.

Number of publications according to the year of publication.

Most studies were partial economic evaluations and found that analytics may improve outcomes and generate savings. A perspective was not often reported, and no study reported a societal perspective. Almost all partial economic evaluations reported savings compared to half of the studies reporting results from full economic evaluations. When grouped according to conflict of interest, no significant differences were found in the percentage of studies that reported savings and improved health. For economic evaluations without a conflict of interest, 61% were performed during development compared to 22% with no conflict of interest. All but 1 reported savings.

In the following paragraphs we will discuss economic results for all 4 data types. An overview of the economic results for all papers can be found in Supplementary Appendix B. A detailed description of all analytics and data used can be found in Supplementary Appendix C.

Analytics for medical history data

The first category consisted of studies that used historic databases containing information on patient demographics and medical history (ie, test results and drug prescriptions) (n = 44).^28–71 All papers presented predictive or prescriptive analytics that assist clinical decision-making using a variety of techniques, such as regression, support vector machines, and Markov decision processes. The risk of readmission (n = 9) and problems pertaining to the emergency department (n = 5) were most often examined and 1 study addressed pediatric care.⁴² Both structured data, such as demographics and laboratory results, and unstructured data, such as free text messages (n = 4),^37,40,44,50 were used and the sample size varied from N = 80 patients⁶⁵ to more than 800 000 urine samples.⁶⁸ This was the only category in which authors referred to the term “big data” (n = 6).^{32,35,37,40,50,60}

Most of the studies in this category were partial economic evaluations (n = 36) and most were conducted during development (n = 31). Results were often limited to model performance (ie, classification accuracy, area under the curve) and were rarely translated into health benefits such as quality-adjusted life-years. Almost all studies found that the analytics could lead to monetary savings, yet only 2 papers included implementation costs of the analytics.^33,62 These costs could, for instance, consist of licensing costs and costs of implementing analytics within a hospital system. Authors often recommended to continue development and focus on improving the analytics. Furthermore, the need for further validation prior to implementation was frequently emphasized.

Analytics for imaging data

Eight studies presented predictive analytics for 7 different types of imaging data (CT, MRI, Chest radiographs, digital cervical smears, mammographies, digital photographs, and ventilation-perfusion lung scans).^72–79 The number of full economic evaluations^{73,75–77,79} and studies performed after development^73–76,78 were both higher than the first group of papers that used medical history data. Four studies measured effects in (quality-adjusted) life-years,^73,75–77 and more than half of the studies included implementation costs of analytics.^73–77 The number of studies that found the analytics could lead to cost-savings was once again quite high (63%).^{72–74,78,79} Just like the studies that used medical history data, authors of studies in this category emphasized the need for further validation prior to implementation. However, several studies also emphasized the balance between the requirements of the technologies (ie, test sensitivity) and potential health benefits and cost-savings.^75,76,79

Analytics for monitoring data

Monitoring data collected with a variety of devices and sensors (ie, airflow monitoring, continuous glucose monitoring, continuous performance tests, infrared cameras, vital signs monitors) was used in 8 studies.^80–87 Five of these studies reported descriptive analytics that monitored patient outcomes and compared this to a range or reference value.^{81,83–85,87} This group of papers differed from those using imaging and medical history data since most analytics were implemented in a medical device. All technologies were evaluated after development, of which many were partial economic evaluations. Roughly half of the studies resulted in more effects^82–84,86 and savings^82,84–87 and included costs of the device and/or analytics.^81,86,87

Analytics for omics data

Eleven papers reported the potential impact of predictive and prescriptive analytics of omics data, often with the aim of applying them as a test in clinical practice.^88–98 Only 2 of these papers focused on the use of Next Generation Sequencing data^94,96 and 1 paper combined multiple types of data (pharmacogenomics, literature, medical history).⁸⁹ The remaining papers utilized microarray data, and all the analytics that were adopted as a test were used in oncology (n = 9).^{88,90–93,95–98}

Compared to the other categories, the percentage of full economic evaluations was high.^{90,92,93,95–98} In half of the studies, the perspective used was that of the payer or the healthcare system. Furthermore, just like the studies that used monitoring data, all economic evaluations were performed after development. Seven studies reported increased effects ,^{88,90–93,96,97} and 6 studies reported that use of analytics would increase costs.^{90,93–95,97,98} All but 1 study included the costs of the analytics or the test in which the analytics were implemented.⁸⁹ Moreover, unlike the other categories, several papers discussed price thresholds at which the analytics or the test would be cost-neutral or dominant (ie, more effects and lower costs) or thresholds at which the analytics or test would be cost-effective (ie, where the incremental cost-effectiveness ratio would be below a specific cost-effectiveness threshold).

Big data analytics

We found that less than a third of all papers (n = 20) might fulfil criteria for classification as “big data analytics” (Table 2). Most papers were included because their volume might be large enough to be considered big data (ie, N > 100 000, imaging data) and studies that used monitoring data were included because of the potential speed with which the data is collected (velocity). Eight of these papers used medical history data,^{32,37,40,44,45,50,60,68} 5 used imaging data,^{72–74,76,78} 4 used monitoring data,^80,82,83,87 and 3 used omics data.^89,94,44,96 Most were partial economic evaluations (n = 15) and 12 were performed after development. All but 5^{44,76,80,83,94} corroborated expectations that big data analytics could result in cost-savings, varying from US$126 per patient⁸⁹ to more than US$500 million for the entire US healthcare system.⁷² However, only a handful of papers included the costs of the analytics.^{73,74,76,87,94,96}

Table 2.

Classification of papers that could be defined as “big data” studies based on the criteria of volume, velocity, and variety. These papers represent a subset of the initial 71 papers

	Volume			Velocity	Variety
Article	Next generation sequencing data	Medical history data with N > 100,000	Imaging data published after 2013	Monitoring data published after 2013	Combines multiple data types
Burton (2019)		X
Duggal (2016)					X
Golas (2018)					X
Hunter-Zinck (2019)		X			X
Jamei (2017)		X
Lee (2015)		X			X
Rider (2019)		X
Wang (2019)		X
Carballido-Gamio (2019)			X
Crespo (2019)			X
Philipsen (2015)			X
Sato (2014)			X
Sreekumari (2019)			X
Brocklehurst (2018)				X
Calvert (2017)				X
Hollis (2018)				X
Sánchez-Quiroga (2018)				X
Brixner (2016)					X
Mathias (2016)	X
Nicholson (2019)	X
Total	2	6	5	4	5

DISCUSSION

In this review, we aimed to determine the health and economic impact of big data analytics for clinical decision-making. We found that expectations of big data analytics with respect to savings and health benefits are not yet reflected in the academic literature. Most studies are partial economic evaluations and the costs of implementing analytics are scarcely included in the calculations. To ensure optimal decision-making, guidelines recommend a full economic evaluation that includes all relevant costs for payers (ie, costs of analytics). Our results align with earlier research noting deployment costs are rarely considered while these costs can be a major barrier to successfully implementing analytics.⁹⁹

We found that a small subset might be classified as big data analytics. We adopted a broad definition of big data to maximize the number of studies that would be considered as studies of big data. Therefore, the actual number of studies would be even lower if papers were to be assessed by a panel of experts. This corroborates a previous study from 2018 which found that quantified benefits of big data analytics are scarce.¹

The studies were grouped into 4 categories according to the data sources used, which were similar to those reported by Mehta et al.¹ Two main differences were that we grouped all databases that reported information relating to a patient’s medical history (instead of separating claims and EHR data) and we included a category that evaluated analytics for monitoring data generated in the hospital. This category was not available in the classification used by Mehta et al. However, they reported some categories (ie, social media and wearable sensors) that are not yet represented in the literature on economic evaluations. None of the studies evaluated technologies that used patient-generated data collected using different methods such as healthcare trackers.

Recommendations for future economic evaluations

Good policy making decisions about the use of analytics requires knowledge of the impact that the analytics will have on costs and health outcomes. With this in mind, policy makers could provide incentives to developers of analytics to perform good-quality economic evaluations. Economic evaluations of analytics are still scarce and the studies that were available often did not adhere to best-practice guidelines, thereby limiting their value to inform decision-making. Often a partial instead of a full economic evaluation was performed, costs of purchasing and implementing the analytics were excluded, or only intermediate outcomes were reported. For payers and policy makers, excluding for instance the costs of the analytics could result in an underestimation of the investment needed to implement the technology or an overestimation of its financial benefits. By means of incentives, policy makers could stimulate developers to adhere to guidelines and best practice recommendations (ie, Drummond,²³ Buisman,²⁶ Morse⁹⁹) This could improve the quality of results and thus their ability to inform decision-making.

We found a relatively high number of studies that performed an economic evaluation of analytics during development compared to other fields (ie, drug or medical device development).^100,101 A possible explanation for this is the high cost of validating and deploying analytics, known important barriers to implementation.^11,99 Few artificial intelligence and big data analytics solutions have been implemented successfully.^3,11 To overcome this challenge, Frohlich et al recommend the use of pilot trials to illustrate the potential effectiveness and efficiency of analytics. These results can then be used to find new investors for clinical research.¹¹ In our results, we also saw that those without a conflict of interest (ie, academia) were more inclined to publish during development which might be explained by the need to attract funding for further development.

Defining big data to assist evaluation

Without consensus on a definition, no objective assessment can be made as to whether investments following the introduction of big data in healthcare have realized expectations, whether they can be considered good value for money, and whether future investments should be stimulated. In our analysis, we found that it likely that a small number of studies have performed an economic evaluation of big data analytics. However, this absolute number is uncertain since a clear definition of “big data” is still lacking almost 10 years after its introduction in healthcare. For policy makers and those who wish to practice evidence-based medicine, it is essential to know where and how big data analytics would result in health and financial benefits before investing in products described in mainstream media as “big data” technologies (ie, Afirma GSC, YouScript).^102,103 This remains a challenging task if there is no consensus on its definition. Therefore, we recommend experts in the field reconsider the possibility of generating a quantitative definition of big data in healthcare.

Defining big data is no easy task, and we think that a definition will only be accepted by the healthcare field if it is developed by a multidisciplinary collaboration of experts from academia, healthcare organizations, insurers, federal entities, policy makers, and commercial parties. Many authors have described the term in slightly different words,^1,104 some have tried to quantify,¹⁰⁵ and others have purposefully refrained from doing so.¹⁴ Auffray et al¹⁴ stated in 2016 that a single definition of big data would probably be “too abstract to be useful” and proposed the use of a workable definition in which big data covers the high volume and diversity of data sources managed with best-practice technologies such as advanced analytics solutions. However, descriptions such as “best-practice,” “advanced,”¹⁴ or “traditional”¹⁰⁶ are time-dependent. What is “traditional” in 2014 is not necessarily “traditional” in 2020. Thus, perhaps a definition of big data should quantify the “data” element, include a concrete list of analytics that are considered advanced or best practice, be time-dependent, and be updated regularly. We recognize that it might be extremely difficult to achieve wide consensus and we do not think this can be realized without support from academic, clinical, policy, federal, and commercial stakeholders.

Limitations

One limitation of our research is that economic evaluations do not always describe the analytics element of the intervention that was being evaluated. For instance, in studies of omics data, the papers generally referred to the tool (ie, Afirma GSC) but did not describe the analytics used in this tool. One way to ensure that economic evaluations that assess a big data technology are included in future reviews would be to specify explicit tools that might contain big data analytics (ie, Afirma GSC) for each data type in a search strategy. However, such a list is likely to be very long, and this will also be challenging without a definition of big data. Research into the economic value of big data analytics might also be facilitated by better reporting in economic evaluations on the data and analytics used for development. Another limitation is that studies that did not refer to cost estimations in their title/abstract were excluded. This could have led to exclusion of studies that perform a cost estimation but do not report this as a primary outcome in the abstract. A possible solution for future research would be to include studies for full-text screening when 1 of the authors is a health economist or employed in a health policy or economics department.

Also, since our review included only published economic evaluations, it is possible that our results are influenced by the absence of an incentive to submit an academic paper and by publication bias. Commercial developers do not always have an incentive to publish but do have an incentive to market their products using the results of economic analyses. If these studies do not include costs of analytics in their estimation of benefits, this would only underline the importance of our recommendations. It is also possible that studies that do not find a technology cost-effective include costs of analytics more often and are rejected for publication because of negative results.

Methodological limitations were that study selection and data extraction were performed by a single reviewer due to the size of the hits from the search strategy and the fact that Business Review Complete (BSC) was not included in the literature search. While this may have resulted in the exclusion of some relevant studies, we expect this number to be small. Moreover, this does not affect the conclusions of our study. Our search was limited to analytics for decision-making of clinical experts at the individual patient level. There are many other ways in which analytics could improve health, such as managing epidemics and policy making to improve population health that were beyond the scope of this article. To conclude, it is possible that developers sometimes have a valid reason for not including costs of analytics which we did not consider in this study.

CONCLUSION

This is the first study to assess the health and economic impact of big data analytics for clinical decision-making. At present the potential benefits of big data analytics for clinical practice cannot yet be corroborated with academic literature despite high expectations. We found that economic evaluations were sometimes used to estimate the potential of analytics. However, many studies were partial economic evaluations and did not include costs of implementing analytics. Therefore, economic evaluations that adhere to best-practice guidelines should be encouraged. This and the lack of an appropriate definition of big data complicate justification of future expenses and makes it exceedingly difficult to determine whether expectations of big data analytics have thus far been realized. Therefore, we recommend key experts in the field of data science in healthcare reconsider the possibility of defining big data analytics for healthcare.

FUNDING

This work was supported by the European Union’s Horizon 2020 Research and Innovation Programme grant number 644906.

AUTHOR CONTRIBUTIONS

LB contributed to developing the research question, the search strategy, execution of the search, performed study selection, data extraction, and revision of the manuscript. JA contributed to developing the research question, assisted in study selection and data extraction, and revision of the manuscript. CUG contributed to the conceptual design of the research question and study and revision of the manuscript. WR contributed to developing the research question, assisted in study selection and data extraction, and revision of the manuscript. All authors (LB, JA, CUG, and WR) approved the final version for publication.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.