INTRODUCTION
Predictive analytics in healthcare has generated growing enthusiasm, fueled by the availability of rich patient data from electronic health records and a growing body of published predictive models, such as those predicting hospital readmission and clinical deterioration1. Predictive analytics applies techniques from data mining, statistics, modeling, and artificial intelligence to use data to make predictions about risk1. In healthcare, these approaches offer opportunities to improve clinical outcomes, reduce costs, and support population health2. However, little is known about how healthcare systems organize and manage predictive analytics at the hospital or system level to ensure safe, effective, transparent, and equitable algorithm deployment2.
METHODS
To address this question, we surveyed healthcare leaders with the most local knowledge of predictive analytics activities at all non-academic healthcare system member sites of The Scottsdale Institute (SI), a not-for-profit organization of 60 non-profit healthcare systems committed to sharing best practices, with a particular focus on information technology and innovation3. The survey was developed with input from a representative sample of key stakeholders, emailed by SI leadership to non-academic SI members (with reminders sent at 2 weeks and 1 week prior to survey closing), and completed between 4/13/2021 and 5/172021 using Research Electronic Data Capture (REDCap)4. We examined the association between healthcare system characteristics (i.e., number of system beds [>2000 versus other], and primary population served [urban versus other]) and having a dedicated predictive analytics team using chi-square statistics. A P value < 0.05 denoted statistical significance. The study was approved by the University of Chicago Institutional Review Board.
RESULTS
The response rate was 60% (25/42)5. Respondents were primarily leaders in clinical analytics or medical informatics (Table 1). The healthcare systems represented were diverse in geography and populations served; the majority had 10 or more acute care hospitals and 2000 or more system beds caring for both adults and children (Table 1). Most (16/25, 64%) reported having a team or individual focused on clinical applications of predictive algorithms, with analytics (5/16, 31%), informatics (4/16, 25%), and information technology (IT) (3/16, 19%) leaders most responsible for establishing the programs. Most programs included fewer than nine members (12/16, 75%), have existed for five or fewer years (11/16, 69%), have implemented six or fewer algorithms (11/16, 69%) (with a mix between buying and building algorithms), include diverse roles, and have broad responsibilities (Table 2), although only a minority (6/25, 24%) have a dedicated budget for predictive analytics. Algorithms most commonly focused on identifying risk for acute clinical deterioration or excess healthcare utilization. Healthcare leaders viewed acceptance by clinical teams and the technology needed to integrate algorithms into care as the most significant threats (Table 2). Healthcare systems with predictive analytics teams were not significantly more likely to have more than 2000 hospital beds (44% vs. 56%, p=0.57) or primarily serve urban patients (56% vs. 64%, p=0.51).
Table 1.
Respondent and Hospital Characteristics
| Hospital characteristics | Responses, n (%) (N=25) |
|---|---|
| Respondent | |
| Data Analytics Leader (e.g., CAO) | 11 (44) |
| Medical Informatics Leader (e.g., CMIO, CHIO) | 9 (36) |
| Nursing Informatics Leader (e.g., CNIO) | 2 (8) |
| Chief Information Officer | 2 (8) |
| Other (CMO) | 1 (4) |
| Primary population area served | |
| Suburban | 11 (44) |
| Urban | 9 (36) |
| Rural | 5 (20) |
| Primary patients served | |
| Both adults and children | 14 (56) |
| Just adults | 11 (44) |
| Location of healthcare organization | |
| Northeast | 2 (8) |
| South | 9 (36) |
| Midwest | 9 (36) |
| West | 5 (20) |
| Number of acute care hospitals | |
| 1–3 | 3 (12) |
| 4–5 | 2 (8) |
| 6–9 | 4 (16) |
| 10 or more | 16 (64) |
| Number of staffed beds | |
| 350–500 beds | 1 (4) |
| 501–100 beds | 4 (17) |
| 1001–1500 beds | 6 (24) |
| 1501–2000 beds | 2 (8) |
| >2000 beds | 12 (50) |
Abbreviations: CAO Chief Analytics Officer; CHIO Chief Health Information Officer; CMIO Chief Medical Informatics Officer; CMO Chief Medical Officer; CNIO Chief Nursing Information Officer
Table 2.
Survey Responses
| Survey responses |
Responses,
n
(%) (N=25)* |
|---|---|
| What members of your healthcare organization are on the predictive analytics team? | |
| Data Scientist | 12 (75) |
| Clinical Analytics Expert | 11 (69) |
| Physician | 9 (56) |
| Clinical Informatics Expert | 8 (50) |
| Information Technology Expert | 7 (44) |
| Nurse | 5 (31) |
| Clinical Operations Leader | 3 (19) |
| Process/Quality Improvement Expert | 3 (19) |
| What is the responsibility of the team or individual focused on predictive analytics?*** | |
| Identify algorithms of potential value | 13 (81) |
| Develop algorithms | 11 (69) |
| Facilitate the deployment of predictive algorithms | 16 (100) |
| Evaluate the safety and accuracy of proposed algorithms in the local patient population | 13 (81) |
| Evaluate the safety and accuracy of already implemented algorithms | 10 (63) |
| Other responsibilities | 1 (6) |
| Do you tend to buy or internally build your predictive analytic algorithms?**** | |
| Buy all | 2 (8) |
| Mostly buy | 11 (44) |
| Even between build or buy | 3 (12) |
| Mostly build | 8 (32) |
| Build all | 1 (4) |
| What are the current focus areas of the models deployed in your healthcare system?*** | |
| Sepsis risk/identification | 14 (88) |
| Hospital readmission risk | 14 (88) |
| Inpatient length of stay prediction | 10 (67) |
| Ambulatory no-show prediction | 10 (63) |
| Acute care utilization prediction | 6 (38) |
| Cardiac arrest risk | 5 (31) |
| ICU transfer risk | 4 (25) |
| Other | 9 (57) |
| What do you think are the greatest threats to predictive analytics in healthcare in the next 3–5 years? | |
| Acceptance by clinical teams | 14 (56) |
| Technology integration | 13 (52) |
| Liability issues | 9 (36) |
| Concerns about data sharing with outside vendors | 9 (36) |
| Limited generalizability of algorithms across institutions | 9 (36) |
| Regulatory issues | 7 (28) |
| Cybersecurity | 5 (20) |
| Other | 2 (8) |
*n=25 unless otherwise specified
**n=16 as these questions were only asked to survey respondents who had an individual or team focused on clinical applications of predictive algorithms
***n=16 as these questions were only asked to survey respondents who had an individual or team focused on clinical applications of predictive algorithms. In addition, the respondents could select all answers that correctly answered the survey question
****n=25 as these questions were asked to all survey respondents. In addition, the respondents could select all answers that correctly answered the survey question
DISCUSSION
To our knowledge, this is the first US survey examining how healthcare systems integrate predictive analytics into everyday clinical care. Despite targeting large non-profit healthcare systems focused on informatics and innovation (as indicated by their SI membership), only 64% reported having a team or individual accountable for the clinical application of predictive algorithms. This is notable as many experts have advocated for creating teams and processes to ensure predictive models are safe, effective, equitable, and successfully adopted into clinical practice6. If large healthcare systems are struggling to develop teams and processes to appropriately manage predictive analytics programs, it is likely that smaller healthcare systems will be even more underprepared to safely integrate predictive algorithms into clinical practice
For those organizations that did have established programs, we found that program responsibilities varied from those solely focused on deploying algorithms to others involved in the identification, development, validation, and deployment of predictive models. This inter-hospital variation was not surprising as it reflects the lack of evidence-based frameworks to guide the adoption of predictive algorithms in healthcare. Our study was limited by a small sample; unadjusted statistical analyses; and a focus on larger innovative healthcare systems, which may limit the generalizability of our findings but likely provide the most optimistic estimates for organization-level management of predictive analytics. Future work is needed to assess the management of predictive analytics programs in healthcare to ensure safe algorithm deployment.
Acknowledgements
We thank those who helped refine survey questions and disseminate the survey, including Sachin Shah, MD, John Fahrenbach, Ph.D., Cheng-Kai Kao, MD, Tomas Villanueva DO, MBA, Willian Dardani, and Jodi Wachs, MD. We also thank the healthcare leaders in the Scottdale Institute network for completing the survey. None of those acknowledged were compensated.
Author Contribution
Study concept and design: J. C. R., V. M. A, C. A. U. Acquisition of data: J. C. R., G. R., J. G. Analysis and interpretation of data: all authors. First drafting of the manuscript: J. C. R. Critical revision of the manuscript for important intellectual content: all authors. Statistical analysis: J. C. R., C. A. U. Study supervision: C. A. U, V. M. A. Data access and responsibility: J. C. R., C. A. U. had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Funding
Dr. Arora is supported by a career development award from the National Heart, Lung, and Blood Institute (K24 HL136859). Research Electronic Data Capture (REDCap) at the University of Chicago is hosted by the Center for Research Informatics and funded by the University of Chicago Biological Sciences Division and the Institute for Translational Medicine, CTSA grant number UL1 TR000430 from the National Institutes of Health. No other support was received for this study.
Declarations
Conflict of interest
The authors declare that they do not have a conflict of interest.
Footnotes
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Parikh RB, Kakad M, Bates DW. Integrating predictive analytics into high-value care: the dawn of precision delivery. Jama. 2016;315(7):651–652. doi: 10.1001/jama.2015.19417. [DOI] [PubMed] [Google Scholar]
- 2.Eaneff S, Obermeyer Z, Butte AJ. The case for algorithmic stewardship for artificial intelligence and machine learning technologies. Jama. 2020;324(14):1397–1398. doi: 10.1001/jama.2020.9371. [DOI] [PubMed] [Google Scholar]
- 3.The Scottsdale Institute – The healthcare executive resource for information management. Available at: http://www.scottsdaleinstitute.org/.
- 4.Harris PA, Taylor R, Minor BL, et al. The REDCap consortium: Building an international community of software platform partners. Journal of biomedical informatics. 2019;95:103208. doi: 10.1016/j.jbi.2019.103208. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Sheehan KB. E-mail survey response rates: A review. Journal of computer-mediated communication. 2001;6(2):JCMC621.
- 6.Wiens J, Saria S, Sendak M, et al. Do no harm: a roadmap for responsible machine learning for health care. Nature medicine. 2019;25(9):1337–1340. doi: 10.1038/s41591-019-0548-6. [DOI] [PubMed] [Google Scholar]