INTRODUCTION
Although complex, high-need patients account for the majority of health care spending,1 the use of predictive analytics for pro-active patient management of high-risk populations has been limited. The Veterans Health Administration (VHA) developed the Care Assessment Needs (CAN) score2 to help primary care teams identify high-risk patients. The CAN score reflects clinical and demographic characteristics that predicted future hospitalization and mortality for 4,598,408 VHA primary care patients2 with robust areas under the curve (AUCs) for predicting hospitalization (0.84), death (0.86), and hospitalization and/or deaths (0.82). The original CAN score algorithm had 90 input variables; the current version has 36 variables and has similar predictive accuracy. All VHA primary care providers and teams have access to a dashboard of CAN scores for their patient panels calculated weekly. The CAN score is expressed as a percentile of probabilities ranging from 0 percentile (lowest risk) to 99th percentile (highest risk). In this paper, we describe the population identified by the CAN score report and assess primary care team experience using the CAN score.
METHODS
Demographic and clinical characteristics and VHA utilization were assessed for VHA-enrolled primary care patients with CAN scores in the top quartile (n = 1,718,558) during 2016. We assessed CAN score use per month during a 3-month observation period (2/2016–4/2016), which was tracked electronically. We assessed user acceptability of CAN scores through an online survey of primary care providers and nurses who logged into the CAN score report during the observation period. The 5-item 5-point Likert scales in our survey were based on the 10-item Systems Usability Survey.3 Responses ranged from strongly agree to strongly disagree with items assessing CAN score accuracy, frequency, and usability (Table 2). We calculated item frequencies and conducted a retrospective content analysis of the survey’s open-text comment field.
Table 2.
Responses from n = 400 primary care personnel regarding CAN report
| Question regarding usability of CAN report | Agree or strongly agree | Neutral | Disagree or strongly disagree |
|---|---|---|---|
| Do you think the CAN score accurately represents your patient population? N (%) | 261 (69%) | 72 (19%) | 41 (10%) |
| I use the CAN score report regularly. N (%) | 252 (68%) | 70 (19%) | 52 (14%) |
| It’s easy to use the CAN report. N (%) | 279 (79%) | 47 (12%) | 33 (9%) |
| I find the CAN score helpful. N (%) | 277 (73%) | 66 (18%) | 33 (9%) |
| I feel confident in my ability to use the CAN report for clinical practice. N (%) | 269 (72%) | 73 (20%) | 32 (9%) |
RESULTS
The CAN algorithm identifies a high-risk population with multiple comorbidities at high risk for poor outcomes (Table 1). We found that there was wide variability in use of CAN scores among 8650 primary care providers and 6433 nurse care managers. During the three survey months, there were an average of 6450 uses of the CAN report among an average of 1850 individual users. Because surveys were anonymous, we were unable to count the number of unique respondents from the n = 400 responses we received. A majority of respondents reported that they use the score regularly (68%), that the score accurately represented their patient population (69%), and that they were confident in their ability to use the CAN score for clinical practice (72%) (Table 2). Qualitative responses indicated CAN scores were often used for care coordination or palliative care services. Respondents also indicated a need for additional help using the score, and for the CAN system to be more interactive and more fully integrated in the electronic health record.
Table 1.
Patient demographics, comorbidities (identified by ICD-9 codes), and VHA utilization among Veterans enrolled in primary care in 2016, by CAN score percentile
| Top 1% N = 18,056 |
Top 5% N = 100,795 |
Top 10% N = 179,520 |
Top 25% N = 1,718,558 |
|
|---|---|---|---|---|
| Count (%) of deaths during FY2016 | 11,121 (61.6) | 31,630 (31.4) | 30,327 (16.9) | 81,514 (4.7) |
| Mean probability of death within a year | 0.52 | 0.26 | 0.13 | 0.04 |
| Mean diagnosis count | 5.3 | 3.9 | 2.8 | 2.0 |
| Mean age | 82.8 | 81.6 | 80.4 | 73.8 |
| Men (%) | 99.1 | 98.8 | 98.7 | 97.9 |
| Hypertension (%) | 75.6 | 69.8 | 64.5 | 62.2 |
| COPD (%) | 53.9 | 39.7 | 29.2 | 20.0 |
| Diabetes (%) | 41.9 | 41.3 | 37.4 | 35.4 |
| Dementia (%) | 28.2 | 22.4 | 13.2 | 4.0 |
| Depression (%) | 28.2 | 21.3 | 16.2 | 16.3 |
| Mean body mass index (BMI) | 27.6 | 28.3 | 28.9 | 30.5 |
| Mean hospitalization in a year | 1.7 | 1.0 | 0.6 | 0.3 |
| Mean mental health visits/year | 2.1 | 2.9 | 3.3 | 5.4 |
| Mean primary care visits/year | 8.8 | 9.6 | 8.7 | 8.4 |
| Mean emergency visits/year | 2.1 | 1.1 | 0.7 | 0.4 |
DISCUSSION
Since 2012, VHA has used a predictive model to identify patients at highest risk for hospitalization and death for primary care teams. Among clinicians who used the CAN report, most believed that the CAN score accurately identified their high-risk patients, and that they used the score regularly. Comments indicated, however, similarly to other studies of predictive analytics,4 that CAN score results were not always clinically actionable. The limited number of report users compared with potential users illustrates the challenge of integrating a risk score into clinical practice workflow.
The results we report are preliminary, and limited by our low survey response rate. Our responses may be biased in favor of clinicians with greater CAN score familiarity. Nonetheless, our results provide a starting point for understanding and maximizing the clinical utility of predictive indexes in routine practice. The CAN score is being applied in a number of quality improvement projects and research studies with the goal of learning not only the effectiveness of intensive management for these patients but also the clinical processes needed to make use of CAN score information.5, 6 The CAN score illustrates how a health care system can harness electronic health record data to identify high-risk patients who may benefit from clinical interventions. Additional work is needed to fully characterize the heterogeneity of these patients’ needs, and to determine the best methods for using the score to target patients in need of specific services.
Acknowledgements
Data for this report were developed by the national evaluation team at the PACT Demonstration Lab Coordinating Center and the VHA Office of Clinical Systems Development & Evaluation. We would like to thank Idamay Curtis, Rebecca Piegari, and Sophie Lo for their assistance with data acquisition and administrative support. Dr. Nelson had full access to all study data and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Compliance with Ethical Standards
Conflict of Interest
The authors declare that they do not have a conflict of interest.
Footnotes
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Yoon J, Scott JY, Phibbs CS, Wagner TH. Recent trends in veterans affairs chronic condition spending. Population health management. 2011;14(6):293–8. doi: 10.1089/pop.2010.0079. [DOI] [PubMed] [Google Scholar]
- 2.Wang L, Porter B, Maynard C, Evans G, Bryson C, Sun H, et al. Predicting risk of hospitalization or death among patients receiving primary care in the Veterans Health Administration. Med Care. 2013;51(4):368–73. doi: 10.1097/MLR.0b013e31827da95a. [DOI] [PubMed] [Google Scholar]
- 3.Brooke JSUS, et al. A “quick and dirty” usability scale. In: Jordan PW, et al., editors. Usability Evaluation in Industry. London, UK: Taylor & Francis; 1996. pp. 189–194. [Google Scholar]
- 4.Amarasingham R, Patzer RE, Huesch M, Nguyen NQ, Xie B. Implementing electronic health care predictive analytics: considerations and challenges. Health Aff (Millwood). 2014;33(7):1148–54. doi: 10.1377/hlthaff.2014.0352. [DOI] [PubMed] [Google Scholar]
- 5.Zulman DM, Pal Chee C, Ezeji-Okoye SC, et al. Effect of an intensive outpatient program to augment primary care for high-need veterans affairs patients: A randomized clinical trial. JAMA internal medicine. 2017;177(2):166–75. doi: 10.1001/jamainternmed.2016.8021. [DOI] [PubMed] [Google Scholar]
- 6.Yoon J, Chang E, Rubenstein LV, Park A, Zulman DM, Stockdale S, et al. Impact of Primary Care Intensive Management on High-Risk Veterans’ Costs and Utilization: A Randomized Quality Improvement Trial. Ann Intern Med. 2018;168:846–854. doi: 10.7326/M17-3039. [DOI] [PubMed] [Google Scholar]