Abstract
BACKGROUND
Improving the ability to risk-stratify patients is critical for efficiently allocating resources within healthcare systems.
OBJECTIVE
The purpose of this study was to evaluate a physician-defined complexity prediction model against outpatient Charlson score (OCS) and a commercial risk predictor (CRP).
DESIGN
Using a cohort in which primary care physicians reviewed 4302 of their adult patients, we developed a predictive model for estimated physician-defined complexity (ePDC) and categorized our population using ePDC, OCS and CRP.
PARTICIPANTS
143,372 primary care patients in a practice-based research network participated in the study.
MAIN MEASURES
For all patients categorized as complex in 2007 by one or more risk-stratification method, we calculated the percentage of total person time from 2008–2011 for which eligible cancer screening was incomplete, HbA1c was ≥ 9 %, and LDL was ≥ 130 mg/dl (in patients with cardiovascular disease). We also calculated the number of emergency department (ED) visits and hospital admissions per person year (ppy).
KEY RESULTS
There was modest agreement among individuals classified as complex using ePDC compared with OCS (36.7 %) and CRP (39.6 %). Over 4 follow-up years, eligible ePDC-complex patients had higher proportions (p < 0.001) of time with: incomplete cervical (17.8 % vs. 13.3 % for OCS; 19.4 % vs. 11.2 % for CRP), breast (21.4 % vs. 14.9 % for OCS; 22.7 % vs. 15.0 % for CRP), and colon (25.9 % vs. 18.7 % for OCS; 27.0 % vs. 18.2 % for CRP) cancer screening; HbA1c ≥ 9 % (15.6 % vs. 8.1 % for OCS; 15.9 % vs. 6.9 % for CRP); and LDL ≥ 130 mg/dl (12.4 % vs. 7.9 % for OCS; 11.8 % vs 9.0 % for CRP). ePDC-complex patients had higher rates (p < 0.003) of: ED visits (0.21 vs. 0.11 ppy for OCS; 0.17 vs. 0.15 ppy for CRP), and admissions in patients 45–64 and ≥ 65 years old (0.11 vs. 0.10 ppy AND 0.24 vs. 0.21 ppy for OCS).
CONCLUSION
Our measure for estimated physician-defined complexity compared favorably to commonly used risk-prediction approaches in identifying future suboptimal quality and utilization outcomes.
Electronic supplementary material
The online version of this article (doi:10.1007/s11606-015-3357-8) contains supplementary material, which is available to authorized users.
INTRODUCTION
The US healthcare system must address the needs of an aging and increasingly chronically ill population in the setting of unsustainable healthcare spending. Complex patients, often burdened by multiple chronic conditions and psychosocial issues, have more frequent interactions with healthcare systems and are at higher risk for poor health outcomes and avoidable acute care utilization.1–4 Despite advancements in team-based and non-visit-based approaches to care, the primary care teams that bear disproportionate responsibility for the management of these patients remain overstretched and under-resourced to address their needs. Given the concentration of health expenditures in a subset of complex patients,5 improving the efficiency of care delivery to this group may help contain costs.6–12
Identifying complex, high-risk primary care patients is an important first step that allows healthcare systems to better allocate resources and target interventions. Current quantitative methods for identifying the complex patients at highest risk for suboptimal future clinical quality and utilization outcomes rely primarily on diagnosis-based and utilization-based algorithms to predict future utilization.13–23 These tools miss clinical characteristics that are not present in billing data and may not capture non-clinical contributors to patient complexity.
We previously characterized patient complexity from the physician’s perspective by having primary care physicians (PCPs) review lists of their own patients and identify patients they considered complex. Patients identified as complex by these PCPs had only modest overlap with patients identified as complex using traditional risk-stratification methods.24,25 Clinicians consider medical, behavioral, and socioeconomic complexity domains when identifying complex patients;24 however, asking clinicians to assess each of their patients is impractical, and current population-level approaches generally do not take this comprehensive view. Approximating empiric physician assessment of patient complexity using quantitative approaches might serve as a useful proxy and allow population-level identification of high-risk patients that also require significant effort for primary care teams to manage.
In this analysis, we: 1) developed a predictive model for a physician’s qualitative assessment of their patients’ complexity [estimated physician-defined complexity (ePDC)], and 2) evaluated how patients identified as complex by ePDC compared with those identified by two commonly used risk-stratification approaches in a cohort of primary care patients followed over 4 years. We hypothesized that ePDC would more effectively identify patients with suboptimal outpatient clinical quality and acute care utilization compared to established risk-stratification approaches.
METHODS
Patients and Setting
We conducted analyses in a cohort of 143,372 adult patients receiving primary care within the Massachusetts General Hospital Primary Care Practice-based Research Network from 2005–2011. We used 3 years of data for predictive model development (2005–2007) and followed patient outcomes over the subsequent 4 years (2008–2011). We obtained all cohort data from an electronic data repository containing demographic, clinical, appointment, and billing data.
Developing and Validating a Predictive Model for Estimated Physician-Defined Complexity
We used an established cohort in which PCPs reviewed 4302 of their own randomly selected adult patients and subjectively identified 1126 patients (26 %) as complex in response to the question, “In your view, do you consider this patient a ‘complex patient’?”24,25 We derived candidate variables (eTable 2) for modeling from demographic, diagnostic, procedure, medication, laboratory, and prior primary care and acute care utilization data. Using data from 2005–2007, we randomly split this cohort into two subsets: two-thirds for model development and one-third for model validation. From the development subset, we generated 1000 bootstrap samples. For each sample, we ran separate logistic regression models with backward elimination (significance at the 0.05 level) to identify predictors for ePDC, stratifying our models by patient age (<45, 45–64, and ≥65 years) to account for differences in the relative contribution of different complexity domains by age category.24,25 We included only those variables chosen in greater than 20 % of bootstrap samples. We evaluated model discrimination using c-statistics and calculated model test characteristics: specificity, sensitivity, and positive and negative predictive value. C-statistics were > 0.76 for all models (0.82 for < 45 and 45–64 years and 0.77 for ≥ 65 years). Test characteristics and information on the final, age-stratified predictive models are available online (eTables 1&2). For our analysis, we selected the model cutoff for ePDC with the highest overall accuracy (e.g., agreement between model prediction and PCP qualitative assessment).
Applying Complexity Measures to the Population
We compared ePDC to outpatient Charlson score26 (OCS) and a proprietary, commercial risk predictor (CRP) used at our institution. We used 3 years of ICD-9 diagnosis codes from outpatient visits, procedures codes, problem lists, laboratory and medication data to define conditions and generate an OCS for each patient. The CRP employs a methodology based on Episode Treatment Groups27,28 and 12 months of preceding billing data, including diagnosis and procedure codes to predict total cost of care in the subsequent year.
We defined complexity thresholds for each algorithm using: 1) a risk score threshold for ePDC that reflected maximal model accuracy in the development data set (14.5 % complex); 2) an OCS threshold of ≥ 4 (11.7 % complex); and 3) a risk score threshold for the CRP matching the proportion of complex patients identified using ePDC (14.1 % complex). Using an OCS threshold of ≥ 3 (18.7 % complex) did not change the direction of any of our findings.
Clinical Quality and Utilization Outcomes
For all eligible patients during the 2008–2011 follow-up period, we determined the start of follow-up time based on test eligibility and censored follow-up time based on the first of any of the following events: death, test ineligibility, or 3 years after the last clinic visit. We calculated the percentage of time (total number of days for which an eligible patient was unscreened divided by the total number of eligible person days in the follow-up period) for which breast, cervical and colon cancer screening was incomplete. We calculated the percentage of time (total number of days for which an eligible patient had an out-of-range test divided by the total number of eligible person days in the follow-up period) for which the Hemoglobin A1c (HbA1c) was ≥ 9 % (for patients with diabetes) or the low-density lipoprotein (LDL) was ≥ 130 mg/dl (for patients with diabetes and cardiovascular disease, based on diagnoses of coronary artery disease, cerebrovascular disease and/or peripheral vascular disease), using the last lab value carried forward. We calculated the number of different utilization events per person year by dividing the number of primary care outpatient visits and “no show” visits, emergency department visits, or hospital admissions by total follow-up time for all patients in the complexity cohort. Finally, we calculated 30-day readmission rates within each complexity cohort.
Statistical Analysis
We used chi-square or t-tests, as appropriate, to compare characteristics of patients and prospective clinical quality and utilization outcomes over 4 years between complexity cohorts. We compared: 1) ePDC-complex groups to non-complex patients, and 2) ePDC complexity groups (ePDC only and ePDC combined with OCS or CRP, respectively), to those that were complex by OCS and the CRP only. Finally, we stratified our analyses for admissions by age (< 45, 45–64, age ≥ 65 years old) to account for potential impacts of age on clinical decision-making.
RESULTS
Patient Cohort
Our baseline cohort of 143,372 adult patients had a mean age of 49 years, was 57.4 % female and 23.1 % non-white race/ethnicity. We identified 19.2 % of these patients as complex by ePDC or OCS; 36.7 % of which were complex by both algorithms (Fig. 1a). One-fifth (20.5 %) of patients were complex by ePDC or CRP; 39.6 % of which were complex by both algorithms (Fig. 1b)
Figure 1.
a Complexity as defined by physician-defined complexity (ePDC) and outpatient Charlson Score (OCS).. b Complexity as defined by physician-defined complexity (ePDC) and a commercial risk predictor (CRP).
Comparing Characteristics of Complex Patients by Different Algorithms
ePDC-complex patients were younger and less likely to be commercially insured, English-speaking, male or married, and more likely to live in low socioeconomic status census block groups than OSC-complex or CRP-complex patients. Compared to both OSC-complex and CRP-complex patients, ePDC-complex patients had a greater number primary care visit “no shows” at baseline, were prescribed a greater number of high-risk medications, and had higher rates of chronic obstructive pulmonary disease, diabetes, depression, and psychiatric diagnoses (Table 1).
Table 1.
Patient-level Characteristics: Estimated Physician-Defined Complexity vs. Outpatient Charlson Score and Estimated Physician-Defined Complexity vs. the Commercial Risk Predictor
| Characteristics | Physician-Defined Complexity (ePDC) vs. Outpatient Charlson (OCS) | Physician-Defined Complexity (ePDC) vs. Commercial Risk Predictor (CRP) | ||||
|---|---|---|---|---|---|---|
| Complex by | Complex by | |||||
| OCS Only n = 6,691 |
ePDC Only n = 10,725 |
ePDC & OCS n = 10,115 |
CRP Only n = 8,592 |
ePDC Only n = 9,195 |
ePDC & CRP n = 11,645 |
|
| Age in years, mean (SD) | 62.3 (15.2) | 59.0 (15.9)‡ | 69.3 (14.2) ‡ | 60.1 (17.5) | 59.1 (15.5) ‡ | 67.9 (15.2) ‡ |
| < 45, % | 15.9 | 15.2‡ | 3.9‡ | 23.8 | 14.2‡ | 6.2‡ |
| 45–64, % | 30.5 | 54.5‡ | 34.8‡ | 25.7 | 55.8‡ | 36.4‡ |
| ≥ 65, % | 53.6 | 30.3‡ | 61.4‡ | 50.5 | 30.1‡ | 57.4‡ |
| Female, % | 49.2 | 63.9‡ | 53.9‡ | 57.6 | 62.0‡ | 56.7 |
| Race | ||||||
| White, % | 84.1 | 73.5‡ | 82.2‡ | 81.6 | 73.6‡ | 81.0‡ |
| Black, % | 5.5 | 7.5‡ | 6.3‡ | 5.7 | 7.6‡ | 6.4‡ |
| Hispanic, % | 5.3 | 13.2‡ | 7.0‡ | 6.8 | 12.8‡ | 8.1‡ |
| Asian, % | 3.0 | 3.2‡ | 2.4‡ | 3.8 | 3.3‡ | 2.4‡ |
| Other, % | 2.1 | 2.7‡ | 2.1‡ | 2.1 | 2.8‡ | 2.1‡ |
| Primary Insurance | ||||||
| Commercial, % | 48.2 | 34.9‡ | 23.8‡ | 47.9 | 36.2‡ | 24.2‡ |
| Medicare, % | 44.4 | 45.8‡ | 65.3‡ | 43.0 | 43.5‡ | 64.6‡ |
| Medicaid/State/Free/Uninsured, % | 7.4 | 19.3‡ | 10.9‡ | 9.0 | 20.3‡ | 11.2‡ |
| Primary Language, English, % | 93.5 | 85.7‡ | 90.3‡ | 91.6 | 86.0‡ | 89.5‡ |
| Married, % | 61.7 | 39.3‡ | 44.7‡ | 59.3 | 41.0‡ | 42.6‡ |
| CBG Non High School Graduation Rate | ||||||
| < 5 %, % | 24.7 | 14.0‡ | 15.0‡ | 23.4 | 14.5‡ | 14.5‡ |
| 5–25 %, % | 51.0 | 48.9‡ | 49.7‡ | 50.2 | 49.2‡ | 49.4‡ |
| ≥ 25 %, % | 13.6 | 25.7‡ | 20.9‡ | 15.8 | 24.7‡ | 22.3‡ |
| CBG Median Household Income | ||||||
| < $40,000, $ | 20.4 | 35.1‡ | 30. 5‡ | 23.8 | 33.2‡ | 32.7‡ |
| $40,000–70,000, $ | 43.7 | 42.2‡ | 42.8‡ | 43.0 | 42.9‡ | 42.2‡ |
| ≥ $70,000, $ | 30.7 | 18.2‡ | 22.0‡ | 28.7 | 19.0‡ | 20.9‡ |
| ≥ 2 Primary Care “No Shows” in 3 years, % | 8.8 | 37.1‡ | 28.8‡ | 9.8 | 37.0‡ | 30.0‡ |
| Number of e-Prescribed medications in 3 years, mean (SD) | 12.2 (8.2) | 18.6 (8.1)‡ | 23.6 (7.6) ‡ | 12.8 (8.5) | 17.6 (7.9) ‡ | 23.7 (7.5) ‡ |
| > 10 Medications e-Prescribed in 3 years, % | 52.6 | 82.9‡ | 93.0‡ | 56.9 | 80.3‡ | 93.7‡ |
| Number of e-Prescribed High-Risk Medications in 3 years, mean (SD) | 1.3 (1.6) | 2.8 (1.9) ‡ | 2.9 (1.9)‡ | 1.5 (1.7) | 2.5 (1.9)‡ | 3.1 (1.9) ‡ |
| ≥ 1 High-Risk Medications e-Prescribed in 3 years, % | 55.0 | 83.3‡ | 84.5‡ | 60.4 | 80.5‡ | 86.6‡ |
| Chronic Diseases | ||||||
| Chronic Obstructive Pulmonary Disease, % | 23.1 | 24.3 | 40.2‡ | 16.0 | 26.9‡ | 36.1‡ |
| Vascular Disease, % | 26.2 | 8.7‡ | 45.2‡ | 16.9 | 15.5* | 35.0‡ |
| Depression, % | 18.7 | 57.4‡ | 46.1‡ | 24.5 | 53.2‡ | 50.8‡ |
| Diabetes, % | 12.4 | 25.7‡ | 45.3‡ | 7.7 | 30.8‡ | 38.7‡ |
| Hypertension, % | 53.9 | 38.1‡ | 67.5‡ | 35.7 | 46.6‡ | 57.0‡ |
| End-stage Renal Disease or Transplant, % | 0.4 | 0.9‡ | 2.4‡ | 0.9 | 0.4‡ | 2.6‡ |
| Any Psychiatric Diagnosis, % | 31.4 | 72.4‡ | 63.5‡ | 35.7 | 69.3‡ | 67.2‡ |
*p < 0.05, †p < 0.01, ‡p < 0.001
CBG = Census Block Group
High-Risk Medications = medications in the following classes: Benzodiazepines, Antipsychotics, Opiates, Muscle Relaxants
Vascular Disease = Coronary Artery Disease or Cerebrovascular Disease or Peripheral Vascular Disease
Any psychiatric diagnosis = Depression, Anxiety, Panic Disorder, Schizophrenia, other psychotic disorders, Bipolar Disorder
Clinical Quality Outcomes
Over 4 years of follow-up, patients classified as complex by ePDC only spent a greater proportion of time overdue and unscreened for preventable cancers compared to patients complex by OCS and CRP only (Table 2). Differences in the proportion of time unscreened by complexity cohort were as follows (p < 0.001 for all): 1) cervical cancer – 17.8 % (ePDC) vs. 13.3 % (OCS); 19.4 % (ePDC) vs. 11.2 % (CRP); 2) breast cancer – 21.4 % (ePDC) vs. 14.9 % (OCS); 22.7 % (ePDC) vs. 15.0 % (CRP); and 3) colon cancer – 25.9 % (ePDC) vs. 18.7 % (OCS); 27.0 % (ePDC) vs. 18.2 % (CRP).
Table 2.
Clinical Quality Outcomes: Estimated Physician-Defined Complexity vs. Outpatient Charlson Score and Estimated Physician-Defined Complexity vs. the Commercial Risk Predictor
| Measure | Physician-Defined Complexity (ePDC) vs. Outpatient Charlson (OCS) | Physician-Defined Complexity (ePDC) vs. Commercial Risk Predictor (CRP) | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Complex by | Complex by | |||||||||||
| OCS Only n = 6,691 |
ePDC Only n = 10,725 |
ePDC & OCS n = 10,115 |
CRP Only n = 8,592 |
ePDC Only n= 9,195 |
ePDC & CRP n = 11,645 |
|||||||
| # Eligible Patients | % of follow-up time with incomplete test | # Eligible Patients | % of follow-up time with incomplete test | # Eligible Patients | % of follow-up time with incomplete test | # Eligible Patients | % of follow-up time with incomplete test | # Eligible Patients | % of follow-up time with incomplete test | # Eligible Patients | % of follow-up time with incomplete test | |
| Cancer Screening | ||||||||||||
| Breast Cancer | 2240 | 14.9 | 4549 | 21.4‡ | 2832 | 21.3‡ | 2899 | 15.0 | 3807 | 22.7‡ | 3574 | 20.0 |
| Cervical Cancer | 1468 | 13.3 | 3916 | 17.8‡ | 1607 | 20.1‡ | 2735 | 11.2 | 3178 | 19.4‡ | 2345 | 17.1‡ |
| Colorectal Cancer | 4123 | 18.7 | 6100 | 25.9‡ | 5376 | 23.4 | 4426 | 18.2 | 5416 | 27.0‡ | 6060 | 22.7 |
| Diabetes | ||||||||||||
| LDL ≥ 130 | 1088 | 7.9 | 3159 | 12.4‡ | 4887 | 8.3 | 966 | 9.0 | 3133† | 11.8‡ | 4913 | 8.8 |
| A1C ≥ 9 | 1089 | 8.1 | 3184 | 15.6‡ | 4912 | 10.4 | 972 | 6.9 | 3162 | 15.9‡ | 4934 | 10.3 |
| Vascular Disease | ||||||||||||
| LDL ≥ 130 | 1348 | 5.7 | 996 | 10.4‡ | 3593 | 6.7 | 1315 | 5.3 | 1128 | 9.5 | 3461 | 6.8* |
*p < 0.05, †p < 0.01, ‡p < 0.001
LDL = Low-Density Lipoprotein, A1C = Hemoglobin A1c
Over 4 years of follow-up among patients with diabetes, compared to patients complex by OCS and CRP only, patients complex by ePDC only spent a greater proportion of time with hemoglobin A1c > 9 % (15.6 % vs. 8.1 % for OCS; 15.9 % vs. 6.9 % for CRP, p < 0.001 for both) and LDL > 130 mg/dl (12.4 % vs. 7.9 % for OCS; 11.8 vs. 9.0 for CRP, p < 0.001) (Table 2). For patients with cardiovascular disease, patients complex by ePDC only spent a greater proportion of time with LDL > 130 mg/dL compared to patients complex by OCS and CRP only (10.4 % vs. 5.7 % for OCS; 9.5 % vs 5.3 % for CRP, p < 0.001).
Utilization Outcomes
Over 4 years of follow-up, patients complex by ePDC only made significantly (p < 0.001 for all) more primary care visits (3.78 vs. 2.92 visits per person year (ppy) for OCS only; 3.98 vs. 2.08 ppy for CRP only), had higher rates of primary care “no-shows” (0.45 vs. 0.16 “no-shows” ppy for OCS only; 0.43 vs. 0.15 “no-shows” ppy for CRP only), and emergency department visits (0.21 vs. 0.11 visits ppy for OCS only; 0.17 vs. 0.15 ppy for CRP only) (Table 3).
Table 3.
Health Care Utilization Outcomes: Estimated Physician-Defined Complexity vs. Outpatient Charlson Score and Estimated Physician-Defined Complexity vs. the Commercial Risk Predictor
| Measure | Comparing Physician-Defined Complexity (ePDC) to Outpatient Charlson (OCS) | Comparing Physician-Defined Complexity (ePDC) to the Commercial Risk Predictor (CRP) | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Complex by | Complex by | |||||||||||
| OCS Only n = 6,691 |
ePDC Only n = 10,725 |
ePDC & OCS n = 10,115 |
CRP Only n = 8,592 |
ePDC Only n = 9,195 |
CRP Only n = 8,592 |
|||||||
| Events | Rate (ppy) | Events | Rate (ppy) | Events | Events | Events | Rate (ppy) | Events | Rate (ppy) | Events | Rate (ppy) | |
| Primary Care Visits | 73,226 | 2.92 | 153,877 | 3.78‡ | 150,137 | 4.26‡ | 100,607 | 3.07 | 116,945 | 3.41‡ | 187,069 | 4.50‡ |
| Primary Care No-Shows | 4070 | 0.16 | 18,308 | 0.45‡ | 14,027 | 0.40‡ | 6240 | 0.19 | 15,135 | 0.44‡ | 17,200 | 0.41‡ |
| ED Visits | 2869 | 0.11 | 8622 | 0.21‡ | 9283 | 0.26‡ | 4773 | 0.15 | 5778 | 0.17‡ | 12,127 | 0.29‡ |
| Admissions | 4004 | 0.16 | 5894 | 0.15‡ | 14,210 | 0.40‡ | 5085 | 0.16 | 4143 | 0.12‡ | 15,961 | 0.38‡ |
| < 45 | 412 | 0.11 | 621 | 0.10 | 490 | 0.33‡ | 649 | 0.08 | 332 | 0.07† | 779 | 0.28‡ |
| 45-64 | 751 | 0.10 | 2476 | 0.11‡ | 4234 | 0.33‡ | 997 | 0.12 | 1985 | 0.10‡ | 4725 | 0.30‡ |
| 65+ | 2841 | 0.21 | 2797 | 0.24‡ | 9486 | 0.46‡ | 3439 | 0.21 | 1826 | 0.19‡ | 10,457 | 0.46‡ |
| 30-Day Re-admission Ratea | 3828 | 15.4 | 5612 | 13.0‡ | 13,472 | 18.9‡ | 4888 | 14.1 | 3947 | 12.5‡ | 15,137 | 18.3‡ |
*p < 0.05, †p < 0.01, ‡p < 0.001
aRate refers to percentage of eligible hospital discharges with readmission in the subsequent 30 days
ppy = per person years
ED = Emergency Department
denom = denominator
Patients complex by ePDC only had lower overall hospital admission rates compared to patients complex by OCS only (0.15 vs. 0.16 admits/py, p < 0.001). However, when stratified by age, ePDC-complex patients had higher subsequent hospitalization rates in older age groups (0.11 ppy for ePDC only vs. 0.10 ppy for OCS only in the 45–64 year subgroup, p = 0.002; and 0.24 ppy for ePDC only vs. 0.21 ppy for OCS only in the ≥ 65 year subgroup, p < 0.001) (Table 3). Patients complex by ePDC only had lower 30-day readmission rates compared to patients complex by OCS only (13.0 % vs 15.4 %, p = 0.001). Patients complex by ePDC only had lower hospital admission and readmission rates compared to patients complex by CRP only (0.12 vs. 0.16 admits ppy and 12.5 % vs. 14.1 % readmitted, p < 0.001 for both). Compared to patients complex by OCS and CRP only, patients who were complex by ePDC and OCS and ePDC and CRP, respectively, had the highest rates for all utilization outcomes.
DISCUSSION
We previously showed that primary care physicians (PCPs) take into account patient needs across medical, social, behavioral, and environmental dimensions when identifying their complex patients.24,25 Using data available in health system electronic data repositories, we sought to approximate qualitative physician assessment of complexity by creating a predictive model for PCP-defined patient complexity. We found that this unique measure of estimated physician-defined complexity (ePDC) both successful identified patients that physicians define complex (eTable 1) and prospectively identified patients at increased risk for suboptimal outpatient quality and healthcare utilization outcomes.
This ePDC model had only modest overlap with the outpatient Charlson score (OCS) and a commercial risk predictor (CRP), suggesting that ePDC identified patients with attributes missed by these other methods. Compared to OCS-complex and CRP-complex patients, ePDC-complex patients had a higher prevalence of psychosocial issues, including: mental health diagnoses and proxies for poor adherence and poverty, such as high primary care “no show” rates and residence in neighborhoods with lower socioeconomic indicators.
Over the ensuing 4 years, ePDC-complex patients generally had poorer clinical quality outcomes and higher “no-show” rates compared to OCS-complex and CRP-complex patients. Although the magnitude of the findings in clinical quality outcomes may not be clinically significant within our system due to high screening rates, the effect could be greater in healthcare systems with lower screening rates. Directing health system resources and effective interventions to ePDC-complex patients could improve care for high-risk patients, while enhancing achievement of quality measures used in incentive-based payment approaches and reducing practice income loss from frequent “no-shows.”
ePDC was a stronger predictor of future emergency department utilization compared to OCS-complex and CRP-complex patients, identifying a high-risk group that had a twofold increased utilization rate compared to OCS. Although ePDC performed well in some age strata, ePDC was not generally a stronger predictor of future admissions and readmissions than OCS or CRP. However, patients identified as complex by two different risk-stratification approaches had the highest rates of future acute care utilization. For example, patients complex by both ePDC and OCS had over twofold higher rates of emergency department (ED) visits and admissions and 23 % more 30-day readmissions than patients identified as complex by OCS only. Similar differences were seen for patients complex by both ePDC and CRP. These results suggest that adding ePDC to traditional comorbidity and utilization-based approaches identified a population that was significantly more likely to use acute care services in the near term. This additive effect may be explained by the predictive potential of psychosocial factors captured by the ePDC model. Given the increasing focus on the highest-utilizers of acute care services, ePDC may have significant value in patient identification for intensive care management programs.
Our results have implications for resource allocation and risk-stratification within healthcare delivery systems. Extensive literature demonstrates a link between various diagnosis-based and prior utilization-based measures of complexity, acute care utilization, and costs.19–23,29–32 Generally, these approaches use administrative claims/billing, and less commonly, electronic medical record data to create risk groupings that predict future utilization and costs.33–35 More limited literature demonstrates that adding outpatient utilization information and psychosocial factors improve predictive models for acute care utilization.35–37 Prior studies of physician’s ability to predict death or acute care utilization results show mixed results;13–18,22 however, our quantitative model, designed to predict physician-defined complexity from increasingly available data, shows promise as an additional tool in identifying and stratifying high-risk patients for population management interventions. Health care delivery systems could replicate our approach in their own context or add proxies for psychosocial complexity that are available in their data repositories to strengthen their risk-stratification approaches. Given that primary care physician review of each patient in their panel is time consuming and difficult to implement broadly, approaches, like ePDC, that combine the PCP perspective, psychosocial risk factors, and traditional medical measures of severity may have broad applicability in primary care risk stratification and resource allocation, and warrant further study.
ePDC may identify patients that are not only at high risk for poor outcomes and utilization of care, but may place a higher burden on primary care teams. Although we were unable to directly assess primary care effort, ePDC-complex patients are more psychosocially complex, have high risk for poor clinical quality outcomes, make more primary care visits, more often “no show” for visits, and present more care coordination complexity due to more frequent acute care utilization. Providing enhanced support to PCPs who manage ePDC-complex patients may reduce clinician burnout, improve clinician experience, and attract new graduates to primary care. ePDC may also be a better marker for PCP panel workload than traditional risk-prediction measures. This suggests the need to further study the use of ePDC-like models in risk adjustment of PCP payments or in the allocation of resources like care management teams or behavioral health supports to clinicians that care for a disproportionate share of ePDC-complex patients.
ePDC is a quantitative predictive model designed to estimate physician-defined complexity within one academic primary care system. Criteria used by physicians to assess patient complexity may be variable. Further validation of ePDC in other settings and populations and research on how physicians identify complex patients is warranted. The accuracy of models will vary depending on available data in other settings. Although our specific ePDC model may not be generalizable to other settings with different physicians and staff support, both the approach used to develop the ePDC model and the addition of psychosocial predictors to traditional risk-prediction models should be. We primarily used electronically available patient data for our analyses, which is limited by missing and unverified data. This may result in some bias; however, this pragmatic approach offers greater generalizability for applications within organized health systems. All clinical quality and acute care utilization outcomes were taken from electronic repositories, and as such, out-of network data were not available potentially biasing results towards the null. Lacking access to claims data, we applied models to our health system’s billing data—a reasonable proxy for the practical use of risk-prediction software in health systems, given the general lack of available claims data. Finally, our approach to identifying patients at risk for poor outcomes was indirect. ePDC and psychosocial predictors should be studied as direct predictors of outcomes in risk prediction models.
This study adds to the literature in several key ways. First, the concept of identifying high-risk patients whom primary care physicians also identify as complex would be a unique starting point for interventions aimed at improving physician experience and efficiency of healthcare delivery simultaneously. Second, using a model such as ePDC to augment existing risk prediction approaches is useful—particularly as healthcare delivery systems seek better ways to identify complex patients for intensive care management interventions. Finally, only limited literature suggests a role for social factors and physician assessment in identifying high-risk patients. While our approach is admittedly indirect in these regards, we do believe it adds to this literature.
In conclusion, a prediction algorithm for physician-defined complexity, which identifies patients that PCPs find complex to manage, was also a predictor for suboptimal clinical quality outcomes and future acute care utilization. Our findings may enhance current risk adjustment and prediction approaches and help identify and risk-stratify complex patients that are both at high risk for suboptimal clinical and utilization outcomes and challenging for physicians to manage. As health systems work to identify complex patients for care redesign in order to improve efficiency of care and reduce costs, our approach informs next steps in the field of risk-prediction.
Electronic supplementary material
Below is the link to the electronic supplementary material.
(DOCX 37 kb)
Acknowledgements
We would like to thank Dr. Adrian Zai from Laboratory of Computer Sciences at Massachusetts General Hospital for his work in developing the technology platform used by physician in their assessment of their patients.
Conflict of Interest
The authors declare that they do not have a conflict of interest.
Footnotes
SUPPORT
Supported by a grant from Partners Community Healthcare, Inc, and institutional funding through the Massachusetts General Hospital Primary Care Practice-Based Research and Quality Improvement Network. Dr. Hong is supported by a KL2/Catalyst Medical Research Investigator Training award (an appointed KL2 award) from Harvard Catalyst, The Harvard Clinical and Translational Science Center (National Center for Research Resources and the National Center for Advancing Translational Sciences, National Institutes of Health Award KL2 TR001100).
REFERENCES
- 1.AHRQ, Agency for Healthcare Research and Quality. Closing the quality gap: A critical analysis of quality improvement strategies. 2007;7.
- 2.Atlanticare Special Care Center. Special care center. http://www.atlanticare.org/index.php/special-care-center. Accessed April 1, 2015.
- 3.Bell J, Mancuso D, and Krupski TE. A. A randomized controlled trial of king county partners' rethinking care intervention: Health and social outcomes up to two years post-randomization. 2012.
- 4.Bielaszka-DuVernay C. Vermont’s blueprint for medical homes, community health teams, and better health at lower cost. Health Affairs (Project Hope) 2011;30(3):383–386. doi: 10.1377/hlthaff.2011.0169. [DOI] [PubMed] [Google Scholar]
- 5.Cohen SB, Yu W. The concentration and persistence in the level of health expenditures over time: Estimated the U.S. population, 2008–2009. Med Expenditure Panel Surv. 2012;Statistical Brief #354.
- 6.Brown RS, Peikes D, Peterson G, Schore J, Razafindrakoto CM. Six features of medicare coordinated care demonstration programs that cut hospital admissions of high-risk patients. Health Aff (Millwood) 2012;31(6):1156–1166. doi: 10.1377/hlthaff.2012.0393. [DOI] [PubMed] [Google Scholar]
- 7.Dorr DA, Wilcox A, Donnelly SM, Burns L, Clayton PD. Impact of generalist care managers on patients with diabetes. Health Serv Res. 2005;40:1400–1421. doi: 10.1111/j.1475-6773.2005.00423.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Peikes D, Chen A, Schore J, Brown R. Effects of care coordination on hospitalization, quality of care, and health care expenditures among medicare beneficiaries: 15 randomized trials. JAMA. 2009;301(6):603–618. doi: 10.1001/jama.2009.126. [DOI] [PubMed] [Google Scholar]
- 9.McCall N, Cromwell J, Urato C. Evaluation of medicare care management for high cost beneficiaries (CMHCB) demonstration: Massachusetts general hospital and massachusetts general physicians organization. 2010;RTI Project Number 0207964.025.000.001.
- 10.Boult C, Reider L, Leff B, et al. The effect of guided care teams on the use of health services: Results from a cluster- randomized controlled trial. Arch Intern Med. 2011;171:460–466. doi: 10.1001/archinternmed.2010.540. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Counsell SR, Callahan CM, Clark DO, et al. Geriatric care management for low-income seniors: A randomized controlled trial. JAMA. 2007;298(22):2623–2633. doi: 10.1001/jama.298.22.2623. [DOI] [PubMed] [Google Scholar]
- 12.Waxmonsky JA, Glese A, McGinnis GF, et al. Colorado access’ enhanced care management for high-cost, high-need medicaid members. J Ambul Care Manag. 2011;34(2):183–191. doi: 10.1097/JAC.0b013e31820f64be. [DOI] [PubMed] [Google Scholar]
- 13.Pattison M, Romer AL. Improving care through the end of life: Launching a primary care clinic-based program. J Palliat Med. 2001;4:249–254. doi: 10.1089/109662101750290335. [DOI] [PubMed] [Google Scholar]
- 14.Moss AH, Lunney JR, Culp S, et al. Prognostic significance of the “surprise” question in cancer patients. J Palliat Med. 2010;13(7):837–840. doi: 10.1089/jpm.2010.0018. [DOI] [PubMed] [Google Scholar]
- 15.Moss AH, Ganjoo J, Sharma S, et al. Utility of the “surprise” question to identify dialysis patients with high mortality. Clin J Am Soc Nephrol. 2008;1:1379–1384. doi: 10.2215/CJN.00940208. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Cohen LM, Ruthazer R, Moss AH, Germain MJ. Predicting six month mortality for patients who are on maintenance hemodialysis. Clin J Am Soc Nephrol. 2010;5(1):72–79. doi: 10.2215/CJN.03860609. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Olfson M, Mechanic D, Boyer CA, Hansell S, Walkup J, Weiden PJ. Assessing clinical predictions of early rehospitalization in schizophrenia. J Nery Ment Dis. 1999;187(12):721–729. doi: 10.1097/00005053-199912000-00003. [DOI] [PubMed] [Google Scholar]
- 18.Allaudeen N, Schnipper JL, Orav EJ, Wachter RM, Vidyarthi AR. Inability of providers to predict unplanned readmissions. J Gen Intern Med. 2011;26(7):771–776. doi: 10.1007/s11606-011-1663-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Pope GC, Ellis RP, Ash AS, Ayanian JZ, Bates DW, Burstin H, Iezzoni LI, Marcantonio E, Wu B. Diagnostic cost group hierarchical condidtion category models for Medicare risk adjustment. Health Care Financ Adm. 2000. [PMC free article] [PubMed]
- 20.Hughes JS, Averil RF, Eisenhandler J, Goldfield NI, Muldoon J, Neff JM, Gay JC. Clinical Risk Groups (CRGs): a classification system for risk-adjusted capitation-based payment and health care management. Med Care. 2004;42(1):81–90. doi: 10.1097/01.mlr.0000102367.93252.70. [DOI] [PubMed] [Google Scholar]
- 21.Stam PJ, van Vilet RC, van de Ven WP. Diagnostic Pharmacy-based, and Self-reported Health Measures in Risk Equalization Models. Med Care. 2010;48(5):448–457. doi: 10.1097/MLR.0b013e3181d559b4. [DOI] [PubMed] [Google Scholar]
- 22.Freund T, Mahler C, Erler A, Gensichen J, Ose D, Szecsenyi J, Peters-Klimm F. Identification of patients likely to benefit from care management programs. Am J Manag Care. 2011;17(5):345–352. [PubMed] [Google Scholar]
- 23.Kansagara D, Englander H, Salanitro A, Kagen D, Theobald C, Freeman M, Kripalani S. Risk prediction models for hospital readmission: a systematic review. J Am Med Assoc. 2011;306(15):1688–1698. doi: 10.1001/jama.2011.1515. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Grant RW, Ashburner JM, Hong CS, Chang Y, Barry MJ, Atlas SJ. Defining patient complexity from the primary care physician’s perspective: A cohort study. Ann Intern Med. 2011;155(12):797–804. doi: 10.7326/0003-4819-155-12-201112200-00001. [DOI] [PubMed] [Google Scholar]
- 25.Grant RW, Wexler DJ, Ashburner JM, Hong CS, Atlas SJ. Characteristics of “complex” patients with type 2 diabetes mellitus according to their primary care physicians. Arch Intern Med. 2012;172(10):821–823. doi: 10.1001/archinternmed.2012.1229. [DOI] [PubMed] [Google Scholar]
- 26.Charlson ME, Charlson RE, Peterson JC, Marinopoulos SS, Briggs WM, Hollenburg JP. The charlson comorbidity index is adapted to predict costs of chronic disease in primary care patients. J Clin Epidemiol. 2008;61(12):1234–1240. doi: 10.1016/j.jclinepi.2008.01.006. [DOI] [PubMed] [Google Scholar]
- 27.Dang DK, Pont JM, Portmoy MA. Episode treatment groups: An illness classification and episode building system–part II. Med Interface. 1996;9(4):122–128. [PubMed] [Google Scholar]
- 28.Dang DK, Pont JM, Portnoy MA. Episode treatment groups: An illness classification and episode building system–part I. Med Interface. 1996;9(3):118–122. [PubMed] [Google Scholar]
- 29.Kansagara D, Englander H, Salanitro A, et al. Risk prediction models for hospital readmission: A systematic review. JAMA: J Am Med Assoc. 2011;306(15):1688–1698. doi: 10.1001/jama.2011.1515. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Kronick R, Dreyfus T, Lee L, Zhou Z. Diagnostic risk adjustment for medicaid: The disability payment system. Health Care Financ Rev. 1996;17(3):7–33. [PMC free article] [PubMed] [Google Scholar]
- 31.Ash AS, Zhao Y, Ellis RP, Schlein KM. Finding future high-cost cases: Comparing prior cost versus diagnosis-based methods. Health Serv Res. 2001;36:194–206. [PMC free article] [PubMed] [Google Scholar]
- 32.Petersen LA, Byrne MM, Daw CN, Hasche J, Reis B, Pietz K. Relationship between clinical conditions and use of veterans affairs health care among medicare-enrolled veterans. Health Serv Res. 2010;45(3):762–791. doi: 10.1111/j.1475-6773.2010.01107.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Wahls TL, Barnett MJ, Rosenthal GE. Predicting resource utilization in a veterans health administration primary care population: comparison of methods based on diagnoses and medications. Med Care. 2004;42(2):123–128. doi: 10.1097/01.mlr.0000108743.74496.ce. [DOI] [PubMed] [Google Scholar]
- 34.Rosen AK, Loveland SA, Anderson JJ, Hankin CS, Breckenridge JN, Berlowitz DR. Diagnostic cost groups (DCGs) and concurrent utilization among patients with substance abuse disorders. Health Serv Res. 2002;37(4):1079–1103. doi: 10.1034/j.1600-0560.2002.67.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Haas LR, Takahashi PY, Shah ND, Stroebel RJ, Bernard ME, Finnie DM, Naessens JM. Risk-stratification methods for identifying patients for care coordination. Am J Manag Care. 2013;19(9):725–732. [PubMed] [Google Scholar]
- 36.Lemke KW, Weiner JP, Clark JM. Development and validation of a model for predicting inpatient hospitalization. Med Care. 2012;50(2):131–139. doi: 10.1097/MLR.0b013e3182353ceb. [DOI] [PubMed] [Google Scholar]
- 37.Donnan PT, Dorward DW, Mutch D, Morris AD. Development and validation of a model for predicting emergency admissions over the next year (PEONY): a UK historical cohort sutdy. Arch Intern Med. 2008;168(13):1416–1422. doi: 10.1001/archinte.168.13.1416. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
(DOCX 37 kb)