Measuring Population Health in a Large Integrated Health System to Guide Goal Setting and Resource Allocation: A Proof of Concept

Abstract

In integrated health care systems, techniques that identify successes and opportunities for targeted improvement are needed. The authors propose a new method for estimating population health that provides a more accurate and dynamic assessment of performance and priority setting. Member data from a large integrated health system (n = 96,246, 73.8% female, mean age = 44 ± 0.01 years) were used to develop a mechanistic mathematical simulation, representing the top causes of US mortality in 2014 and their associated risk factors. An age- and sex-matched US cohort served as comparator group. The simulation was recalibrated and retested for validity employing the outcome measure of 5-year mortality. The authors sought to estimate potential population health that could be gained by improving health risk factors in the study population. Potential gains were assessed using both average life years (LY) gained and average quality-adjusted life years (QALYs) gained. The simulation validated well compared to integrated health system data, producing an AUC (area under the curve) of 0.88 for 5-year mortality. Current population health was estimated as a life expectancy of 84.7 years or 69.2 QALYs. Comparing potential health gain in the US cohort to the Kaiser Permanente cohort, eliminating physical inactivity, unhealthy diet, smoking, and uncontrolled diabetes resulted in an increase of 1.5 vs. 1.3 LY, 1.1 vs. 0.8 LY, 0.5 vs. 0.2 LY, and 0.5 vs. 0.5 LY on average per person, respectively. Using mathematical simulations may inform efforts by integrated health systems to target resources most effectively, and may facilitate goal setting.

Background

Health systems continue to search for useful and actionable means to optimize health care quality and improve patient outcomes.¹ One potential tool to inform goal setting and resource allocation is a mechanistic computer simulation that creates a simulated population that integrates important diseases and risk factors. This approach permits “experiments,” such as asking how population health would change if particular programs were enacted. Given that budgets are typically constrained, simulations can be used when conventional experiments would not be feasible, thus allowing alternative programs for health improvement to be explored.

Current approaches for estimating potential health improvement have a number of limitations including not incorporating knowledge about a population's leading causes of mortality, the risk factors underlying these causes of mortality, or underlying population characteristics that may fuel competing risks for mortality.² To address these limitations and develop a more accurate and actionable means of assessing potential for health improvement, the study team created a mathematical simulation that can be tailored to a population's particular characteristics and that considers competing causes of mortality together with their risk factors. This type of simulation holds the potential to improve estimation of health system performance and ultimately can be a useful tool to inform resource allocation within. This study illustrates a proof of concept using member data from a large integrated health system to estimate the domains with greatest potential health gains, laying the groundwork for evaluating which interventions in those domains are best suited for resource allocation.

Methods

The study team collaborated with leadership at a large integrated health care organization to identify an approach to measure and inform optimization of population health with enough rigor to satisfy academic researchers and sufficient actionability to satisfy leadership at the front lines of population health improvement. In place of methods such as regression models or other types of statistical analyses, which can be useful in estimating the impact of various factors on health, a simulation was developed because of its capability to dynamically perform hypothetical experiments and estimate health outcomes such as potential life years (LYs) and quality-adjusted life years (QALYs) gained in a population.

This proof of concept involved (1) adapting a simulation for the US population to the population of the integrated health system, (2) assessing whether it had sufficient validity for subsequent analyses, and (3) determining the domains wherein the greatest potential population health benefit exists. Study methods were approved by the Southern California Kaiser IRB.

Adapting the simulation for the integrated health system

To address health system-specific population and health risk patterns, the study team adapted a previously validated simulation that could estimate the potential LYs and QALYs gained in the US population³ to a large integrated health system member population. This model incorporates a Monte Carlo simulation in which hypothetical individuals are generated with characteristics including age, sex, the 19 most important mortality-causing conditions, and 27 risk factors with clinically and statistically significant relationships with those 19 mortality causes. The top causes of mortality were compiled from the Centers for Disease Control and Prevention tables of the top 10 causes of mortality by sex in each 10-year age group in the United States.⁴ The model estimates outcomes that include expectancies of LYs and QALYs. Importantly, hypothetical members can be generated who replicate the baseline characteristics of a real member cohort, such as that in the integrated health system. Each hypothetical member then goes through a virtual life in the simulation, developing new risk factors and/or new conditions, or having existing risk factors and/or conditions resolve, until they die. The simulation structure has been designed with flexibility in mind so that additional risk factors or mortality-driving conditions can be added or removed as necessary based on a population's characteristics. The quality of evidence for risk factor–condition links were graded and only the highest grades (eg, those with a highest likelihood of causality) were used in the base case analysis, in particular randomized controlled trials, meta-analyses, and non-randomized longitudinal studies with control groups.^5,6 However, to preserve comparability with other studies that do not distinguish between risk factor–condition links with higher versus lower likelihoods of causation, the team performed a secondary analysis incorporating risk factor–condition links regardless of likelihood of causality. A complete explanation of the simulation and input tables are available in the Supplementary Data and in the previously published literature.³

Study population

Kaiser Permanente (KP) Southern California's electronic health records (EHRs) were used to construct a hypothetical population using member data from 2010–2015. KP Southern California is an integrated health system serving nearly 4.4 million members in southern California. Members were selected if they were patients of KP Health Group, had a response for self-reported health in 2010 or 2011 (thereby enabling 5–6 years of follow-up through the analysis year of 2017), and were 15 years or older at the time of the self-reported health response. Data collected included member-specific data on age, sex, mortality-causing conditions, and risk factors, as well as a commonly used measure of self-reported heath (“Would you say that in general your health is: poor/fair/good/very good/excellent?”) that has a validated conversion to estimating quality adjustments necessary for calculating QALYs.⁷ For risk factor and condition variables for which member-specific data were unavailable, the study team assigned risks/conditions based on average US prevalences by age and sex. Forty-six percent of the population required at least 1 imputed value (EHR data were missing for smoking [9.4%], physical inactivity [5.8%], diet [20.3%], and flu vaccination [34.4%]). As a comparator group, an age- and sex-matched cohort of the US population was created. Individuals in the US cohort had risks and conditions based on average US prevalences by age and sex (Table 1).

Table 1.

Population Summary Statistics

	Total KP population N (%)	KP Los Angeles subpopulation N (%)	KP San Diego subpopulation N (%)	US population* N (%)
Overall	96,246 (100%)	56,658 (100%)	39,588 (100%)	96,246 (100%)
Sex
Male	25,264 (26.2%)	14,412 (25.4%)	10,852 (27.4%)	25,264 (26.2%)
Female	70,982 (73.8%)	42,246 (74.6%)	28,736 (72.6%)	70,982 (73.8%)
Age group (years)
<25	8133 (8.4%)	5005 (8.8%)	3128 (7.9%)	8133 (8.4%)
25–44	41,341 (43.0%)	25,255 (44.6%)	16,086 (40.6%)	41,341 (43.0%)
45–64	39,611 (41.2%)	22,642 (40.0%)	16,969 (42.9%)	39,611 (41.2%)
65+	7161 (7.4%)	3756 (6.6%)	3405 (8.6%)	7161 (7.4%)
By risk factor
High cholesterol	17,336 (18.0%)	9791 (17.3%)	7545 (19.1%)	57,940 (60.2%)
Depression	7376 (7.7%)	4015 (7.1%)	3361 (8.5%)	7376 (7.7%)
Bipolar	17 (0.02%)	8 (0.01%)	9 (0.02%)	3369 (3.5%)
Anxiety	9830 (10.2%)	5569 (9.8%)	4261 (10.8%)	27,719 (28.8%)
Diabetes	7910 (8.2%)	4798 (8.5%)	3112 (7.9%)	8951 (9.3%)
Hepatitis	680 (0.71%)	409 (0.7%)	271 (0.7%)	1155 (1.2%)
Hypertension	19,975 (20.8%)	11,583 (20.4%)	8392 (21.2%)	28,008 (29.1%)
HIV	45 (0.05%)	21 (0.04%)	24 (0.06%)	289 (0.3%)
Obese (BMI ≥30 kg/m2)	16,221 (16.9%)	9633 (17%)	6588 (16.6%)	43,696 (45.4%)
Current smoker	6451 (6.7%)	3626 (6.4%)	2794 (7.1%)	17,132 (17.8%)
No flu vaccine	14,103 (14.7%)	7981 (14.1%)	6118 (15.5%)	58,518 (60.8%)
Physical inactivity	62,064 (64.5%)	37,205 (65.7%)	24,859 (62.8%)	76,131 (79.1%)
Unhealthy diet	78,443 (81.5%)	46,527 (82.1%)	31,876 (80.5%)	89,846 (93.35%)

^*Represents a sex- and age-matched cohort.

BMI, body mass index; KP, Kaiser Permanente.

Simulation validation

In order to be able to both calibrate as well as validate the simulation, 2 separate populations are required. It is recommended that model validation be performed on a distinct data set compared to model calibration. If one larger data set is partitioned into 2 subsets, it is preferred that this partitioning should be independent of the outcome measure or predictors, yet not entirely random. For this reason, the study team used geographic criteria to partition the larger data set into 2 subsets, one that serves as the calibration or “development” cohort and the other that serves as the validation cohort. Once validation was assessed, the team reunited the 2 separate cohorts to complete the final analyses. To establish a development cohort (eg, input(s) in the model may be varied in pursuit of best performance) and a validation cohort (eg, model performance assessed without additional changes), the complete member population was divided into 2 groups of approximately similar population sizes divided along geographic regions approximating the Los Angeles (N = 56,658) and San Diego (N = 39,588) metropolitan areas and their surrounding regions. The populations were divided to create a more realistic representation of 2 separate populations rather than a random partitioning, which would be more likely to lead to 2 similar populations. The cohort characteristics can be found in Table 1. The simulation was calibrated to KP data using longitudinal clinical data of 5-year survival from the Los Angeles cohort, which was randomly designated as the development cohort. The simulation was calibrated to minimize the difference between model predictions of 5-year mortality and clinical 5-year mortality data. To avoid overfitting the model,⁸ the team sought to vary as few inputs as possible during calibration—needing to vary only 1 variable (self-reported health) to obtain good model fit.

The prespecified test for model validation was the ability of the simulation to predict 5-year mortality compared to observed mortality in the San Diego cohort. The simulation has been validated previously for predicting life expectancy and condition-specific mortality compared to the 2014 US life tables (Supplementary Data).^3,9 The simulation's performance was assessed for discrimination for both the Los Angeles and San Diego cohorts using an area under the curve (AUC) C-statistic, and for calibration by comparing estimated with observed rates of 5-year mortality across age- and self-reported health strata. Once validation tests were satisfactory, the model was used to estimate the longer term outcomes that were of greatest interest to the health system (LY and QALY, and metrics calculated from these, such as potential years of life gained and potential QALYs gained). QALYs measure trade-off-based preferences between quality of life and mortality, and are calculated by summing health utilities over time, where a health utility is represented on a scale of 0 (death) to 1 (perfect health). A reference health utility for each individual was calculated based on an individual's age and self-reported health and was varied as the individual ages in accordance with the Beaver Dam study.¹⁰ To capture the impact of risk factors that do not cause mortality directly but impact quality of life, risk factors that were highly prevalent, contributed substantially to quality of life, and have utility values available were incorporated, including alcohol abuse,¹¹ depression disorder,¹² bipolar,¹³ anxiety,¹⁴ diabetes,¹⁵ HIV,¹⁶ and obesity,¹⁷ and were given utility decrements based on the literature. When a disutility-causing factor was present, the QALYs accrued decreased, and if a factor resolved, the QALYs accrued returned to their baseline value. In this proof-of-concept analysis, only a limited number of conditions/risks were assigned utility decrements. Ideally, in a full analysis, all conditions and risk factors would be given utility values.

Estimating potential health gained

Potential years of life gained was defined as LYs with risk(s) removed minus actual LYs. Potential years of life gained was estimated for both the combined KP cohort using member data, as well as for a matched US cohort. To determine potential years of life to be gained from removing risk factors in the member population, baseline life expectancy was compared with life expectancy under the counterfactual scenario of removing each modifiable risk factor. Modifiable risk factors were defined as those an individual and/or health system has the potential to attenuate or eradicate (eg, smoking, obesity) versus non-modifiable risk factors that were not (eg, family history, BRCA 1/2 genes). As a secondary analysis, potential quality-adjusted years of life gained also were also calculated based on QALYs.

Results

Data were obtained for 96,246 individuals in KP Southern California. KP members were generally healthier than the US population, with 6.7% smokers and 16.9% obese compared with 17.8% and 45.4% in the US cohort, respectively. Population summary statistics are presented in Table 1. Note that the Los Angeles and San Diego areas were only analyzed separately for model validation and were combined for all other analyses.

Simulation validation

When stratified by age and self-reported health, the simulation performed well at predicting 5-year mortality in both the calibration and validation populations (Fig. 1A, B). When predicting 5-year member mortality in the Los Angeles calibration population, the simulation had an AUC of 0.88. When stratified by age group, the AUC for members under 34 years of age, between 34 and 49 years of age, and 49 years or older were 0.63, 0.77, and 0.84, respectively. The 10th and 90th percentile age groups had an AUC of 0.83 and 0.49, respectively. When predicting 5-year member mortality in the San Diego validation population, the simulation had an AUC of 0.89. When stratified by age group, the AUC for members younger than 34 years of age, between 34 and 49 years of age, and 49 years or older were 0.73, 0.83, and 0.83, respectively. The 10^th and 90^th percentile age groups had an AUC of 0.81 and 0.92, respectively.

FIG. 1.

Five-year mortality rate, observed and simulation outcome for simulation (A) calibration using Los Angeles cohort and (B) validation using San Diego cohort.

Potential years of life gained

Figure 2A shows the estimated average number of LYs lost to preventable death in the KP and US populations. At baseline, the US population and the KP member population were estimated to have average life expectancies of 83.2 years and 84.7 years, respectively. The US cohort age distribution is significantly older than the standard US population because it is age and sex matched to the KP cohort and therefore the life expectancy of the cohort is higher than the average US life expectancy.

FIG. 2.

Estimated impact of eliminating risk factors on average (A) life years gained and (B) quality-adjusted life years gained. HTN, hypertension; KP, Kaiser Permanente; QALY, quality-adjusted life year.

When comparing potential health gain in the US cohort to the KP cohort, eliminating physical inactivity, unhealthy diet, smoking, uncontrolled diabetes, and high cholesterol resulted in increases of 1.5 LY vs. 1.3 LY, 1.1 LY vs. 0.8 LY, 0.5 LY vs. 0.2 LY, 0.5 LY vs. 0.5 LY, and 0.4 LY vs. 0.2 LY on average per person, respectively (Fig. 2A). Note that these estimates are not necessarily additive because of interactions between risk factors and because of the nonlinear impact of mortality rate multiplication on life expectancy.

Potential QALYs gained

Figure 2B shows the average estimated number of QALYs lost to preventable causes of death in the US and KP populations. At baseline, the US population and the KP member population were estimated to have average quality-adjusted life expectancies of 68.2 and 69.2 QALYs, respectively.

When adjusting for quality of life and comparing the US cohort to the KP cohort, eliminating obesity, physical inactivity, uncontrolled diabetes, anxiety, and unhealthy diet resulted in an increase of 3.1 vs. 1.2 QALYs, 1.1 vs. 0.9 QALYs, 0.8 vs. 0.7 QALYs, 0.8 vs. 0.3 QALYs, and 0.8 vs. 0.6 QALYs on average per person, respectively (Fig. 2B).

Sensitivity analyses comparing the impact of level of evidence restrictions demonstrated that the inclusion of lower levels of evidence generally increased predicted heath improvements (Fig. 3A, B).

FIG. 3.

Inclusion of all levels of evidence estimated impact on eliminating risk factors on average (A) life years gained and (B) quality-adjusted life years gained. HTN, hypertension; KP, Kaiser Permanente; LY, life year; QALY, quality-adjusted life year.

Discussion

Prioritizing intervention targets and measuring progress toward achieving population health goals remain challenges for integrated health care systems and public health decision makers. Using a simulation, the study team was able to predict the potential health gain achieved by eliminating risk factors in a patient population, as well as to replicate population mortality rates. By indicating which domains have the greatest potential impact on health, this type of tool has the potential to be used to prioritize interventions.

Determining where the greatest potential health gains can be achieved may be a useful tool for decision makers when weighing the benefits of alternative intervention foci. In this integrated health system member population, the simulation predicted that the greatest amount of health gain in LYs could be achieved by reducing the prevalence of risk factors including physical inactivity, unhealthy diet, uncontrolled diabetes, and smoking.

The greatest potential health benefit in QALYs could be achieved by reducing the risk factors obesity, physical inactivity, depression, and uncontrolled diabetes. These population-specific estimates can be used to select target intervention areas, when considered in conjunction with probable adherence, effectiveness, and cost-effectiveness of relevant interventions.

The KP population had a higher estimated life expectancy than the US population becaue of fewer current smokers and lower levels of obesity. Accordingly, the total potential LYs to be gained is greater in the US population than in the KP population. Additionally, the difference in risk factor prevalence resulted in designating different priority areas for interventions between the US and KP populations. To maximize life expectancy, while physical inactivity and diet are top priority areas for both populations, smoking is one of the top 3 priorities in the US population, whereas attenuating diabetes rounds out the top 3 priorities for KP. Because of the distinct results when different measures are considered (LYs vs. QALYs), it is necessary to consider whether mortality alone or mortality combined with morbidity is the outcome most aligned with the priorities of health system members.

Others have used simulations and other methods to estimate potential LYs gained by reducing risk factors in a population,^4,18,19 and to determine the optimal treatment course for an individual based on her/his risk factors,²⁰ but the present simulation is notable in its ability to predict future impacts of risk factors on mortality-causing conditions. Simulations such as those used by Maciosek et al⁴ and Farley et al¹⁸ and regression analysis as used by Kindig et al¹⁹ are insufficient to capture the dynamic interaction that exists between risk factors and mortality-causing conditions. While also predicting future impacts of risk factors on mortality-causing conditions, Eddy et al²⁰ differs from the present analysis because it is less applicable for decision making on the population level. Similarly, although able to examine many key risk factors and conditions in and across populations, the US Burden of Diseases² is unable to incorporate the health status of a real-life patient cohort.

This simulation also addresses some of the limitations of using amenable mortality as a health care indicator as described by Mackenbach et al.²¹ Amenable mortality, which represents the mortality attributable to risk factors and conditions responsive to health care intervention,²² is similar to what the present study has called “potential health gain,” but relies on retrospective mortality data (eg, the deaths of today) rather than calculating downstream health impact (eg, the deaths of tomorrow).

This proof of concept illustrates many scenarios for modeling the health of a large integrated health system. First and most importantly, by assessing population life expectancy repeatedly over time, it can be observed whether the health system is improving its population's health. Indeed, one may observe that the primary goal of a health system is to produce health, so measuring overall health should be the paradigmatic metric of performance. Further, increments or decrements in health could potentially be used as the basis for a health-based criterion for reimbursement for health systems rather than a volume-based reimbursement (eg, a “Health Value Unit” rather than a “Resource-Based Relative Value Unit”). A reimbursement linkage may be necessary to incentivize health organizations to track the aggregate health of their populations. Second, considering competing mortality risks creates the ability to run innumerable “experiments” across diverse populations, including populations with unique risk factor profiles that would otherwise be infeasible. Third, because the simulation allows for customization based on population characteristics, it can be used to compare the health states of different populations to one another. Finally, the simulation was created in collaboration with stakeholders, allowing for a more thorough exploration of the functionalities that are useful for actual decision makers.

Limitations

This study has several limitations. First, the simulation was only validated over 5 years because this was the longest duration of follow-up for which adequate data were available.

Second, the age- and sex-matched US cohort used in the analyses represents an imperfect control for the health system population and as a result differences between the 2 populations, although suggestive of health system health benefits, cannot be confidently credited to the health system. Correspondingly, the estimates derived from the KP population studied in this proof of concept are unlikely to apply directly to other populations because KP members were generally healthier than the average in the US. Similarly, although the KP member prevalence of chronic conditions such as obesity and hypertension closely reflect the lower prevalence of these conditions in California as a whole,²³ the generally lower prevalence of chronic conditions overall could reflect a true difference in prevalence in the KP population or could represent poor data quality or alternate measurement standards that could impact the validity of results produced from the simulation. However, as this proof of concept seeks to demonstrate the potential utility of the simulation to assess health change within a health system, if the measure errors remain consistent within the system over time, this simulation still should be a potentially valid method to assess health change over time for goal setting within the health system.

As the KP member data were restricted to those who had a self-reported health response, it is possible that nonresponse bias occurred in the selection of the cohort used in these analyses. The Self-Reported Health questionnaire is standard practice and therefore non-completion may be suggestive of poorer health. Similarly, member data were selected from the available EHR and did not capture member visits that occurred outside of the KP health system and may not represent a complete data source. This may have decreased the predictive strength of the simulation; however, when compared to administrative data, prior reports using the EHR alone have been shown to still have acceptable accuracy and predictive value.²⁴

Third, this simulation only considered the top causes of mortality (albeit accounting for 79% of deaths) and therefore may underestimate the impact of risk factor reduction on life expectancy. Therefore, the simulation results do not to apply to persons with mortality-causing conditions not included in the simulation (eg, systemic lupus erythematosus) or to frail individuals. However, for future work, the structure of the simulation is flexible and capable of accommodating additional risk factors or conditions. Additionally, the simulation validation performed less well among individuals older than age 50 years reporting poor health, which represents a potentially key demographic in terms of health care resource use. Therefore, this simulation may have less predictive value for this age group and simulation modifications may need to be considered if using this tool specifically in this population.

Fourth, the scope of this proof of concept was limited and did not permit assessment of QALY weightings for every relevant health condition. This limitation likely underestimates the health impact of QALYs gained by reducing risk factors and conditions for which QALY weightings were unavailable.

Fifth, similar to other estimation approaches for health metrics, this simulation is limited by the data that are used in it. Not every member had complete risk factor and condition data and US averages were used for imputation when these data were missing (46% had at least 1 value imputed). This likely overestimates the impact of reducing risk factors as the US average risk factor prevalences are higher than in the health system population.

Finally, interactions between health conditions, risk factors, and health gain are immensely complex and still incompletely understood. Previously developed measures, such as comorbidity scores,^25–27 have shown the predictive value of quantifying the prevalence of multiple conditions. By simulating individuals with these comorbid conditions, the simulation seeks to capture more precisely, based on presence or absence of individual comorbidities, the impact on health that comorbidity scores quantify. Although simulation results likely represent a good estimate of potential health gains, the intricacies of these relationships are incompletely captured in this proof-of-concept analysis and further improvements that capture correlations in condition development/resolution and synergistic effects of comorbidities may improve the performance of this simulation.

Conclusion

This proof of concept has estimated the potential population health gains possible by targeting various risks in the member population of a large integrated health system. Mathematical simulations such as this one can be used to improve efforts to target population health and evaluate health system performance.

Author Disclosure Statement

The authors declare that there are no conflicts of interest. This study was funded in part by Kaiser Permanente Southern California and in part by the New York University School of Medicine Department of Population Health.

Supplementary Material

Supplementary Data