Impact Of Risk Adjustment For Socioeconomic Status On Medicare Advantage Plan Quality Rankings

Abstract

Sociodemographically disadvantaged patients have worse outcomes on some quality measures that inform Medicare Advantage plan ratings. Performance measurement that does not adjust for sociodemographic factors may penalize plans that disproportionately serve disadvantaged populations. We assessed the impact of adjusting for socioeconomic and demographic factors (sex, race/ethnicity, dual eligibility, disability, rurality, and neighborhood disadvantage) on Medicare Advantage plan rankings for blood pressure, diabetes, and cholesterol control. After adjustment, 20.3 percent, 19.5 percent, and 11.4 percent of Medicare Advantage plans improved by one or more quintiles in rank on the diabetes, cholesterol, and blood pressure measures, respectively. Plans that improved in ranking after adjustment enrolled higher proportions of disadvantaged enrollees. Adjusting quality measures for socioeconomic factors is important for equitable payment and quality reporting. Our study suggests that plans serving disadvantaged populations would have improved relative rankings for three important outcome measures if socioeconomic factors were included in risk-adjustment models.

Medicare Advantage plans are private managed care plans that receive capitated payments to provide Medica-recovered services. In 2016 the plans insured 17.6 million people, or 31 percent of the Medicare population.¹ To improve the quality of care for enrollees, the Centers for Medicare and Medicaid Services (CMS) publicly reports and compensates Medicare Advantage plans on the basis of a composite five-star rating that reflects each plan’s performance on more than thirty quality measures.² Before 2017, star ratings were not adjusted for any enrollee characteristics,³ though plans serving larger proportions of sociodemographically disadvantaged enrollees might have had lower performance on these measures.^4,5

Adjusting measures of clinical performance for socioeconomic factors has been a controversial topic, yet socioeconomic disadvantage has been shown to affect health independent of health system performance.⁶ Incorporating adjustments for a comprehensive set of sociodemographic factors, including socioeconomic status, could produce fairer comparisons of Medicare Advantage plans’ performance and reduce incentives to avoid caring for disadvantaged populations.^3,7 However, critics have raised the concern that risk adjustments could mask disparities by adjusting away any lower-quality care provided to disadvantaged populations and setting a lower standard for providers that disproportionately serve them.^3,7

After consideration of these issues, expert groups including the National Quality Forum; the Department of Health and Human Services; and the National Academies of Sciences, Engineering, and Medicine recently endorsed the notion that performance measurement should consider socioeconomic status and other demographic factors.^8–10 These three groups have recommended that quality measures be individually evaluated for socioeconomic status adjustment, and they note that adjustment may be warranted if sociodemographic factors are known to affect performance for the measure of interest.⁹ Yet since socioeconomic status is a multidimensional construct that consists of both individual and environmental factors, there is ongoing deliberation and a lack of empirical evidence about how to adjust measures and which factors to include in adjusted models.^8–10 Additionally, high-quality socioeconomic indicators are not widely available in Medicare and other insurance data.⁸

In 2017 CMS began adjusting star ratings for dual eligibility (both Medicaid and Medicare) and disability, using an interim measure called the Categorical Adjustment Index. This adjustment is applied to the overall star rating based on the proportion of dually eligible or disabled enrollees in each plan.¹¹ A study of the impact of these adjustments found little change in adjusted plan rankings: Over 96 percent of plans experienced no change in star rankings.¹² However, adjustment for only these two factors, which are indirect measures of socioeconomic status, may be insufficient to account for differences in the characteristics of enrollees across plans.

We evaluated changes in Medicare Advantage plans’ relative performance in three measures—blood pressure, diabetes, and cholesterol control—after adjusting for selected widely available individual- and area-level sociodemographic factors that are not included in CMS’s risk-adjustment methodology.⁸ We further assessed the enrollee composition of the plans whose relative rankings improved following adjustment.

Study Data And Methods

DATA SOURCES AND STUDY POPULATION

We obtained person-level Medicare Healthcare Effectiveness Data and Information Set (HEDIS) data for 2012 and 2013 from CMS. These data included information about 605,208 and 623,363 enrollees in 525 and 522 plans, respectively, who were eligible for one or more measures of blood pressure, diabetes, and cholesterol control. The Master Beneficiary Summary File provided sociodemographic characteristics. Using enrollees’ nine-digit ZIP codes, we linked the data to a measure of neighborhood disadvantage and to a rurality indicator.¹³

We excluded observations that could not be merged to the Master Beneficiary Summary File (n = 3,486), enrollees who resided outside the US (n = 29,201), those who died in the measurement year (n = 3,182), those in plans with fewer than one hundred enrollees (n = 7,456), and those in two plans that reported an implausibly high rate (100 percent) of blood pressure control (n = 5,044). Each year of data was then divided into three separate data sets by enrollees’ eligibility for individual outcome measures. The final sample sizes for each outcome measure ranged from 175,229 to 269,789 enrollees in 379–512 plans. (Additional details on the construction of the study cohort and sample sizes for each outcome measure and year are in online appendix A1.)¹⁴

STUDY VARIABLES

Our primary outcomes, measured at the patient level, included dichotomous measures of blood pressure controlled to below 140/90 mmHg for enrollees with hypertension, hemoglobin A1c controlled to below 9.0 percent for enrollees with diabetes, and low-density lipoprotein cholesterol controlled to below 100mg/dl for enrollees with a prior-year cardiac event (for example, a myocardial infarction). We chose these dichotomous variables because they have been included as quality measures in the calculation of star ratings and are significantly affected by sociodemographic factors.^15,16 Also, intermediate outcome measures such as these are more consistently affected by social risk factors than are process measures.¹⁵

Independent variables included sex, race/ethnicity, dual eligibility, disability, rurality, and neighborhood disadvantage. These sociodemographic variables have been associated with the outcomes of our study.¹⁰ Race/ethnicity was measured using the Research Triangle Institute race code, which improves the identification of Asian and Hispanic enrollees, compared to the traditional measure of race/ethnicity in the Medicare enrollment database.¹⁷ Enrollees were identified as dual eligibles if they were enrolled in Medicaid or any cost-sharing program for at least one month. Enrollees were defined as disabled if they were originally enrolled in Medicare for disability, end-stage renal disease, or both. Rurality was measured using the Urban-Rural Classification Scheme from the National Center for Health Statistics.¹³ Neighborhood disadvantage was measured using the Area Deprivation Index, a composite score generated from seventeen socioeconomic variables.^6,18 We updated the score using data from the American Community Survey from 2009.¹⁹ Area Deprivation Index scores were divided into twenty groups of equal size, or ventiles, with a separate category for missing data.

STATISTICAL ANALYSIS

We used bivariate and multivariable regression models to test the association between socioeconomic status and other demographic factors with the three selected quality outcomes. We first identified significant predictors of person-level blood pressure, diabetes, and cholesterol control in 2012 data. (Results of bivariate and multivariable regression models used to identify significant predictors are shown in appendix exhibit A1.)¹⁴ We used Akaike information criterion to select the one best-fit model for each outcome (results are shown in appendix exhibit A2).¹⁴ We then applied these risk-adjustment models to person-level 2013 data, including a random effect to account for enrollees clustering in plans, to obtain predicted performance scores for each enrollee. These scores indicated the predicted outcome control (between 0 and 1) for each enrollee, after accounting for the enrollee’s sociodemographic factors. The individual scores in each plan were averaged to create a predicted plan performance score. All subsequent analyses were conducted at the plan level.

Risk-adjusted plan performance scores were calculated by multiplying the person-level national mean by the predicted plan performance score with the plan effect and dividing the result by the predicted plan performance score without the plan effect.²⁰ Spearman rho was used to assess the difference between adjusted and observed scores. We further assessed the change in score by calculating the absolute percentage-point difference between the adjusted and observed scores for each plan.

Because CMS reports Medicare Advantage plan ratings categorically, we split observed and adjusted scores into equal-size quintiles and assessed the change in quintile rank for each plan. This method differs from CMS’s clustering algorithm for assigning star ratings to HEDIS measures.²¹ Recent evidence indicates that socioeconomic status adjustment primarily improved quintile rankings for plans that were already on the threshold of achieving a higher quintile rank.¹² Thus, to provide a more detailed description of the magnitude of plan movement after adjustment, we ordered plans numerically by observed score and by adjusted score, and we determined the absolute change in ordered rank.

To assess the enrollee composition of plans that changed their quintile ranking following adjustment, in each plan we calculated the fractions of black, dually eligible, and disabled enrollees, as well as the fraction of enrollees living in the most disadvantaged (top 15 percent of neighborhood disadvantage) or most rural areas. We used one-way analysis-of-variance tests to assess the association between plans’ enrollee composition and an increase in rank of one or more quintiles after adjustment.

Lastly, we conducted sensitivity analyses that excluded the largest plan from the 2012 data set and included a fixed intercept for the plan. (Results of regression models including and excluding the largest plan are in appendix exhibit A3, and a scatter-plot comparison of the observed and adjusted rates of outcome control, including a fixed effect, is in appendix exhibit A4.)¹⁴

All analyses were conducted using Stata, version 14.2, and SAS, version 9.4. Brown University’s Institutional Review Board approved the study protocol.

LIMITATIONS

Several study limitations should be noted. First, this study was limited by a lack of generalizability beyond the three quality measures.

Second, our study could not assess the financial implications of sociodemographic adjustment because CMS uses composite star ratings to derive payment bonuses and penalties. We chose instead to focus on individual quality measures that have been associated with racial and socioeconomic disparities in prior research.^4,22,23 This study was designed not to replicate CMS methodology but instead to display the implications of including a more comprehensive set of sociodemographic measures on risk-adjusted performance in blood pressure, diabetes, and cholesterol control.

Third, the cholesterol control indicator expired as a HEDIS measure in 2015, but we included the measure given that it has been found to be associated with socioeconomic status.

Fourth, although including comorbid medical conditions and other detailed clinical data could explain variations in control and could attenuate the conditional associations of sociodemographic factors, these clinical data were not available in our study. More importantly, CMS does not adjust for clinical factors in assessing performance for HEDIS measures.

Fifth, although our study included a more comprehensive set of socioeconomic status indicators than is currently used for quality-measure adjustment in Medicare Advantage plans, we may have incompletely measured socioeconomic status. There is an ongoing debate about which factors to include in socioeconomic status adjustment models and how best to define these factors.⁸ The National Academies of Sciences, Engineering, and Medicine identified additional social factors—including social support, marital status, gender identity, and language—that may affect health outcomes.¹⁰ However, information on many of these indicators is either not collected or not readily available in existing data (as would be needed for federal policy purposes), or there is insufficient evidence for their inclusion in an adjustment.¹⁰

Lastly, our study did not address the concern that adjustment for sociodemographic variables could hide important disparities in the quality of care and set lower standards of quality for plans with more disadvantaged patient populations. This concern must be balanced by the implications of not including these factors in a risk-adjustment model. Specifically, plans that serve more vulnerable populations maybe unfairly penalized because some quality metrics are influenced by social factors that are largely outside of plans’ control.⁷ This may create incentives for plans to avoid enrolling vulnerable people.^3,7

Study Results

Exhibit 1 describes the sociodemographic characteristics of enrollees eligible for quality measures in the 2012 derivation cohort. In the 2012 cohort, 62 percent of eligible enrollees had controlled blood pressure, 79 percent had controlled diabetes, and 61 percent had controlled cholesterol. Also, 66.4 percent were white, 14.2 percent were black, 23.3 percent were dually eligible, and 31.8 percent were enrolled in Medicare because of disability (referred to as “disabled” for brevity) (data not shown). Compared to the populations of enrollees with controlled blood pressure, diabetes, or cholesterol, the uncontrolled populations had greater proportions of enrollees who were black, dually eligible, disabled, or residing in the most disadvantaged or rural areas.

EXHIBIT 1.

Socioedemographic characteristics of Medicare Advantage plan enrollees eligible for control measures in 2012

	Blood pressure		Diabetes		Cholesterol
Characteristic	Controlled (n = 110,446; 62%)	Uncontrolled (n = 67,296; 38%)	Controlled (n = 207,374; 79%)	Uncontrolled (n = 54,979; 21%)	Controlled (n = 119,844; 61%)	Uncontrolled (n = 76,677; 39%)
Female	56.2%	58.0%	51.1%	50.3%	33.0%	43.6%
Race/ethnicity
Non-Hispanic white	69.1	63.1	62.0	57.7	72.7	70.4
Black/African American	13.1	18.7	14.5	20.1	9.2	15.1
Hispanic	12.1	12.7	13.6	15.3	11.2	10.1
Asian/Pacific Islander	4.2	3.4	7.2	4.5	4.8	2.7
American Indian/Alaska Native	0.3	0.3	0.3	0.6	0.2	0.3
Other	1.0	1.0	2.0	1.5	1.6	1.1
Unknown	0.3	0.3	0.4	0.3	0.3	0.2
Dually eligiblea	26.4	29.0	21.1	34.5	15.3	26.3
Neighborhood disadvantage
Top 15% most disadvantagedb	11.7	14.0	11.6	17.5	10.4	16.1
Missing	15.9	16.7	13.7	16.8	14.5	17.3
Originally enrolled in Medicare because of disability	25.4	26.9	32.0	48.3	29.3	40.9
Most rural areasc	4.2	4.7	2.9	4.5	3.3	5.7

SOURCE Authors’ analysis of data from the Healthcare Effectiveness Data and Information Set for 2012, the Master Beneficiary Summary File for 2012, the Area Deprivation Index for 2013, and the Urban-Rural Classification Scheme of the National Center for Health Statistics.

NOTES Percentages might not sum to 100 because of rounding. All differences between controlled and uncontrolled comparisons were significant (p < 0:001).

^aFor Medicare and Medicaid.

^bNeighborhoods in the top three most disadvantaged ventiles, as measured by the Area Deprivation Index.

^cNon–core micropolitan counties, the National Center for Health Statistics’s classification for the least populated counties.

The average plan in 2012 included a mean of 15.3 percent black, 31.1 percent dually eligible, and 35.5 percent disabled enrollees, as well as 15.1 percent enrollees living in the most disadvantaged neighborhoods and 4.6 percent enrollees living in the most rural areas. (Additional plan characteristics are in appendix exhibit A5.)¹⁴

In adjusted models, sex, race/ethnicity, dual eligibility, disability, rurality, and neighborhood disadvantage were found to be significant predictors of the intermediate outcomes, with the exception of disability as a predictor of blood pressure control (appendix exhibit A1).¹⁴ Disability was thus excluded from the final multivariable logistic regression model for blood pressure. The models accounted for 0.6 percent of the variation in enrollee-level control for blood pressure, 3.8 percent for diabetes, and 4.1 percent for cholesterol. We did not find evidence of collinearity in our models.

The relationship between adjusted and observed plan scores varied across the three quality measures. (A scatterplot comparison of observed and adjusted rates of outcome control is shown in appendix exhibit A6.)¹⁴ The correlation between adjusted and observed rankings was stronger for blood pressure control (R² = 0.97) than for diabetes control (R² = 0.86) or cholesterol control (R² = 0.83). The R² value indicates how strongly the adjusted and observed rankings were correlated, when the two were compared using linear regression. Adjusting plans’ performance for sociodemographic factors resulted in a mean absolute change in score of 2.1 percentage points for blood pressure control, 4.1 percentage points for diabetes control, and 4.2 percentage points for cholesterol control.

The numbers of plans that improved their relative performance by one or more quintiles after adjustment were 52 (11.4 percent) for blood pressure, 92 (20.4 percent) for diabetes, and 74 (19.6 percent) for cholesterol (exhibit 2). (Additional details on the distribution of plan movement across quintiles are in appendix exhibit A7.)¹⁴ The numbers of plans that decreased in rank by one or more quintiles were 57 (12.5 percent) for blood pressure, 111 (24.5 percent) for diabetes, and 90 (23.8 percent) for cholesterol. Among the 375 plans eligible for all three measures, 72 declined in quintile rank for one measure, 60 declined for two measures, and 14 declined for three measures after adjustment (data not shown). Conversely, 67 plans increased in rank for one measure, 29 plans increased for two measures, and 18 plans increased for three measures after adjustment (data not shown). Those that rose into the highest (fifth) quintile included 8 (1.8 percent) of the plans in the blood pressure cohort, 31 (6.8 percent) of the plans in the diabetes cohort, and 25 (6.6 percent) of the plans in the cholesterol cohort (appendix exhibit A7.)¹⁴ Conversely, plans that dropped into the lowest quintile after adjustment included 10 (2.2 percent) of the plans in the blood pressure cohort, 18 (4.0 percent) of the plans in the diabetes cohort, and 18 (4.8 percent) of the plans in the cholesterol cohort (appendix exhibit A7.) ¹⁴ Among all eligible plans, adjustment was associated with a mean absolute change in ordered rank of 23.8 ranking positions for blood pressure control, 51.2 positions for diabetes control, and 45.6 positions for cholesterol control (data not shown).

EXHIBIT 2.

Medicare Advantage plans’ change in quintiles of rank after adjustment for patients’ sociodemographic characteristics

	Blood pressure		Diabetes		Cholesterol
Change	No. (457)	Percent	No. (453)	Percent	No. (379)	Percent
−4
−3					3	0.8
−2			6	1.3	11	2.9
−1	57	12.5	105	23.2	76	20.1
0	348	76.2	250	55.2	215	56.7
1	47	10.3	70	15.5	48	12.7
2	5	1.1	19	4.2	20	5.3
3			3	0.7	5	1.3
4					1	0.3

SOURCE Authors’ analysis of data from the Healthcare Effectiveness Data and Information Set for 2013, the Master Beneficiary Summary File for 2013, the Area Deprivation Index for 2013, and the Urban-Rural Classification Scheme of the National Center for Health Statistics.

NOTES Percentages might not sum to 100 because of rounding. Change in quintile rank was calculated by splitting observed and adjusted scores into five equal-size groups, assigning each group a number from 1 to 5, and subtracting the observed quintile rank from the adjusted quintile rank. Empty cells indicate that no plans in the outcome measure moved the number of ranks represented by that row.

Compared with plans that did not change quintile rank after adjustment for sociodemographic factors, those that improved in quintile rank enrolled higher proportions of black, dually eligible, and disabled enrollees, as well as higher proportions of enrollees living in the top 15 percent of disadvantaged neighborhoods and in the most rural areas (although this finding was not significant for the blood pressure control measure) (exhibit 3). For example, for the diabetes control measure, compared to plans that did not change rank, those that improved in rank had higher proportions of black (21.6 percent versus percent), dually eligible (56.6 percent versus 29.4 percent), and disabled (58.0 percent versus 40.2 percent) enrollees, and higher proportions of enrollees living in the most disadvantaged (22.7 percent versus 18.1 percent) and most rural (6.2 percent versus 5.1 percent) areas. Conversely, plans that declined in rank enrolled lower proportions of socioeconomically disadvantaged enrollees. For example, for the diabetes control measure, plans that declined in rank had lower proportions of black (12.8 percent versus percent), dually eligible (12.9 percent versus 29.4 percent), and disabled (28.9 percent versus 40.2 percent) enrollees, compared with plans that did not change in rank. These plans also enrolled lower proportions of enrollees living in the most disadvantaged (9.4 percent versus 18.1 percent) and most rural (1.2 percent versus 5.1 percent) areas.

EXHIBIT 3.

Medicare Advantage plans’ mean percentages of socioeconomically disadvantaged enrollees, by change in quintile after adjustment for patients’ sociodemographic characteristics

	Change in rank
	Blood pressure control (n = 457)			Diabetes control (n = 453)			Cholesterol control (n = 379)
Characteristic	Improved (n = 52)	Declined (n = 57)	No change (n = 348)	Improved (n = 92)	Declined (n = 111)	No change (n = 250)	Improved (n = 74)	Declined (n = 90)	No change (n = 215)
Black/African American	34.5%	4.2%	13.7%	21.6%	12.8%	15.8%	16.0%	6.5%	10.9%
Dually eligiblea	67.7	7.5	23.7	56.6	12.9	29.4	51.9	11.5	19.9
Originally enrolled in Medicare because of disability	44.0	16.4	25.0	58.0	28.9	40.2	48.7	27.5	35.7
Top 15% most disadvantaged neighborhoodsb	26.2	5.9	11.6	22.7	9.4	18.1	21.2	7.4	15.3
Most rural areasc	5.3d	2.2d	4.9d	6.2	1.2	5.1	7.0	1.3	4.9

SOURCE Authors’ analysis of data from the Healthcare Effectiveness Data and Information Set for 2013, the Master Beneficiary Summary File for 2013, the Area Deprivation Index for 2013, and the Urban-Rural Classification Scheme of the National Center for Health Statistics.

NOTES Mean percentages were compared using one-way analysis-of-variance tests. Change in quintile rank was calculated as explained in the notes to exhibit 2. All results were significant (p < 0:001) except where noted.

^aFor Medicare and Medicaid.

^bNeighborhoods in the top three most disadvantaged ventiles, as measured by the Area Deprivation Index.

^cNon–core micropolitan counties (explained in the notes to exhibit 1).

^dp = 0.13.

Discussion

Our findings indicate that adjusting Medicare Advantage plan quality rankings for individual- and area-level sociodemographic factors results in substantial changes in rank for cholesterol and diabetes control and more moderate changes for blood pressure control. Plans that improved in rank enrolled larger proportions, and plans that declined in rank enrolled smaller proportions, of disadvantaged beneficiaries. In addition, sociodemographic adjustment moved a small proportion of these plans into the highest or lowest quintiles, which are thresholds for the most substantial payment bonuses or penalties in Medicare’s Star Ratings system. Adjusting for sociodemographic characteristics moved fewer plans across quintile thresholds for blood pressure control than for cholesterol or diabetes control. This may be explained in part by prior evidence that shows a lack of association between disability and dual eligibility with the blood pressure control measure, while these two indicators were otherwise the strongest contributors to disparities in overall plan rankings.²⁴ Furthermore, the population of enrollees eligible for the HEDIS cholesterol control measure has more advanced heart disease than those eligible for the blood pressure control measure, given the inclusion criteria for these measures: To be eligible for the cholesterol control measure, an enrollee must have a prior-year history of a cardiac event, while eligibility for the blood pressure control measure requires only a diagnosis of hypertension.

Similar studies of hospital rankings have found more modest changes in rank after adjustment for sociodemographic indicators.^25,26 However, these studies used less comprehensive measures of socioeconomic status and assessed area-level status using less geographically discrete five-digit (compared to nine-digit) ZIP codes.²⁴ Five-digit ZIP codes capture large and demographically heterogeneous areas and are known to therefore misclassify socioeconomic status.²⁷ In contrast, we used a measure of neighborhood socioeconomic disadvantage at the more rigorously defined level of the census block group, linked at a more granular level by nine digit ZIP codes.

Our study also differed from CMS’s interim socioeconomic status adjustment method. Before implementing the Categorical Adjustment Index as an interim adjustment to the overall star rating, CMS considered directly adjusting specific measures, similar to the method used in this study.¹¹ Although applying an adjustment factor to the overall star rating has several practical advantages, including ease of adjustment and transparency of unadjusted rankings, we chose to adjust individual measures known to be associated with socioeconomic status. Expert groups have raised concerns that the interim Categorical Adjustment Index adjustment for disability and dual eligibility alone is insufficient to address the impact of patients’ sociodemographic characteristics on plan ratings.¹²

This study contributes important empirical evidence to the ongoing discussion of whether and how to adjust Medicare Advantage plan rankings and payments for socioeconomic status. We found that for two of the three measures in our study, adjusting for patient- and area-level sociodemographic characteristics substantially improved the rankings of plans that care for vulnerable populations, while rankings declined for plans that care for less vulnerable populations. Our findings suggest that lower social risk may be an important contributor to the higher rankings observed in plans that serve less disadvantaged populations. When considering whether to adjust Medicare Advantage plans’ intermediate health outcome measures, policy makers should consider further evaluating the sociodemographic predictors identified in this study. Our study demonstrates that these factors are strongly associated with performance and can be collected in Medicare enrollment data and publicly available data sets.

Conclusion

Determining whether and how to adjust Medicare Advantage plan quality measures for sociodemographic factors is critically important to equitable payment and quality reporting. Our results suggest that plans serving disadvantaged populations would have improved relative performance—and plans serving advantaged populations would have decreased relative performance—on blood pressure, cholesterol, and diabetes control measures if sociodemographic factors were included in risk-adjustment models.