. 2020 Jul 22;98(3):847–907. doi: 10.1111/1468-0009.12468

Table 2.

Approaches to Assess Risk Selection Contributing to Bonuses but Removed in Evaluation

Selection Strategy	Analytic Approach to Assess Extent of Selection	Interpretation
Reconfiguration of ACOs to favor lower‐cost providers	Compare savings estimates from main approach holding ACO TINs or clinician NPIs constant vs. approach allowing changes in provider composition after first performance year	Greater savings estimated when allowing provider turnover would suggest selective inclusion of lower‐cost providers. Since ACO benchmarks adjust for changes in TINs, only increases in savings from within‐TIN changes in clinician composition could contribute to bonuses and constitute risk selection.
Gaming of CMS attribution algorithm via manipulation of TINs used for billing	1. Compare savings with and without adjustment for patient characteristics when employing the CMS attribution algorithm 2. Using CMS attribution algorithm, compare savings when holding ACO composition fixed as sets of TINs vs. NPIs	1. Attenuation of savings by patient covariate adjustment in analysis using the CMS attribution rules but not in our main approach (using only PCP office visits for attribution) would suggest risk selection enabled by the CMS rules that was removed in our evaluation but could have contributed to bonuses. 2. If ACOs strategically changed the TINs used for billing by member clinicians to induce attribution of lower‐cost patients, savings estimates should be attenuated by holding ACOs constant as sets of NPIs.
Exclude high‐risk patients or physicians with high‐risk patients	Compare differences in exit from ACO‐attributed populations between higher‐ vs. lower‐risk patients (or differences in exit from ACO TINs between PCPs with higher‐ vs. lower‐risk patients) with analogous differences in patient or PCP exit among non‐ACO TINs	Disproportionate exit of higher‐risk patients or PCPs with higher‐risk patients that is greater for ACOs than non‐ACO providers would suggest potential risk selection (a necessary but not sufficient observation).

Selection Strategy

Analytic Approach to Assess Extent of Selection

Interpretation

Reconfiguration of ACOs to favor lower‐cost providers

Compare savings estimates from main approach holding ACO TINs or clinician NPIs constant vs. approach allowing changes in provider composition after first performance year

Greater savings estimated when allowing provider turnover would suggest selective inclusion of lower‐cost providers. Since ACO benchmarks adjust for changes in TINs, only increases in savings from within‐TIN changes in clinician composition could contribute to bonuses and constitute risk selection.

Gaming of CMS attribution algorithm via manipulation of TINs used for billing

1. Compare savings with and without adjustment for patient characteristics when employing the CMS attribution algorithm

2. Using CMS attribution algorithm, compare savings when holding ACO composition fixed as sets of TINs vs. NPIs

1. Attenuation of savings by patient covariate adjustment in analysis using the CMS attribution rules but not in our main approach (using only PCP office visits for attribution) would suggest risk selection enabled by the CMS rules that was removed in our evaluation but could have contributed to bonuses.

2. If ACOs strategically changed the TINs used for billing by member clinicians to induce attribution of lower‐cost patients, savings estimates should be attenuated by holding ACOs constant as sets of NPIs.

Exclude high‐risk patients or physicians with high‐risk patients

Compare differences in exit from ACO‐attributed populations between higher‐ vs. lower‐risk patients (or differences in exit from ACO TINs between PCPs with higher‐ vs. lower‐risk patients) with analogous differences in patient or PCP exit among non‐ACO TINs

Disproportionate exit of higher‐risk patients or PCPs with higher‐risk patients that is greater for ACOs than non‐ACO providers would suggest potential risk selection (a necessary but not sufficient observation).

Abbreviations: ACO, accountable care organizations; CMS, Centers for Medicare and Medicaid Services; NPI, national provider identifier; PCP, primary care physician; TIN, tax identification number.