Methodological Challenges of Measuring Primary Care Delivery to Pediatric Medicaid Beneficiaries Who Use Community Health Centers

Abstract

Efforts to measure quality of care have focused on ambulatory care providers. We examined the performance of community health centers serving children on Medicaid in 3 states. Descriptive analysis showed considerable patient population heterogeneity, and regression analysis demonstrated that variation explained by the assigned provider was small (mean R² = 4.3%) compared with the variation explained by patient demographic variables (mean R² = 29.9%). The results reinforce the need for caution when one is attributing quality differences to provider performance.

Efforts to measure the performance of ambulatory care providers have encountered methodological challenges.^1–4 Patient characteristics have been shown to be an important determinant of outcomes, making risk adjustment difficult.^5–8 The relative contribution of provider-level factors in explaining performance variability has appeared low.^9,10 We used administrative data to explore patient- and provider-level determinants of performance variability within the community health center (CHC) delivery system.

METHODS

Our data source was 2003–2004 Medicaid Analytic Extract files for 3 states (California, Massachusetts, and Texas). We restricted the study sample to Medicaid beneficiaries aged 18 years or younger who had at least 6 months of fee-for-service eligibility over 2 years, and whose usual source of care was a CHC. We defined usual source of care as the provider that rendered more than 75% of primary care visits in the 2-year study period.¹¹ To ensure adequate sample size, we included only CHC providers that had more than 200 patients assigned by our algorithm.

We examined 2 quality measures based on the concept of ambulatory-care–sensitive conditions^12,13 (avoidable emergency department visits and avoidable hospitalizations) and 2 cost measures (inpatient care costs and outpatient care costs). Cost measures were specified as the log-transformed cost per beneficiary per month. Inpatient costs included inpatient, emergency department (ED), and other acute care; outpatient costs included outpatient visits and services, home-based services, laboratory services, and pharmacy.

Using descriptive statistics, we examined demographic characteristics for each CHC’s assigned population. We then performed patient-level ordinary least squares linear regression on costs, and patient-level logistic regression on the avoidable event, specifying robust standard errors clustered at the provider level. We assigned explanatory variables CHC, age, race or ethnicity (as 5 dichotomous race/ethnicity options), a dichotomous disability status, and a case mix control variable constructed per beneficiary by using the Johns Hopkins Adjusted Clinical Group algorithm.¹⁴ We ran each state as a separate sample.

To examine the contribution of assigned CHC in the regression model, we ran each model 3 times: (1) including all explanatory covariates listed previously, (2) including only patient-level covariates, and (3) including only CHC dummy covariates. We report the difference between the full model and the subset specifications to measure the contribution of provider and patient variables 2 ways: as the sole covariates (termed CHC covariates), or when subtracted from the full model (termed CHC increment). F-tests for model significance are included. For the ordinary least squares regressions, we state the adjusted R² value. For the logistic avoidable event regressions, we report the McFadden’s pseudo-R² value.¹⁵

RESULTS

We included 85 CHCs (45 from California, 24 from Texas, and 16 from Massachusetts) with a total of 90 512 assigned patients in the study sample. Table 1 presents selected demographic characteristics. Across the states, the mean coefficient of variation for race/ethnicity parameters was 1.04 and for percentage disabled was 0.86, indicating high variability among CHCs on several measured demographic variables.

TABLE 1—

Key Characteristics of Community Health Center–Assigned Pediatric Populations of Medicaid Beneficiaries in California, Massachusetts, and Texas: 2003–2004

Population Variable	Mean (SD)	Range	CV
California (45 CHCs)
Age, y	6.1 (1.8)	3.2–10.0	0.29
ADG count	3.6 (0.4)	2.8–5.1	0.12
Disabled, %	5.0 (4.9)	0.0–27.1	0.98
Race/ethnicity, %
White	24.8 (23.5)	2.3–83.8	0.95
Black	6.9 (10.2)	0.0–41.8	1.48
Asian	1.4 (1.6)	0.0–7.5	1.11
Hispanic	55.5 (24.0)	4.3–91.7	0.43
CHC assigned patient count	1276 (1349)	204–5655	1.06
Massachusetts (16 CHCs)
Age, y	8.5 (1.0)	6.9–9.9	0.12
ADG count	3.3 (0.3)	2.8–4.0	0.09
Disabled, %	8.6 (7.2)	0.4–29.6	0.84
Race/ethnicity, %
White	21.8 (12.0)	0.8–52.6	0.55
Black	14.2 (17.8)	0.1–66.7	1.25
Asian	9.5 (18.6)	0.1–72.2	1.94
Hispanic	28.0 (20.9)	0.1–64.1	0.75
CHC assigned patient count	707 (541)	202–1765	0.76
Texas (24 CHCs)
Age, y	6.8 (1.9)	2.1–10.2	0.28
ADG count	3.4 (0.4)	2.6–4.2	0.11
Disabled, %	2.3 (1.7)	0.0–6.2	0.75
Race/ethnicity, %
White	13.9 (15.2)	0.3–58.8	1.09
Black	10.2 (13.1)	0.0–38.2	1.28
Asian	0.2 (0.3)	0.0–0.9	1.25
Hispanic	75.0 (25.3)	24.8–99.6	0.34
CHC assigned patient count	908 (835)	210–3632	0.92

Note. ADG = adjusted diagnostic group; CHC = community health center; CV = coefficient of variation.

Table 2 presents the regression R² results, showing that patient-level covariates accounted for the majority of explanatory power in all models. The R² values for the full models ranged from 21.3% (Texas, avoidable hospitalizations) to 47.9% (Texas, outpatient cost). The R² explained by the assigned CHC—calculated as the average of the CHC covariates model and the value of the CHC increment—ranged from 1.8% (Texas, avoidable hospitalizations) to 12.2% (Texas, avoidable ED visits). Massachusetts models had the lowest R² explained by assigned CHC (mean = 2.0%), whereas Texas had the highest (mean = 6.0%). In general, avoidable hospitalization and mean inpatient cost had lower proportions of variation explained by the assigned provider, and outpatient cost and avoidable ED visits had higher proportions explained by the assigned provider.

TABLE 2—

Comparison of Explanatory Power of Community Health Center and Patient Parameters in Regression Models for Cost and Avoidable Event Dependent Measures: California, Massachusetts, and Texas, 2003–2004

Model	California, R2	Massachusetts, R2	Texas, R2
Avoidable ED visita
1a. Full model (all covariates)	30.2**	32.8**	35.3**
1b. Patient-level covariates only	25.5**	29.5**	23.4**
1c. CHC covariates only	6.0**	4.9**	12.6**
1d. CHC increment (1a–1b)	4.8**	3.4**	11.9**
1e. Patient increment (1a–1c)	24.2**	27.9**	22.8**
Avoidable hospitalizationa
2a. Full model (all covariates)	24.5**	23.0**	21.3**
2b. Patient-level covariates only	22.4**	21.6**	19.2**
2c. CHC covariates only	3.5**	1.6*	1.4**
2d. CHC increment (2a–2b)	2.1**	1.4	2.1**
2e. Patient increment (2a–2c)	20.9**	21.4**	19.9**
Inpatient costb
3a. Full model (all covariates)	25.2**	37.8**	45.6**
3b. Patient-level covariates only	23.8**	37.6**	42.9**
3c. CHC covariates only	2.0**	0.9**	6.7**
3d. CHC increment (3a–3b)	1.4**	0.2**	2.7**
3e. Patient increment (3a–3c)	23.2**	37.0**	38.9**
Outpatient costb
4a. Full model (all covariates)	47.5**	38.5**	47.9**
4b. Patient-level covariates only	39.2**	37.2**	44.3**
4c. CHC covariates only	9.7**	2.7**	7.0**
4d. CHC increment (4a–4b)	8.3**	1.3**	3.6**
4e. Patient increment (4a–4c)	37.8**	35.9**	40.9**

Note. CHC = community health center; ED = emergency department. All data are in percentages. We constructed CHC increment value in 1d, 2d, 3d, and 4d by subtracting model b from the full model (a). We constructed patient increment value in 1e, 2e, 3e, and 4e by subtracting model c from the full model (a). P values denote F-test results for model significance (a–c), or F-test of significance of adding CHC or patient increments (d and e).

^aLogistic regression—McFadden’s pseudo R² values.

^bOrdinary least squares regression—adjusted R² values.

*P < .05; **P < .001.

DISCUSSION

Previous work has detailed the methodological challenges of evaluating the performance of physicians and other provider types.^10,16,17 This analysis extends those findings to evaluation of quality and cost at CHCs. This study had several limitations. It was performed on administrative data, which constrained the type and number of quality measures available. In addition, the study was focused on a pediatric population with Medicaid insurance in California, Massachusetts, and Texas, and may not apply to other populations.

As payment reform is implemented, it is essential that systems reward good performance rather than patient population differences.^18,19 The extreme ranges in patient characteristic distributions in Table 1 show clustering among CHCs, and our regression models explained less than half the variance in quality and cost. The unexplained variance and heterogeneity across CHCs suggest that their patient populations likely also differ in unmeasured characteristics (e.g., homelessness, lack of social support, severity of diagnoses) that may be associated with increased costs and utilization.^20–22 In recent research, physicians with higher proportions of poor, underinsured, and minority patients on their panels were found to achieve lower quality scores.^6,23 If pay-for-performance schemes do not adequately adjust for case mix, they may penalize providers who serve safety-net populations.^24–26 The ongoing challenge in payment reform is to identify methodologies that measure and reward true differences in ambulatory care provider performance.