Association Between Participation in a Multipayer Medical Home Intervention and Changes in Quality, Utilization, and Costs of Care

Abstract

IMPORTANCE

Interventions to transform primary care practices into medical homes are increasingly common, but their effectiveness in improving quality and containing costs is unclear.

OBJECTIVE

To measure associations between participation in the Southeastern Pennsylvania Chronic Care Initiative, one of the earliest and largest multipayer medical home pilots conducted in the United States, and changes in the quality, utilization, and costs of care.

DESIGN, SETTING, AND PARTICIPANTS

Thirty-two volunteering primary care practices participated in the pilot (conducted from June 1, 2008, to May 31, 2011). We surveyed pilot practices to compare their structural capabilities at the pilot’s beginning and end. Using claims data from 4 participating health plans, we compared changes (in each year, relative to before the intervention) in the quality, utilization, and costs of care delivered to 64 243 patients who were attributed to pilot practices and 55 959 patients attributed to 29 comparison practices (selected for size, specialty, and location similar to pilot practices) using a difference-in-differences design.

EXPOSURES

Pilot practices received disease registries and technical assistance and could earn bonus payments for achieving patient-centered medical home recognition by the National Committee for Quality Assurance (NCQA).

MAIN OUTCOMES AND MEASURES

Practice structural capabilities; performance on 11 quality measures for diabetes, asthma, and preventive care; utilization of hospital, emergency department, and ambulatory care; standardized costs of care.

RESULTS

Pilot practices successfully achieved NCQA recognition and adopted new structural capabilities such as registries to identify patients overdue for chronic disease services. Pilot participation was associated with statistically significantly greater performance improvement, relative to comparison practices, on 1 of 11 investigated quality measures: nephropathy screening in diabetes (adjusted performance of 82.7% vs 71.7% by year 3, P < .001). Pilot participation was not associated with statistically significant changes in utilization or costs of care. Pilot practices accumulated average bonuses of $92 000 per primary care physician during the 3-year intervention.

CONCLUSIONS AND RELEVANCE

A multipayer medical home pilot, in which participating practices adopted new structural capabilities and received NCQA certification, was associated with limited improvements in quality and was not associated with reductions in utilization of hospital, emergency department, or ambulatory care services or total costs over 3 years. These findings suggest that medical home interventions may need further refinement.

Professional associations, payers, policy makers, and other stakeholders have advocated the patient- centered medical home, a team-based model of primary care practice intended to improve the quality, efficiency, and patient experience of care.^1,2 In general, medical home initiatives have encouraged primary care practices to invest in patient registries, enhanced access options, and other structural capabilities in exchange for enhanced payments—often operationalized as per-patient per-month fees for comprehensive care services.^3,4 Dozens of privately and publicly financed medical home pilots are under way, and most use recognition by the National Committee for Quality Assurance (NCQA) to assess practice structural capabilities.^4,5

Recent evidence reviews suggest that early “medical home” interventions have yielded modest improvements at best in quality and patient experience, with little evidence of effects on costs of care.^6–8 However, these reviews included studies that preceded development of NCQA medical home recognition criteria and lacked significant financial support from payers, potentially limiting their applicability to current medical home efforts.^9,10 More recent evaluations have assessed medical home pilots including only 1 payer (potentially a small fraction of some practices’ patient panels) occurring over a 1- or 2-year time frame (possibly insufficient to observe effects requiring longer time frames),^11–13 or within large, integrated delivery systems atypical of most primary care practices.^14,15 We hypothesized that a multipayer medical home initiative involving a longer intervention period and substantial financial support would be more likely to be associated with measurable improvements in quality and efficiency.

Methods

We evaluated associations between implementation of the Pennsylvania Chronic Care Initiative (PACCI), a statewide multipayer medical home pilot, and changes in the quality, utilization, and cost of care. This study was approved by the RAND human subjects protection committee and the University of Pennsylvania institutional review board, with implied informed consent for practice structural surveys and waived consent for analyses of deidentified claims data.

Pilot Design

The pilot was initiated in Pennsylvania’s southeast region among volunteering small- and medium-sized primary care practices from June 1, 2008, to May 31, 2011. As detailed elsewhere,^16,17 the PACCI was designed by a coalition of payers, clinicians, and delivery systems led by the Governor’s Office of Health Care Reform and implemented as a series of regional medical home pilots, beginning with the southeast region.

Practices in southeast Pennsylvania were invited to participate by the 6 health plans (3 commercial and 3 Medicaid managed care plans) and 3 professional organizations (the Pennsylvania Academy of Family Physicians, American College of Physicians, and American Academy of Pediatrics) participating in the PACCI, with the goal of enrolling a mix of practice sizes and specialties (family practice, internal medicine, pediatric, and nurse-managed health centers).

The intervention consisted of technical assistance including a Breakthrough Series Learning Collaborative,¹⁸ web-based Improving Performance in Practice (IPIP) disease registries to create monthly quality indicator reports,¹⁹ and assistance from IPIP practice coaches to facilitate practice transformation and achievement of NCQA Physician Practice Connections–Patient-Centered Medical Home (PPC-PCMH) recognition (with level 1 recognition required by the second pilot year).²⁰ Performance improvement efforts targeted asthma for pediatric practices and diabetes for practices serving adults.

To support and motivate these improvements, each participating practice was eligible to receive a $20 000 “practice support” payment in year 1 and annual bonus payments per full-time equivalent clinician (ie, per physician or nurse practitioner) that varied based on NCQA medical home recognition level and practice size, ranging from $28 000 per clinician in NCQA level 1 practices with 10 to 20 clinicians to $95 000 per clinician in solo NCQA level 3 practices.

Comparison Practices

Within 3 months of recruiting pilot practices, the state selected comparison practices. To do this, a state contractor obtained lists of candidate practices from participating health plans and, without performing a strict 1-to-1 match, identified a group of comparison practices that had the same approximate composition as the pilot practices in practice size, specialty (pediatrics, family practice, internal medicine), location (urban, suburban), and affiliation with local health systems. Data on quality, utilization, and costs were unavailable for comparison practice selection.

Pilot Data, Practice Survey, and Claims Data

We obtained, from Pennsylvania’s demonstration conveners, annual data on each pilot practice’s level of NCQA PPC-PCMH recognition (none or level 1, 2, or 3) and amounts of each pilot practice’s bonus payments.

Drawing from a survey instrument designed to assess practice readiness for the medical home,²¹ we devised a new surveyto measure practices’ structural capabilities, including presence of performance feedback, disease management, registries, reminder and outreach systems for patients with chronic disease, and electronic health records (EHRs) (instrument avail- able from authors on request).

We fielded the practice survey twice to each pilot and comparison practice: a “baseline” survey in September 2010, querying capabilities just prior to June 2008 (the pilot’s beginning), and a final survey in June 2011, querying capabilities at the pilot’s end. We addressed each survey to 1 leader per practice, identified by telephone call, who could report accurately on the practice’s structural capabilities.

We requested from each of the 6 participating health plans all medical and prescription drug claims and enrollment data spanning June 1, 2006, to May 30, 2011 (2 years prior to and 3 years after the pilot inception date of June 1, 2008), for their members who, at any time during this 5-year period, had 1 or more medical claims (for any service) with a pilot or comparison practice.

Patient Attribution

In each of 4 time periods (the preintervention period and intervention years 1, 2, and 3), we attributed patients to the primary care clinicians (specialty designations family practice, general practice, internal medicine, pediatrics, adolescent medicine, geriatric medicine, and nurse practitioner) who provided the plurality of qualifying services (Current Procedural Terminology codes 9920x, 9921x, 9924x, 99381–99387, 99391–99397, 99401–99404, 99411–99412, 99420–99429, 99339–99340, 99341–99345, 99347–99350, G0402, G0438, G0439), with the most recent service breaking ties. In sensitivity analyses, we reattributed patients based on the majority (>50%) of qualifying services.

Measures of Quality, Utilization, and Costs

We calculated claims-based process measures of quality using NCQA Healthcare Effectiveness and Data Information Set (HEDIS) specifications, following published recommendations²² and making adjustments as necessary to account for the limited available look-back period. Using laboratory data available from one commercial health plan, we also calculated 2 measures of diabetes control, also based on HEDIS specifications.

To measure utilization, we calculated rates of hospitalization (all-cause), emergency department (ED) visits (all-cause), and ambulatory visits following published recommendations.²³ We also calculated rates of ambulatory care–sensitive hospitalizations and ED visits, because these may be more likely to represent avoidable (and potentially wasteful) resource utilization, relative to all-cause hospitalizations and ED visits. Because health care prices can be sensitive to local market factors, we calculated standardized prices for all claims using Optum normalized pricing software.²⁴ The eAppendix in the Supplement presents measure specifications.

Statistical Analyses

We compared pilot and comparison practices and their preintervention patient populations using Wilcoxon rank sum and Fisher exact tests. To characterize pilot practices’ adoption of structural capabilities, we compared baseline and postintervention rates of capability possession (eg, rates at which practices had EHRs) using Liddell exact tests to account for repeated measurements, excluding 3 practices that did not respond to both surveys.

To mitigate the potential influence of patient selection (which could in theory be induced by pilot participation), in our main analyses we assigned patients to practices based on preintervention attribution only. Under this intent-to-treat framework, any patients switching their primary care physicians (ie, becoming attributed to another practice) during the intervention would still be attributed to their preintervention practices for analysis. Two potential drawbacks of preintervention attribution are nondetectability of interventions targeting patients new to a practice (eg, enhanced patient intake procedures) and potential confounding by time-varying patterns of health care consumption.²⁵ Therefore, we performed sensitivity analyses using sequential cross-sectional attribution (ie, reattributing patients in each year based on their visits in that year) to assign patients.

To evaluate associations between the PACCI intervention and changes in quality measure performance, we fit generalized logistic models using propensity weights to balance pilot and comparison practices’ shares of patients from each health plan and baseline performance on each measure. By giving more weight to observations from comparison practices whose baseline performance resembled that of pilot practices, this weighting method produced “average treatment effect on the treated” (ATT) estimates, answering the question: “Among practices closely resembling the pilot practices, what changes are associated with the intervention?”²⁶ The dependent variable in each model was patient-level receipt of the indicated service, and independent variables were indicators for time period (preintervention and each intervention year), interactions between time period and pilot/comparison status, indicators for the health plan contributing each observation and patient enrollment in a health maintenance organization (HMO), and fixed effects (dummy variables) for each practice.

For measures of utilization, we fit negative binomial models, using ATT propensity weighting to balance health plans and baseline utilization rates. The dependent variable in each model was the utilization count in the time period of interest. Independent variables were indicators for time period (preintervention and each intervention year); interaction between time period and pilot/comparison status; indicators for the health plan contributing each observation and patient enrollment in an HMO; patient age, sex, and preintervention Charlson comorbidity score²⁷; and practice fixed effects. In sensitivity analyses, we refit the utilization models using 2-part models (logistic and negative binomial).

For measures of cost, we fit propensity-weighted models with the same independent variables but used a linear functional form, following the methods of recent similar evaluations.²⁸ In sensitivity analyses, we refit the cost models using log-transformed costs.²⁹

We also identified patients who had multiple all-cause hospitalizations or ED visits (2 visits or ≥3 visits) within a single year. We then fit logistic models to identify pilot-associated changes in the prevalence of these “multiple use” patients, using the same independent variables as the other utilization models.

In all models, we used generalized estimating equations with robust standard errors to account for heteroscedasticity, autocorrelation, and clustering of patients within practices.^30,31 Because these methods can be sensitive to missing data, which are created when patients change health plans, we included only health plan members who were continuously enrolled during the study period in our main models. However, because of small numbers of patients contributing observations for hemoglobin A_1c and low-density lipoprotein cholesterol control and pediatric asthma controller medication use, models for these measures included patients lacking continuous enrollment.

To display adjusted performance data, we used recycled prediction methods, which allow estimates of associations on the original scale of the data, accounting for differences in covariate patterns between pilot and comparison practices.^32,33

We conducted sensitivity analyses in addition to those described herein. We repeated our analyses using a plan-by-plan basis, excluding nurse-managed health centers (because no comparison practices were in this category), and including health plan members who were not continuously enrolled during the study period. Because reductions in utilization and costs of care might be more achievable and detectable among patients with chronic illness and diabetes was targeted by the PACCI, we repeated our utilization and cost models among only patients with diabetes.³⁴

We considered 2-tailed P values <.05 significant. We performed data management and analyses using SAS version 9.2 (SAS Institute), SQL Server 2008 (Microsoft), and R version 3.0.0 (R Foundation).

Results

All 34 volunteering practices were admitted to the pilot, but 2 withdrew prior to its initiation (ie, before receiving any assistance or supplemental funding) for reasons unrelated to the pilot (office manager illness in 1 practice and key staff member departure in the other). All 32 remaining practices completed the pilot; none dropped out. The 6 participating health plans together accounted for the majority of total revenues (median, 65%) among pilot practices.

The 32 pilot and 29 comparison practices were similar in baseline size, specialty, and patient case mix (Table 1). However, 6 pilot practices were nurse-managed health centers, whereas no comparison practices were. In total, 64 243 patients were attributed to pilot practices and 55 959 to comparison practices in the preintervention period.

Table 1.

Baseline Characteristics of Pilot and Comparison Practices

	Pilot (32 Practices, 64 243 Attributed Patients)	Comparison (29 Practices, 55 959 Attributed Patients)	P Valuea
No. of PCPs, median (IQR)	5 (4–6)	4 (3–8)	.99
Main specialty, No.
Family practice	16	16	.91
Internal medicine	9	7
Pediatrics	7	6
Location, No.
Urban	30	27	.99
Suburban	2	2
Nurse-managed health center, No.	6b		0	.03
Patient panel characteristics, median (IQR)c
No. of attributed patients per practice	1245 (481–2245)	1291 (875–2471)	.70
Female sex, %	58 (50–68)	52 (50–59)	.15
Age among adults, y	44 (30–57)	44 (28–56)	.54
Diabetes, %	7 (5–9)	6 (3–9)	.51
Asthma, %	2 (1–4)	2 (1–4)	.89
Practice Charlson score, mean (IQR)	1.00 (0.71–1.42)	1.02 (0.49–1.48)	.77
Insurance type, median (IQR), %
Commercially insured, %	65 (27–98)	39 (19–98)	.94
Health plan A, commercial	18 (6–51)	13 (3–32)	.14
Health plan B, commercial	28 (11–45)	25 (8–71)	.64
Health plan C, Medicaid	23 (2–61)	33 (2–73)	.82
Health plan D, Medicaidd	0 (0–74)	0 (0–90)	.75

Abbreviations: IQR, interquartile range; PCP, primary care physician or clinician (including MDs, DOs, and nurse practitioners).

^aWilcoxon rank sum test for counts, percentages, and other continuous variables; Pearson χ² and Fisher exact tests for categorical variables. Note: all variables are calculated at the practice level.

^bFive of the nurse-managed health centers had patient panels resembling family practices (22%−52% pediatric); the sixth had no pediatric patients.

^cCalculated at the practice level among patients attributed to study practices in the preintervention period based on plurality of qualifying visits.

^dMininum-maximum reported rather than IQR because the IQR is (0–0).

Twenty-nine pilot practices (91%) completed both the baseline and year 3 structural surveys. A low survey response rate (24%) among comparison practices precluded meaningful analysis of comparison practices’ structural capabilities.

All of the pilot practices achieved NCQA PPC-PCMH recognition by the third intervention year, with half achieving level 3 status (Table 2). Pilot practices accumulated average bonuses of $92 000 per primary care physician and reported structural transformation on a wide range of capabilities. For example, use of registries to identify patients overdue for chronic disease services increased from 30% to 85% of pilot practices (P < .001), and electronic medication prescribing increased from 38% to 86% (P = .001).

Table 2.

NCQA Recognition, Financial Bonuses, and Selected Structural Changes Among Pilot Practices

	No. of Practices Contributing Data	Baseline, No. (%) of Practicesa	Pilot Year 3, No. (%)	P Valueb
NCQA recognition level
1	32	NAc	13 (41)	NA
2	32	NA	3 (9)	NA
3	32	NA	16 (50)	NA
Financial bonus per PCP, mean (range)	32	NAd	$92 000 ($19 000-$164 000)	NA
Performance feedback
Quality feedback to PCPs	29	9 (31)	23 (79)	<.001
Utilization or cost feedback to PCPs	29	11 (38)	15 (52)	.22
Monthly or more frequent meetings about quality	29	8 (28)	21 (72)	<.001
Monthly or more frequent meetings about utilization	28	8 (29)	11 (39)	.55
Use of registry for patients
Overdue for screening services	26	9 (35)	16 (62)	.07
Overdue for chronic disease services	27	8 (30)	23 (85)	<.001
Out of target range for chronic disease laboratory values	26	5 (19)	18 (69)	.001
At high risk of disease complications or hospitalization	29	3 (10)	27 (93)	<.001
Care management for patients
At high risk of disease complications or hospitalization	29	3 (10)	27 (93)	<.001
Help in better managing their diabetese	24	14 (58)	23 (96)	.004
Help in better managing their asthmae	28	14 (50)	21 (75)	.09
Routine assessment of self-management needs in chronic illness	28	3 (11)	16 (57)	.001
Referral system for linking to community programs	29	16 (55)	22 (76)	.11
Outreach systems to contact patients due for services
Breast cancer screen	23	6 (26)	16 (70)	.002
Cervical cancer screen	23	7 (30)	13 (57)	.07
Colorectal cancer screen	23	5 (22)	14 (61)	.01
Diabetes: hemoglobin A1c test	24	6 (25)	19 (79)	<.001
Diabetes: cholesterol test	24	6 (25)	19 (79)	<.001
Diabetes: eye examination	24	5 (21)	18 (75)	<.001
Diabetes: nephropathy monitoring	24	5 (21)	17 (74)	<.001
After hospitalization	29	6 (21)	27 (93)	<.001
No appointment for extended periodf	29	13 (45)	21 (72)	.06
EHR capabilities
Patient medication lists	29	17 (59)	27 (93)	.002
Patient problem lists	29	17 (59)	26 (90)	.004
Consultation notes from specialists	29	15 (52)	22 (76)	.02
Hospital discharge summaries	29	17 (59)	22 (76)	.13
Electronic medication prescribing	29	11 (38)	25 (86)	.001
Electronic laboratory test ordering	29	10 (34)	23 (79)	<.001
Electronic radiology test ordering	28	7 (25)	11 (39)	.22
Alerts if ordered tests are not performed	27	6 (22)	11 (41)	.18
Secure messaging to and from patients	29	4 (14)	13 (45)	.004
Access
Weekend care offered regularly	29	18 (62)	20 (69)	.63
Evening care offered ≥2 nights per wk	29	20 (69)	21 (72)	.99
Appointments for new patients within 2 wk	28	13 (46)	20 (71)	.04

Abbreviations: EHR, electronic health record; NA, not applicable; NCQA, National Committee for Quality Assurance; PCP, primary care physician or clinician (including MDs, DOs, and nurse practitioners).

^aBecause of item nonresponse, denominators for percentages are not the same for all entries in the table.

^bLiddell exact test.

^cNo practice had submitted an NCQA recognition application at baseline (ie, prior to the start of the pilot).

^dCumulative over the 3-year pilot and calculable only in pilot year 3.

^eProvided by specially trained nonphysician staff.

^fLonger than clinically appropriate.

Four of the 6 health plans (the largest 2 commercial and largest 2 Medicaid plans) supplied claims data. One of these 4 plans was unable to supply claims dated prior to January 1, 2007, limiting the effective study window to January 1, 2007, to May 30, 2011, for analyses of pooled claims data.

Of the 11 quality measures evaluated, pilot participation was significantly associated with greater performance improvement on 1 measure: nephropathy monitoring in diabetes (Table 3). Point estimates suggested improved performance among pilot practices relative to comparison practices for other diabetes measures and for colorectal cancer screening, but these differences were not statistically significant.

Table 3.

Propensity-Weighted, Adjusted Quality-of-Care Differences Between Pilot and Comparison Practices Among Continuously Enrolled Patients

	No. of Eligible Patients	No. (%)a
		Preintervention	Year 1	Year 2	Year 3
HbA1c testing
Pilot	1079	919 (85.2)	920 (85.3)	952 (88.2)	950 (88.0)
Comparison	606	516 (85.2)	534 (88.2)	528 (87.1)	502 (82.8)
% Difference (95% CI)		NAb	−3.0 (−8.5 to 2.5)	1.1 (−5.8 to 8.0)	5.2 (−0.8 to 11.2)
P value		NA	.29	.75	.09
HbA1c abnormal
Pilot	582	112 (19.2)	105 (18.1)	119 (20.4)	104 (17.9)
Comparison	175	34 (19.2)	26 (14.8)	26 (14.7)	18 (10.3)
% Difference (95% CI)		NA	3.3 (−6.6 to 13.2)	5.6 (−4.8 to 16.1)	7.5 (−1.4 to 16.4)
P value		NA	.51	.29	.10
LDL-C testing
Pilot	1079	880 (81.6)	878 (81.4)	905 (83.9)	906 (84.0)
Comparison	606	494 (81.6)	501 (82.6)	491 (81.0)	479 (79.0)
% Difference (95% CI)		NA	−1.3 (−7.3 to 4.8)	2.9 (−4.8 to 10.7)	5.0 (−0.4 to 10.4)
P value		NA	.68	.46	.07
LDL-C abnormal
Pilot	582	199 (34.2)	191 (32.8)	176 (30.3)	201 (34.6)
Comparison	175	60 (34.2)	54 (30.7)	47 (26.9)	62 (35.2)
% Difference (95% CI)		NA	2.1 (−8.7 to 12.8)	3.4 (−11.2 to 18.0)	−0.6 (−22.4 to 21.3)
P value		NA	.72	.63	.97
Nephropathy monitoring
Pilot	1079	786 (72.8)	842 (78.0)	875 (81.1)	892 (82.7)
Comparison	606	441 (72.8)	440 (72.6)	444 (73.3)	435 (71.7)
% Difference (95% CI)		NA	5.5 (0.5 to 10.5)	7.8 (3.0 to 12.3)	11.0 (5.6 to 16.3)
P value		NA	.03	.002	<.001
Eye examinations
Pilot	1079	447 (41.4)	507 (47.0)	519 (48.1)	531 (49.2)
Comparison	606	251 (41.4)	261 (43.0)	273 (45.0)	277 (45.7)
% Difference (95% CI)		NA	4.0 (−0.4 to 8.3)	3.1 (−4.8 to 11.0)	3.5 (−3.7 to 10.7)
P value		NA	.07	.45	.33
Asthma appropriate medication, pediatric
Pilot	603	543 (90.1)	546 (90.5)	535 (88.7)	541 (89.8)
Comparison	820	739 (90.1)	794 (96.8)	813 (99.1)	816 (99.5)
% Difference (95% CI)		NAb	−6.4 (−13.4 to 0.7)	−10.4 (−25.2 to 4.4)	−9.7 (−22.4 to 3.1)
P value		NA	.08	.17	.14
Breast cancer screening
Pilot	4330	2416 (55.8)	2988 (69.0)	3079 (71.1)	3057 (70.6)
Comparison	2804	1565 (55.8)	1890 (67.4)	1971 (70.3)	1943 (69.3)
% Difference (95% CI)		NA	1.6 (−1.4 to 4.6)	0.7 (−3.5 to 5.0)	1.3 (−3.9 to 6.5)
P value		NA	.29	.73	.62
Cervical cancer screening
Pilot	2549	1700 (66.7)	1876 (73.6)	1998 (78.4)	1917 (75.2)
Comparison	1710	1141 (66.7)	1277 (74.7)	1332 (77.9)	1300 (76.0)
% Difference (95% CI)		NA	−1.1 (−3.9 to 1.7)	0.5 (−3.3 to 4.3)	−0.8 (−6.7 to 5.1)
P value		NA	.43	.79	.79
Chlamydia screening
Pilot	521	180 (34.6)	200 (38.3)	226 (43.3)	249 (47.7)
Comparison	521	180 (34.6)	205 (39.3)	247 (47.4)	247 (47.4)
% Difference (95% CI)		NA	−1.0 (−7.0 to 4.9)	−4.2 (−10.2 to 1.9)	0.3 (−5.9 to 6.5)
P value		NA	.74	.18	.92
Colorectal cancer screening
Pilot	4949	1965 (39.7)	2361 (47.7)	2722 (55.0)	2722 (55.0)
Comparison	3493	1387 (39.7)	1579 (45.2)	1816 (52.0)	1764 (50.5)
% Difference (95% CI)		NA	2.6 (−0.4 to 5.6)	3.1 (−1.0 to 7.1)	4.3 (−3.9 to 12.9)
P value		NA	.09	.14	.30

Abbreviations: LDL-C, low-density lipoprotein cholesterol; NA, not applicable.

^aPercentage point estimates are propensity-weighted recycled predictions from linear probability models adjusting for practice baseline score, health plan contributing each observation, and whether each patient was in a health maintenance organization at the time of the observation. Frequencies are calculated from adjusted, weighted percentages based on the denominators for each measure. Differences, CIs, and P values correspond to marginal differences from the linear probability models.

^bBecause of the inclusion of fixed effects for practices, regression models do not estimate preintervention differences between pilot and comparison.

Pilot participation was associated with a greater increase in the rate of ambulatory care–sensitive hospitalization, relative to comparison practices, in intervention year 2 (Table 4). There were no other statistically significant differences in measures of utilization, costs of care, or rates of multiple same-year hospitalizations or ED visits (Table 5).

Table 4.

Propensity-Weighted, Adjusted Differences in Utilization and Costs of Care Between Pilot and Comparison Practices Among Continuously Enrolled Patients (n=39559 Pilot and n=40234 Comparison Patients)

	No.(Rate per 1000 Patients per mo)[95%CI]a
	Preintervention	Year 1	Year 2	Year 3
Hospitalizations,all-cause
Pilot	479(12.1) [9.4to14.8]	435(11.0) [7.3to14.7]	435(11.0) [7.4to14.6]	427(10.8) [7.1to14.5]
Comparison	489(12.1) [9.3to14.9]	394(9.8) [7.3to12.3]	398(9.9) [7.4to12.4]	386(9.6) [7.1to12.1]
Difference(95%CI)b	NAc	1.2 (−0.9 to 3.8)	1.1 (−1.1 to 3.9)	1.2 (−1.2 to 4.4)
P value	NA	.30	.36	.36
Hospitalizations,ambulatorycare–sensitive
Pilot	67(1.7) [1.3to2.1]	71(1.8) [1.4to2.2]	83(2.1) [1.6to2.6]	83(2.1) [1.6to2.6]
Comparison	68(1.7) [1.1to2.3]	64(1.6) [1.1to2.1]	68(1.7) [1.2to2.2]	72(1.8) [1.4to2.2]
Difference(95%CI)b	NA	0.2 (−0.1 to 0.7)	1.12 (0.82–1.51)	0.3 (−0.3 to 1.0)
P value	NA	.20	.007	.33
EDvisits,all-cause
Pilot	1646(41.6) [31.3to51.9]	1839(46.5) [34.0to59.0]	1851(46.8) [34.2to59.4]	1745(44.1) [31.9to56.3]
Comparison	1673(41.6) [29.7to53.5]	1867(46.4) [34.4to58.4]	1854(46.1) [34.5to57.7]	1698(42.2) [31.9to52.5]
Difference(95%CI)b	NA	0.1 (−3.2 to 3.8)	0.7 (−3.2 to 5.0)	1.9 (−3.1 to 7.5)
P value	NA	.94	.73	.47
EDvisits,ambulatorycare–sensitive
Pilot	1100(27.8) [19.9to35.7]	1187(30.0) [20.6to39.4]	1163(29.4) [20.0to38.8]	1092(27.6) [18.4to36.8]
Comparison	1119(27.8) [18.7to37.0]	1155(28.7) [19.9to37.5]	1123(27.9) [29.5to36.3]	1006(25.0) [17.5to32.5]
Difference(95%CI)b	NA	1.3 (−1.4 to 4.3)	1.5 (−2.1 to 5.5)	2.6 (−1.7 to 7.7)
P value	NA	.35	.43	.25
Ambulatoryvisits,primarycare
Pilot	6496(164.2) [142.0to186.4]	6045(152.8) [123.0to182.6]	5993(151.5) [120.0to183.0]	5811(146.9) [117.3to176.5]
Comparison	6606(164.2) [144.2to184.2]	6047(150.3) [132.9to167.7]	5975(148.5) [132.2to164.8]	5717(142.1) [127.0to157.2]
Difference(95%CI)b	NAc	2.5 (−0.9 to 5.4)	2.9 (−2.4 to 7.4)	4.8 (−1.7 to 10.5)
P value	NA	.14	.27	.14
Ambulatoryvisits,specialist
Pilot	2010(50.8) [24.8to76.8]	2496(63.1) [34.8to91.4]	2623(66.3) [36.4to96.2]	2556(64.6) [36.0to93.2]
Comparison	2044(50.8) [18.6to83.0]	2639(65.6) [29.1to102.1]	2772(68.9) [29.9to107.9]	2740(68.1) [30.7to105.5]
Difference(95%CI)b	NA	−2.4 (−3.8 to 1.4)	−2.6 (−4.5 to 1.8)	−3.5 (−6.4 to 3.0)
P value	NA	.57	.21	.40
Totalcostsofcare,$d
Pilot	389.0 (304.1 to 474.6)	417.8 (307.5 to 529.7)	440.2 (313.3 to 566.9)	430.3 (302.7 to 556.3)
Comparison	389.0 (317.5 to 458.9)	382.4 (300.0 to 460.1)	407.2 (314.1 to 494.0)	395.2 (303.0 to 480.2)
Difference(95%CI)	NA	35.4 (−7.7 to 78.5)	33.0 (−10.4 to 76.4)	35.1 (−22.3 to 92.5)
P value	NA	.11	.14	.23

Abbreviations: ED, emergency department; NA, not applicable.

^aPoint estimates are propensity-weighted recycled predictions from negative binomial regression models adjusting for baseline utilization rates; patient sex, age, and Charlson comorbidity score; health plan contributing each observation; and whether each patient was in a health maintenance organization at the time of the observation. Frequencies are calculated from adjusted, weighted percentages based on the denominators for each measure. Differences, CIs, and P values represent marginal differences from these models.

^bDifference per 1000 patients per month.

^cBecause of the inclusion of fixed effects for practices, regression models do not estimate preintervention differences between pilot and comparison.

^dPoint estimates and CIs for cost are propensity-weighted recycled predictions from linear models adjusting for baseline costs of care; patient sex, age, and Charlson comorbidity score; health plan contributing each observation; and whether each patient was in a health maintenance organization at the time of the observation. Differences, their CIs, and P values correspond to marginal differences from the linear cost models.

Table 5.

Propensity-Weighted, Adjusted Rates of Multiple Hospital and ED Use in Each Year Among Continuously Enrolled Patients Attributed to Pilot and Comparison Practices^a

	No. (%)b
	12 mo Prior to Intervention	Year 1	Year 2	Year 3
2 Hospitalizations withinyear,all-cause
Pilot	554 (1.4)	561 (1.4)	529 (1.3)	480 (1.2)
Comparison	563 (1.4)	471 (1.2)	504 (1.3)	414 (1.0)
OR (95% CI)	NAc	1.22 (0.89–1.68)	1.07 (0.76–1.50)	1.18 (0.84–1.67)
P value	NA	.22	.70	.34
≥3 Hospitalizations within year, all-cause
Pilot	320 (0.81)	358 (0.91)	385 (0.97)	386 (0.98)
Comparison	325 (0.81)	337 (0.84)	354 (0.88)	353 (0.88)
OR (95% CI)	NA	1.09 (0.83–1.42)	1.12 (0.82–1.51)	1.12 (0.79–1.60)
P value	NA	.54	.48	.52
2 ED visits within year, all-cause
Pilot	1977 (5.0)	2255 (5.7)	2305 (5.8)	2132 (5.4)
Comparison	2012 (5.0)	2344 (5.8)	2410 (6.0)	2161 (5.4)
OR (95% CI)	NA	1.00 (0.90–1.10)	0.99 (0.89–1.11)	1.01 (0.88–1.17)
P value	NA	.96	.89	.86
≥3 ED visits within year, all-cause
Pilot	2018 (5.1)	2267 (5.7)	2271 (5.7)	2099 (5.3)
Comparison	2052 (5.1)	2616 (6.5)	2562 (6.4)	2240 (5.6)
OR (95% CI)	NA	0.85(0.71–1.02)	0.88 (0.73–1.06)	0.93 (0.74–1.17)
P value	NA	.09	.17	.54

Abbreviations: ED, emergency department; NA, not applicable; OR, odds ratio.

^aThe number of patients contributing data from pilot practices was 39 559 and from comparison practices, 40 234, in the preintervention period; these patients had continuous enrollment in their health plan during the study period.

^bPoint estimates are propensity-weighted recycled predictions from logistic regression models adjusting for baseline utilization rates; patient sex, age, and Charlson comorbidity score; health plan contributing each observation; and whether each patient was in a health maintenance organization at the time of the observation. Frequencies are calculated from adjusted, weighted percentages based on the denominators for each measure. Odds ratios, CIs, and P values reflect marginal differences between pilot and comparison in each intervention year.

^cBecause of the inclusion of fixed effects for practices, regression models do not estimate preintervention differences between pilot and comparison.

The results of sensitivity analyses were consistent with the primary results with 3 exceptions. First, in models including patients who were not continuously enrolled, pilot participation was statistically significantly associated with better performance for colorectal cancer screening. Second, in models with cross-sectional attribution, pilot participation was statistically significantly associated with worse performance on pediatric asthma appropriate medication use. Third, pilot participation was not statistically significantly associated, in any year, with ambulatory care–sensitive hospitalization rates in models with cross-sectional attribution and in models including patients lacking continuous enrollment.

Discussion

Despite widespread enthusiasm for the medical home concept, few peer-reviewed publications have found that transforming primary care practices into medical homes (as defined by common recognition tools and in typical practice settings) produces measurable improvements in the quality and efficiency of care.^6–8 The southeast region of the PACCI, which featured relatively generous financial support from 6 commercial and Medicaid health plans, is to our knowledge the first multipayer pilot in the nation to report results over a 3-year period of transformation. We found that practices participating in the PACCI pilot adopted new structural capabilities and received NCQA certification as medical homes. Our evaluation also suggests that the quality of diabetes care improved, but we found few statistically significant results and no robust associations with utilization or costs.

Our findings differ with prior evaluations of demonstrations in large, integrated delivery systems^14,15,35 and with cross-sectional studies that lacked an intervention.^36,37 However, the southeast PACCI contained ingredients common to many current pilots, including an emphasis on NCQA recognition.⁴ Our findings are congruent with those of medical home interventions occurring among small primary care practices, even though the PACCI intervention included more participating practices, occurred over a longer time frame, and featured greater financial support.^{11–13,38,39}

Why was the PACCI southeast regional pilot not associated with broad quality improvements, lower utilization, and cost savings? This pilot—the first of the PACCI regions— was focused on quality improvement for chronic conditions and featured early financial rewards for NCQA recognition, possibly distracting from other activities intended to improve the quality and efficiency of care. In subsequent regions, PACCI organizers placed less emphasis on early NCQA recognition so that practices could focus more fully on learning collaborative participation (oral communication with Michael Bailit, MM, Bailit Health Purchasing, November 26, 2010).

In addition, southeast PACCI pilot practices had neither direct incentives to contain costs nor feedback on their patients’ utilization of care. Possibly as a consequence of these features of pilot design, we found that few pilot practices increased their night and weekend access capabilities, which could, in theory, have produced short-term savings by offering patients an alternative to more expensive sites of care (such as hospital emergency departments).

Several reasons may explain the limited (or absent) changes in quality, utilization, and costs of care that we observed, despite structural transformation among pilot practices. First, pilot practices were volunteers and may have been more quality-conscious than other practices, performing closer to their optimal levels at baseline. If so, “ceiling effects” resulting from near-optimal performance would imply that an average practice developing a medical home structure might show greater improvement than what we observed. Therefore, our findings may not generalize to other medical home interventions. Second, the structural survey response rate among comparison practices was low, so we cannot tell whether comparison practices were also transforming during the pilot (eg, due to federal incentives to adopt EHRs). Simultaneous transformation would weaken the ability of pilot practices to differentiate themselves from comparison practices. Third, prior cross-sectional research suggests that relationships between structural capabilities, NCQA recognition, and performance on measures of quality and utilization are modest.^40–42 The elements of practice transformation necessary to produce desired changes in patient care may be different from the capabilities assessed commonly by research surveys and certification tools.

Our study has limitations. First, pilot and comparison practices were not matched perfectly. Although propensity weighting and multiple regression can account for observed differences, unobserved differences could have introduced confounding. Second, failure to find statistically significant results for most diabetes measures may have resulted from insufficient power to detect performance improvement differences smaller than approximately 8 percentage points. Third, the baseline structural survey was retrospective, with potential respondent recall error. Fourth, we were unable to examine changes in patient experience.

Conclusions

One of the first, largest, and longest-running multipayer medical home pilots in the United States, in which participating practices adopted new structural capabilities and received NCQA certification, was associated with limited improvements in quality and was not associated with reductions in utilization of hospital, emergency department, or ambulatory care services or total costs of care over 3 years. These findings suggest that medical home interventions may need further refinement.