The Medical Home Transformation in the Veterans Health Administration: An Evaluation of Early Changes in Primary Care Delivery

Abstract

Objective

To evaluate the effect of medical home implementation on primary care delivery in the Veterans Health Administration (VHA).

Data Sources/Study Setting/Study Design

We link interview-based qualitative data on medical home implementation to quantitative outcomes from VHA clinical encounter data. We use a longitudinal analysis with provider fixed effects (taking advantage of variation in timing of implementation and allowing each provider to serve as a control for him or herself) to test whether patient-aligned care team (PACT) implementation was associated with changes in organizational processes and patient outcomes.

Principal Findings

Among 683 PCPs, caring for 321,295 patients, the uptake of eight of nine PACT structural changes significantly increased from July 2010 to June 2012 as did the percentage of primary care appointments occurring by telephone and hospital discharges contacted within 2 days of discharge. We found that PACT implementation was associated with significant improvements in 2-day post-hospital discharge contact, but not primary care visits occurring by telephone or within 3 days of the requested date. We found no association between medical home implementation and rates of emergency department use by patients.

Conclusions

Medical home implementation at the VHA resulted in large changes in the structure of care but few changes in patient-level outcomes. These results highlight both the complexity of studying the effect of the medical home as well as implementing this model to change primary care delivery.

The patient-centered medical home has increasingly been looked to by policy makers, health care providers, and private insurers as a potential solution to the fragmented and inefficient U.S. health care system (National Committee for Quality Assurance 2011). The medical home model adapts the current model of primary care delivery, often characterized as episodic, reactive, and splintered, by focusing on team-based care that more fully addresses patient needs, including continuity of care, coordination of care, enhanced access, and alignment of incentives with quality and patient safety. The goal of the medical home model is ultimately to achieve better patient outcomes while reducing costs. Over 1,000 practices in the United States have been recognized as medical home practices using the National Committee for Quality Assurance's medical home standards for recognition (The Commonwealth Fund 2010).

Despite its widespread adoption, the evidence that transformation to the medical home model improves care is mixed. A recent systematic review included 14 quantitative medical home evaluations; each included at least three of the five medical home components defined by the Agency for Healthcare Research and Quality—care that is patient-centered, comprehensive, coordinated, and accessible, and follows a systems-based approach to quality and safety (Peikes, Zutshi et al. 2012). The results of these studies were varied and few were scientifically rigorous. Some demonstrated reduced costs (Hughes, Weaver et al. 2000; Unutzer, Katon et al. 2008; Counsell, Calahan et al. 2009; Leff, Reider et al. 2009), while others found increased costs (Hughes, Weaver et al. 2000; Counsell, Calahan et al. 2009) or no statistically significant changes in cost (Unutzer, Katon et al. 2008; Leff, Reider et al. 2009). Similarly, changes in health care utilization were mixed (Hughes, Weaver et al. 2000; Counsell, Callahan et al. 2007; Dorr, Wilcox et al. 2008; Counsell, Calahan et al. 2009; Gilfallan, Tomcavage et al. 2010; Boult, Reider et al. 2011). Similarly, other recent reviews find mixed results (Jackson, Powers et al. 2013), inconsistent methodological approaches (Hoff, Weller et al. 2012), and wide variation in how the medical home is implemented and represented in the studies (Alexander and Bae 2012; Hoff, Weller et al. 2012).

Indeed, most prior studies have tested for the effect of the medical home model by simply comparing patient outcomes before and after medical home implementation, relying on checking whether certain indicators of the medical home model are met (Alexander and Bae 2012). Little attention has been paid to how these elements interact, how well the medical home was implemented, or whether practices were successful in the transformation process, which may lead to mismeasurement of the medical home effect. Negative studies may be partly explained if practices were unsuccessful in fully implementing the medical home model—if the medical home model was not implemented, finding an effect attributable to the medical home would be impossible. On the other hand, positive studies may result from the Hawthorne effect—practices receiving the extra attention that comes with medical home transformation may have changed their health care delivery or practice patterns in a way that alters patient outcomes or utilization patterns even in the absence of successfully implementing the medical home model. In either case, treating the implementation of the medical home as an intervention that can be easily switched on and off could lead to results that do not truly measure the effect of the medical home.

In 2010, the Veterans Health Administration (VHA) began implementing a medical home model in all primary care clinics nationwide. The VHA provides primary care for over 5 million veterans across 160 hospital-based and 783 community-based primary care clinics in the United States. With these changes, the VHA thus became the largest health system to implement the medical home to date in the United States. The VHA's medical home model is designed to improve the organization of primary care to make it more accessible, patient-centered, better at managing high-risk patients, and more efficient (Patient Centered Primary Care Implementation Work Group 2011).

Our objective is to evaluate the effect of implementation of the medical home model in the VHA on care delivered within the VHA. Specifically, we tie qualitative data on how successfully elements of the medical home were implemented at the clinic level to quantitative data on whether medical home implementation resulted in changes in organizational processes of care or patient outcomes. We focus on organizational processes and patient outcomes because they were most directly targeted by medical home implementation and thus we are most likely to see improvements on these measures, if they exist.

Setting

Implementation of the VHA's medical home model (also known within the VHA as the patient-aligned care team model or PACT) began in April 2010. The PACT model focuses on transforming primary care in three areas: practice redesign, access to care, and care management and coordination. Practice redesign focuses on encouraging primary care providers to work in PACT teams to improve communication and teamwork, where a team consists of a primary care provider, a registered nurse, a licensed clinical associate (e.g., licensed practical nurse or medical support assistant), and an administrative clerk. Access to primary care focuses on improving same-day appointments and shifting work from single face-to-face visits to group or telephone-based visits, thus freeing up provider time for urgent visits and for non-visit-based work. Care management and coordination focus on identifying, managing, and coordinating care of high-risk patients as well as improving transitions of care between the primary care setting and other settings such as inpatient care or specialty care.

The VHA's PACT model is similar to the NCQA's PCMH standards, a widely used standard for PCMH. Although the NCQA standards focus on six standards (enhanced access and continuity, identifying and managing patient populations, planning and managing care, providing self-care support and community resources, tracking and coordinating care, and measuring and improving performance) (National Committee for Quality Assurance 2011b), compared with the three the VHA defines, these models encompass similar principles with two exceptions. The NCQA standards include two areas the VHA model does not: (1) providing self-care support and community resources and (2) measuring and improving performance, although the VHA has been actively engaged in this area since the 1990s.

The VHA has dedicated over 1 billion dollars nationally to the medical home transformation, with money distributed to local medical centers that made decisions about how to invest this money locally. Implementation has been an ongoing process, with gradual rollout of structural changes to support the medical home model. While the launch of the medical home model occurred nationally in April 2010, relatively few providers began delivery transformation at that time. In the subsequent 2 years, increasing numbers of providers and primary care staff were required to undergo training on implementing the medical home model and to begin making structural changes to the way they delivered primary care to support the medical home model. The VHA continues to devote financial resources and efforts to implementing this transformation, as well as training of all primary care staff and collaborators (e.g., pharmacists, social workers, behavioral health providers) to support spread and sustainment of the model. Implementation of this model is expected to continue through 2014.

Methods

Overview of Empirical Approach

To examine whether changes in health care delivery in the VHA under the PACT transformation led to changes in organizational processes of care and patient outcomes, we link detailed interview-based qualitative data on PACT implementation to quantitative outcomes from VHA clinical encounter data. We identify the effect of PACT implementation from two sources of variation—the timing and the effectiveness of PACT implementation across study sites. We thus conduct a longitudinal analysis with provider fixed effects (allowing each provider to serve as a control for him or herself) to test whether PACT implementation was associated with changes in organizational processes and patient outcomes.

We examine PACT implementation in one VHA network (also called Veterans Integrated Service Network, or VISN), VISN 4, which covers 104 counties, including the majority of Pennsylvania, Delaware, and West Virginia, and parts of Ohio, New York, and New Jersey. VISN 4 includes 56 primary care sites located in 10 hospital-based and 45 community-based primary care clinics, and provides primary care for over 300,000 veterans. PACT implementation in VISN 4 was nonvoluntary, with the rollout schedule and performance metrics being set by VISN and medical center leaders.

Qualitative Methods and Data

Interviews to Track PACT Implementation

We conducted site visits to each of the 10 VA medical centers in VISN 4 between July and November 2010. At each site visit we interviewed administrators, primary care clinical leads, and members of teams involved in early local PACT implementation in either one-on-one or group settings. A primary goal of these site visits was to establish a baseline understanding of the structural changes planned to support PACT implementation. A total of 120 VHA staff members were interviewed during site visits, ranging from 8 to 16 during each site visit. Interviews were audiorecorded and transcribed.

We then developed a structured interview guide to use in follow-up interviews with the goal of identifying key structural elements of PACT implementation, developed based on review of site visit interview transcripts, field notes taken during observations at PACT training events, VHA PACT-related e-mail communications, examination of PACT tracking measures used by VHA, and review of existing medical home assessment tools outside VHA, and feedback from clinical operations partners in the VISN. After successive revisions, we identified one measure of general support from leadership and 10 measures of structural changes to support PACT implementation: (1) accessing and using data for quality improvement; (2) care management of high-risk patients; (3) nurse medication protocols; (4) transitions from the ED; (5) transitions from the hospital; (6) alternatives to single provider face-to-face visits; (7) changes to enhance access; (8) multidisciplinary teams; (9) team communication and functioning; and (10) using patient-centered methods. The final interview guide consisted of structured questions around each of these key elements.

We developed an accompanying “reference guide,” which outlined the 11 elements and gave examples of how each might be implemented. The reference guide also provided guidance on rating each clinical site on how effective reported structural changes to support PACT implementation were using a 5-point Likert scale. Inter-rater reliability of these ratings was assessed in the following manner: two research assistants trained in use of the interview and reference guides conducted one audiorecorded interview each and completed ratings for both sites. One of the authors (G. T.) then independently listened to the audiorecorded interviews and completed ratings for each site; ratings were compared for inter-rater reliability. This process was repeated on new interviews through two rounds until inter-rater reliability reached greater than 95 percent agreement. Subsequent interviews and ratings were conducted by four research assistants trained and supervised by a senior qualitative researcher, who met regularly with interviewer/coders to discuss any questions that arose from a particular interview or a response to a specific question to ensure validity of the interviews and protect against coder “drift.” See the Appendix for the interview guide and the reference guide for coders.

Interviews were then conducted (by phone or in person, whichever was more convenient to the respondent) with a key contact from each PACT site who had knowledge about PACT implementation (i.e., had day-to-day responsibilities related to PACT implementation at his or her site). In cases where the initial contact was unable to respond to all of the questions, we asked for a second contact. After the initial site visit, four sets of interviews were conducted at 6-month intervals over the 2-year period of this study (July 2010 to June 2012).

Independent Variables Derived from Interviews

We summarized the interview data to create two types of site-level variables. First, we created nine binary variables for nine of the ten structural changes we asked about, indicating whether the site used any of the specific structural changes or not (we did not include responses to queries about accessing and using data for quality improvement as respondents were often confused by this question; see Table 3 for a list of these nine structural measures). For example, in asking about changes to support enhanced access, we created a variable equal to one if a clinical site answered yes to any of the following questions in each time period: Are any strategies in place for enhanced access? Are schedule scrubbing methods in place? Are you extending visit intervals when appropriate? Are you using any other methods to enhance access? Second, we created one scale variable summing the 5-point Likert quality/effectiveness variables across 10 questions—the nine structural measures (after dropping the question on data access) and the measure of support from leadership. The Likert scale ranged from 0 (if a particular structural change had not been made) to 4 (if fully implemented) for each of the 10 measures, resulting in a summary score with a range of 0–40. To ensure that these 10 items function as a summative scale, we calculated a Cronbach's alpha for the 10 items in each of the four time periods of the study, which were 0.77, 0.83, 0.92, and 0.86, supporting use of these items as a summative scale.

We also obtained the dates that each PCP in the VISN became a PACT provider. Providers are considered to be PACT providers once they have started the PACT training process. If a provider was not yet a PACT provider, the interview-derived variables were coded as zero. Once a PCP began the process of transforming to a PACT provider, that provider was assigned the site-level variables defined above.

Quantitative Data and Methods

Data

We used data from the VHA's Corporate Data Warehouse, a data repository comprising data from several clinical and administrative systems that contain data on all patient encounters within the VHA. We used this data to identify all primary care providers in the VISN (from the Primary Care Management Module or PCMM), all primary care visits within the VISN, and to measure the outcome variables of interest (defined below).

Study Sample and Time Period

Using PCMM data, we identified all patients with an assigned VISN 4 PCP at any time during the study period. In addition, we included patients who received primary care during our study period who did not have a PCMM-assigned PCP during the period in which they received the care. Using a standard attribution rule, we assigned these patients to the PCP seen most frequently during the period. In case of ties (where a patient saw two PCPs the same number of times) we assigned the patient to the tied PCP seen first during the study period.

Outcome Variables

We included four outcome variables: three organizational processes (telephone-based primary care visits; appointment within 3 days of desired appointment date; and contact within 2 days of discharge from hospital) and one patient outcome (ED visits). All four variables are measured as binary variables indicating, for each eligible observation, whether the outcome of interest occurred in each 6-month period of the study. Thus, telephone-based visits, measured for all primary care veteran visits in a 6-month period, equals one if the visit was by phone and zero otherwise. Appointment within 3 days of desired date, measured for all primary care appointments requested in each 6 months, equals one if the appointment was within 3 days of the requested appointment date; zero otherwise. For this measure, we only included primary care appointment requests where the request date was within 3 days of the desired date to more accurately measure provider availability for urgent appointments. Contact within 2 days of hospital discharge, measured for all inpatient discharges from a VHA hospital, equals one if the veteran was contacted within 2 days of hospital discharge; zero otherwise. ED visits was measured for all veterans enrolled in primary care and equals one if the veteran visited a VHA ED in that 6-month period; zero otherwise.

Empirical Models

We conduct patient-level analyses, with patients clustered within PCPs and sites of care where the site of care is the level of the experiment. We use longitudinal fixed-effect models to test whether changes in health care delivery in the VA under the PACT transformation lead to changes in organizational processes of care or patient outcomes. We use the following general form to test our hypotheses with a linear probability model:

In this regression the outcome variable is one of our four defined outcomes, indexed to patient (i), PCP (j) and 6-month time period (t). These outcomes are estimated as a function of PACT implementation, a vector of patient-level covariates (age, gender, and DCG risk score), PCP fixed effects (controlling for time invariant differences across providers and allow us to identify the effect of providers changing PACT status and allowing each PCP to serve as a control for him or herself), 6-month time period fixed effects (controlling for secular changes in the outcomes that are common to PACT and non-PACT providers), and a mean zero random error component. The coefficient of interest is α, which represents the effect of a PCP changing PACT implementation status on the outcome of interest.

We measure PACT implementation in three ways: (1) a dummy variable indicating whether each PCP was a PACT provider in that study period; (2) a scale variable measuring the quality or effectiveness of PACT implementation in each study period for those providers who are PACT (non-PACT providers are assigned a value of zero); and (3) a vector of nine dummy variables indicating whether PACT providers had implemented each structural change in each study period. We thus estimate the above equation 12 times—using the four outcome variables in combination with each of the three PACT implementation variables.

In all regressions we calculate robust standard errors to account for nonindependence of observations within the 56 sites (Huber 1967; White 1980).

Results

The study cohort is described in Table 1. We include a total of 321,295 primary care patients in our study, assigned to 683 PCPs. Of these patients, 92 percent had one or more visits with a PCP, resulting in 1,395,629 primary care encounters over the study period. There were additionally 50,970 hospital discharges and 135,568 ED visits in this cohort of primary care patients.

Table 1.

Description of Patient Cohort

Total patients (N)	321,295
Age, median (IQR)	65 (56–78)
Male, %	95.2
DCG risk score, median (IQR)	0.25 (0.09–0.67)
Patients with PACT PCP (at start of study), %	21.5
Patients with PACT PCP (at end of study), %	62.2
Primary care encounters, n	1,395,629
Acute hospital discharges, n	50,970
ED visits, n	135,568

Over the study period the percentage of PCPs who were part of the PACT model more than tripled, increasing from 15 percent in the first time period to 49 percent in the last time period (see Table 2). The percentage of PCPs that implemented specific elements of the PACT model also increased, more than tripling in all cases except the use of nurse medication protocols, which initially increased but started falling in the last study period (see Table 3). The quality of PACT implementation also increased substantially over the study, from 1.7 in the first study period to close to 9 in the last study period among all PCPs regardless of whether they had implemented PACT and increased from 11.7 to 18.4 among PCPs that implemented PACT over the study period.

Table 2.

Number (and %) of PCPs in Study Cohort and Engaged in Patient-Aligned Care Team (PACT) Transformation in Each Study Period

	July to December 2010	January to June 2011	July to December 2011	January to June 2012
Total number of PCPs	548	531	523	505
PACT PCPs	81 (14.78)	145 (27.31)	186 (35.56)	246 (48.71)

Table 3.

Number (and %) of PCPs Making Structural Changes in Support of Patient-Aligned Care Team (PACT) Implementation in Each Study Period; and the Mean (and SD) of the Scale Measure of the Effectiveness or Quality of PACT Implementation

	July to December 2010 (n = 548)	January to June 2011 (n = 531)	July to December 2011 (n = 523)	January to June 2012 (n = 505)
High-risk registries	66 (12.0)	142 (26.7)	176 (33.7)	196 (38.8)
Nurse medication protocols	15 (2.7)	48 (9.0)	97 (18.6)	75 (14.9)
Transitions from ED	53 (9.7)	143 (26.9)	177 (33.8)	240 (47.5)
Transitions from hospital	58 (10.6)	139 (26.2)	178 (34.0)	246 (48.7)
Alternatives to face-to-face and one-on-one visits	59 (10.8)	136 (25.6)	181 (34.6)	246 (48.7)
Enhanced access	78 (14.2)	126 (23.7)	179 (34.2)	246 (48.7)
Multidisciplinary teams	78 (14.2)	126 (23.7)	177 (33.8)	236 (46.7)
Team communication and functioning	51 (9.3)	141 (26.6)	181 (34.6)	238 (47.1)
Patient centeredness	12 (2.2)	135 (25.4)	151 (28.9)	237 (46.9)
PACT quality level among all PCPs*, mean (SD)	1.7 (4.7)	6.1 (10.4)	6.5 (9.6)	8.9 (9.8)
PACT quality level among PCPs that had implemented PACT, mean (SD)†	11.7 (5.5)	22.3 (5.6)	18.9 (5.9)	18.4 (4.8)

^*The PACT quality level is the sum of 5-point Likert scales across 10 individual items. The possible range of values is thus zero to 40.

^†Values included for descriptive purposes only (not used as independent variables in regression-based analyses).

The organizational processes of interest also increased over our study, increasing more among PACT providers than among non-PACT providers for both percentage of visits occurring by telephone and hospital discharges contacted within 2 days of discharge (see Table 4). Paradoxically, however, the percentage of appointments that were made within 3 days of the desired date increased more for non-PACT providers than for PACT providers, although both achieved similar rates of close to 76 percent by the last study period. Rates of ED visits declined slightly among PACT providers (from 9.7 to 8.0 percent) while it increased for non-PACT providers (from 7.5 to 8.8 percent).

Table 4.

Patient-Aligned Care Team (PACT) Organizational Process Measures and ED Visits in Each Study Period among PACT and Non-PACT Providers

	July to December 2010	January to June 2011	July to December 2011	January to June 2012
Primary care visits occurring by telephone, n (%)
PACT providers	0 (0.0)	0 (0.0)	3,543 (1.9)	12,626 (5.7)
Non-PACT providers	0 (0.0)	0 (0.0)	748 (0.4)	2,554 (2.3)
Primary care appointments made within 3 days of desired date, n (%)
PACT providers	14,938 (70.1)	27,756 (68.4)	41,170 (75.0)	44,959 (75.9)
Non-PACT providers	47,621 (64.2)	38,278 (65.4)	28,086 (69.5)	21,598 (75.6)
Acute hospital discharges contacted within 2 days of discharge, n (%)
PACT providers	248 (11.0)	977 (22.1)	2,395 (39.0)	4,624 (56.8)
Non-PACT providers	654 (6.4)	1,030 (12.6)	733 (11.1)	1,834 (36.4)
Primary care patients visiting the ED, n (%)
PACT providers	4,453 (9.7)	7,989 (8.4)	11,928 (9.2)	14,855 (8.0)
Non-PACT providers	17,038 (7.5)	13,977 (7.6)	11,098 (7.4)	8,355 (8.8)

Despite the large increases in both structural changes in support of PACT (in Table 3) and an increase in two of the organizational process measures among PACT providers (in Table 4), when examining regression results we find little evidence of improvements in PACT outcomes related to PACT implementation. Table 5 summarizes these regression results (full regression results are displayed in Tables S1–S3). As shown, becoming a PACT provider is related to statistically significant and clinically meaningful changes in only one of the four outcomes (model 1). Two-day post-discharge contact increased by close to 12 percentage points net secular trends (on a base of 6 percent among non-PACT providers).

Table 5.

Effect of Patient-Aligned Care Team (PACT) Implementation on Organizational Process Measures and ED Visits

	Telephone Visits	Appointment within Three Days of Desired Date	Two-Day Post-Discharge Contact	ED Visits
Model 1: using dummy variable indicating whether each PCP was a PACT provider in that study period
PACT provider dummy	0.007 (0.007)	−0.012 (0.017)	0.119 (0.030)***	−0.003 (0.003)
Model 2: using scale variable measuring the quality or effectiveness of PACT implementation in each study period
PACT quality rating	0.000 (0.000)	−0.002 (0.001)	0.007 (0.001)***	−0.000 (0.000)
Model 3: using nine dummy variables indicating whether PACT providers had implemented each structural change in each study period
Nurse medication protocols	−0.022 (0.014)	0.012 (0.025)	−0.048 (0.051)	0.001 (0.003)
Alternatives to face-to-face and one-on-one visits	−0.005 (0.029)	−0.046 (0.047)	−0.019 (0.126)	−0.004 (0.004)
Enhanced access	−0.011 (0.010)	0.013 (0.026)	−0.199** (0.089)	0.006* (0.003)
Multidisciplinary team	−0.011 (0.010)	−0.045* (0.025)	0.156* (0.081)	−0.002 (0.002)
Team communication and functioning	0.027* (0.015)	0.080** (0.036)	0.128 (0.117)	−0.010* (0.005)
High-risk registries	0.031*** (0.010)	0.027 (0.029)	0.030 (0.088)	−0.007** (0.003)
Post-ED transitions	−0.015 (0.016)	−0.073** (0.031)	−0.138 (0.109)	0.006* (0.003)
Post-hospital transitions	0.012 (0.011)	0.062* (0.033)	0.268** (0.125)	0.000 (0.003)
Patient centeredness	−0.012 (0.011)	−0.042* (0.021)	−0.093 (0.068)	0.007 (0.005)

Note. Robust standard errors in parentheses.

^*p < .10;

^**p < .05;

^***p < .01.

Similarly, when testing the association between the quality of PACT implementation and PACT outcomes (model 2), an increase in the rating scale is associated with a statistically significant increase in 2-day post-discharge contact. A 10-point (or one standard deviation) increase in the quality scale was associated with a 7-percentage-point increase in 2-day post-discharge contact. There is no significant relationship between the quality of implementation and the other three outcomes.

Finally, when examining the effect of specific structural changes on PACT outcomes, we found mixed results (model 3). Having high-risk registries increased the use of telephone visits by 3 percentage points, as did having policies to govern team communication and functioning though the effect was marginally statistically significant. Having team policies was also associated with an 8-percentage-point increase in the percentage of appointments made within 3 days of the desired date. However, having a multidisciplinary team, structures to assist with post-ED transitions, patient centeredness had a negative effect on this outcome. For the outcome of 2-day post-discharge contact, having a multidisciplinary team and structures in place to improve enhanced post-hospital transitions was associated with higher rates of 2-day post-discharge contact (by 16 and 27 percentage points, respectively). Paradoxically, enhanced access was associated with worse performance on this measure. Finally, policies related to team communication and functioning and high-risk registries were associated with a one-percentage-point and 0.7-percentage-point reduction in ED visits, respectively, whereas enhanced access and using methods for post-ED transitions was weakly associated with increased ED visits.

Discussion

We examine the effect of medical home implementation on primary care delivery at the VHA. Using a mixed-methods approach, we combine detailed interview data on medical home-related structural changes and the success of such changes with quantitative data on three organizational processes of care and one patient outcome.

We find that numerous structural changes were made by primary care providers to support implementation of the medical home model and the number of PCPs making these changes increased substantially over time. We also find that measures of medical-home-related processes of care improved over the first 2 years of medical home implementation and, for two of the three that we measured, these improvements were concentrated among providers actively implementing the medical home. However, in full adjusted regressions, only one of these three organizational processes consistently improved in association with medical home implementation. The association between structural changes and processes of care was also inconsistent, with some structural changes associated with improved processes and others associated with worsening processes. Finally, we find no association between medical home implementation and rates of ED use by patients.

These findings add substantially to the current literature on the effects of the medical home. While most prior research has measured the medical home as an on-off switch (Peikes, Zutshi et al. 2012), we evaluate not only whether the medical home works but also whether the success of implementation matters, and whether some structural changes to support the medical home model matter more than others. Not surprisingly, our findings reveal that the medical home model is complex to measure, complex to implement, and the response to implementation is not always predictable.

As one of the first articles to examine the effect of the medical home implementation in the VHA, these findings also add significantly to knowledge of the largest medical home transformation to date in the United States. While these results are somewhat mixed, they are encouraging. Only 2 years into the medical home implementation at the VHA, these results provide evidence that significant changes have occurred in how primary care is being delivered, some of which is improving selected care processes. While we measured three organizational processes that are expected to improve with medical home implementation at VHA, additional changes in processes of care are expected that were not included here, including use of personal health records, telehealth monitoring, and use secure messaging between patients and providers. More work is needed to examine the effects of medical home implementation across all of these processes of care. In addition, while we found mixed effect of the medical home on one measure of utilization (ED visits), it is early in the process, which is expected to continue for an additional 2 years. Follow-up work to follow changes in ED visits and other measures of health care utilization and costs will be needed.

Although these data provide encouraging results regarding the VHA's transformation to the medical home model, the results are also somewhat puzzling in that some of the observed improvements in processes of care could not be linked to the implementation of the medical home. For example, the descriptive data show that the proportion of primary care visits occurring by phone increased substantially over the study period and the rate of increase was double among PACT providers what it was among non-PACT providers. However, in regression models our measures of PACT implementation were not associated with a significant improvement in the use of telephone visits. While statistical power may have limited our ability to detect an effect if it existed, the magnitude of the effect was nonetheless small, at less than 1 percentage point. One possibility is that there was a large degree of spillover from PACT to non-PACT providers. Indeed, in our regression models we note that for several of the organizational processes there are strong secular trends toward improving performance that are common to the PACT and non-PACT providers (see Tables S1–S3). As such, it may be the case that it was not the PACT training and associated changes in care delivery that drove changes in organizational processes. Rather, the new emphasis on improving these processes may have changed the delivery of care regardless of whether the PCP was actively adopting the PACT model. In essence, the widespread emphasis on PACT implementation and its goals throughout the VHA might have created a Hawthorne effect, where all PCPs changed their patterns of care regardless of PACT implementation.

We also found that many of the specific structural changes in care were not linked to improved care delivery or were paradoxically linked to worsened processes of care. While the reason behind these contradictory findings is not explored in this research, one possible explanation is that PCPs struggle to implement all aspects of the medical home simultaneously. Changing the structure of primary care delivery transiently increases workload as primary care staff learns a new system. Thus, it is possible that activities focused on improving one area may worsen care delivery in other areas. Regardless of the reason for these contradictory results, they highlight the complexity and difficulty in successfully implementing the medical home.

Several limitations to our study should be noted. First, this is an observational study and the associations between medical home implementation and changes in processes and outcomes that we find are just that—associations. While we take advantage of the staggered start of the medical home across VISN 4 and include PCP fixed effects (allowing each provider to serve as a control for him or herself) and time fixed effects (to net out secular trend), these methods do not allow us to assess whether medical home implementation causes changes in care. Second, we study the effects of the medical home during a short period after its introduction. The implementation of the medical home in VHA is a work in progress and is not yet complete. Thus, any changes in care documented here (or lack thereof) may simply be the result of a necessary learning process that providers and systems of care must go through when changing their care delivery model and are thus likely to change as the system matures. Third, we study the effect of medical home implementation on a limited set of outcomes, most of which are organizational processes but not patient outcomes. While we find limited effect on the outcomes we examine, the effect of medical home implementation on important outcomes such as health care utilization, costs, continuity of care, and patient satisfaction has not yet been determined. Finally, there are potential limitations to our qualitative data. It may not provide a comprehensive and unbiased measure of medical home implementation for all PCPs. In our follow-up interviews we usually talked with just one person, who was often a champion of PACT. The extent to which this person knew the consistency with which processes of care were changed may be limited. It may not have captured the extent to which medical home practices spilled over to the non-PACT PCPs, which could bias our results to the null. In addition, though we emphasized the neutrality and confidentiality of the interviews, interviewees may have tended to overstate the positives.

Despite these limitations, this study provides an important first look at the experience of medical home implementation within the VHA. It also provides new insights into the complex and unpredictable nature of medical home adoption. As with any delivery reform, changing a model of care is a challenging process. Early results should not temper enthusiasm for improving primary care in the United States. Rather, it provides an opportunity to critically examine and improve upon current models of care.

Supporting Information

Additional supporting information may be found in the online version of this article:

Appendix SA1: Author Matrix.