Engaging multilevel stakeholders in an implementation trial of evidence-based quality improvement in VA women’s health primary care

Abstract

The Veterans Health Administration (VHA) has undertaken primary care transformation based on patient-centered medical home (PCMH) tenets. VHA PCMH models are designed for the predominantly male Veteran population, and require tailoring to meet women Veterans’ needs. We used evidence-based quality improvement (EBQI), a stakeholder-driven implementation strategy, in a cluster randomized controlled trial across 12 sites (eight EBQI, four control) that are members of a Practice-Based Research Network. EBQI involves engaging multilevel, inter-professional leaders and staff as stakeholders in reviewing evidence and setting QI priorities. The goal of this analysis was to examine processes of engaging stakeholders in early implementation of EBQI to tailor VHA’s medical home for women. Four inter-professional regional stakeholder planning meetings were conducted; these meetings engaged stakeholders by providing regional data about gender disparities in Veterans’ care experiences. Subsequent to each meeting, qualitative interviews were conducted with 87 key stakeholders (leaders and staff). Stakeholders were asked to describe QI efforts and the use of data to change aspects of care, including women’s health care. Interview transcripts were summarized and coded using a hybrid deductive/inductive analytic approach. The presentation of regional-level data about gender disparities resulted in heightened awareness and stakeholder buy-in and decision-making related to women’s health-focused QI. Interviews revealed that stakeholders were familiar with QI, with regional and facility leaders aware of inter-disciplinary committees and efforts to foster organizational change, including PCMH transformation. These efforts did not typically focus on women’s health, though some informal efforts had been undertaken. Barriers to engaging in QI included lack of communication across clinical service lines, fluidity in staffing, and lack of protected time. Inter-professional, multilevel stakeholders need to be engaged in implementation early, with data and discussion that convey the importance and relevance of a new initiative. Stakeholder perspectives on institutional norms (e.g., gender norms) and readiness for population-specific QI are useful drivers of clinical initiatives designed to transform care for clinical subpopulations.

BACKGROUND

Women are the fastest growing cohort of Veterans. While the total Veteran population is projected to decrease from 2013 to 2043, the percent of women Veterans is expected to increase by 18% [1]. Over half of women Veterans who served in recent conflicts are users of the Veterans Health Administration (VHA).

With the growing population of women Veterans, women’s health (WH) in VHA has become increasingly important [2]. One particular area of attention has been WH primary care, where there is recognition that the patient-centered medical home—called Patient-Aligned Care Teams (PACT) in VHA [3]—requires tailoring for this population [4], as well as for other populations with special needs [5–7]. Tailoring is needed for women Veterans because they are different than their male counterparts in several important ways [8]. For example, women Veterans are younger (median age, 50 years) than male Veterans (median age, 65 years) [9], and women need gender-specific services. The experience of military sexual trauma (MST) is more common among women Veterans than men (approximately 38% of women versus 4% of men) [10], necessitating trauma-sensitive care. Furthermore, care coordination for women Veterans is complex because many must rely on both VHA and non-VHA providers to get their healthcare needs met [11].

Policy-mandated elements of VHA WH care include comprehensive care provided by designated WH providers, privacy, and having a chaperone present during gender-specific exams; suggested tailoring of PACT for women Veterans includes longer appointment times and smaller patient panels for providers who care primarily for women [8, 12]. In sum, primary care for this population requires considerable reorganization of care and tailoring of services in order to ensure access to needed services, reduce delayed and missed care, and prevent women from discontinuing a course of treatment or leaving VHA care altogether [13].

The process of reorganizing and tailoring PACT for women Veterans requires meaningful and continual engagement of stakeholders in order to be successful and sustained [14, 15]. Studies of PACT implementation have found that “vertically aligned expectations,” i.e., clearly articulated leadership support and engagement throughout an organizational hierarchy, are critical to provider performance in this medical home model [16]. In a highly hierarchical organization such as VHA, multilevel stakeholder engagement means engagement of leadership from the regional level to facility and service-line leadership, to clinic-level leadership and providers delivering care to women Veterans [2]. Without such comprehensive multilevel stakeholder engagement, the intentions of a transformative initiative such as PACT may not be realized in day-to-day practices, thereby compromising or inhibiting high-quality care.

A systematic implementation strategy that fosters multilevel, inter-professional engagement, such as evidence-based quality improvement (EBQI), can facilitate organizational change and spread of innovations that enhance care quality, as demonstrated in numerous studies. EBQI is a form of continuous quality improvement designed specifically to improve uptake of evidence-based programs (EBPs) in routine practice. Rubenstein and colleagues [17] describe four key components of EBQI: (1) research-clinical partnerships; (2) top-down (leaders) and bottom-up (staff) engagement of local site personnel to adapt EBPs to the local context through consensus development and group decision-making among researchers and local-level leaders, providers, and staff; (3) use of research evidence for QI targets and behavior change methods; and (4) ongoing support for EBQI teams from researchers serving as technical experts. A key tenet of EBQI is that multiple levels of leadership must be aligned on priorities, including regional-, facility-, and clinic-level stakeholders. To promote evidence-based priority setting for development of local site level QI efforts, EBQI researchers assemble key literature and local data on the problem to be addressed. To set priorities, stakeholder groups come to consensus using modified Delphi methods. The consensus process is informed by both data and facilitated discussion. To date, EBQI efforts have been successful at improving uptake of evidence-based clinical practices such as collaborative care [18, 19], supported employment [20], and PACT [21, 22]. Despite increased use of EBQI and other similar strategies such as QI collaboratives, more precise analyses of active QI components are still needed in order to understand how this multifaceted approach contributes to improved care [23].

This article focuses on stakeholder engagement in the initial process of launching Women’s Health-PACT (WH-PACT), a 12-site cluster randomized controlled trial of the effectiveness of EBQI in tailoring PACT to the needs of women Veterans [24] (NCT02039856). All 12 participating sites (two EBQI and one control in each of four geographic regions) are members of the VHA Women’s Health Practice-Based Research Network (PBRN), a 60-site national network of VHA facilities designed to ensure that women Veterans are adequately sampled and represented in VHA research and QI [25]. Increasing evidence suggests that stakeholder engagement in the context of PBRNs can promote implementation of clinical initiatives [26].

In WH-PACT, EBQI efforts focus on facilitating local QI activities in order to help sites tailor PACT to women Veterans and thereby improve their patient outcomes. WH-PACT EBQI is a multifaceted implementation strategy [27] comprised of six main activities: (1) four regional inter-disciplinary stakeholder planning meetings using expert panel techniques to identify QI priorities for women Veterans and to develop “QI roadmaps”; (2) formative evaluation [28], including qualitative interviews conducted with key stakeholders, i.e., leaders and staff; (3) development and training of a local QI champion and QI team members at EBQI-assigned sites to pursue one or more QI projects from the roadmaps; (4) ongoing practice facilitation and expert review and feedback on local QI proposals and progress; (5) monthly across-intervention site calls to facilitate collaboration and spread of effective QI innovations; and (6) technical work groups designed to provide additional support in evidence-based, stakeholder-prioritized areas [24]. The planning meetings (activity 1) and key stakeholder interviews (activity 2) focused on educating and raising awareness among stakeholders and learning about readiness [29] for implementation of EBQI. This paper reports on findings from these two activities, both of which took place prior to launching QI training and projects at the sites.

METHODS

Site selection and settings

All participating sites are members of the PBRN. Each PBRN site has a Site Lead, typically a clinician or researcher, who facilitates women’s health research and QI at the site by interacting with medical center and local clinical leaders. To recruit sites for this study (as described further in [24]), the Principal Investigator (EY, a senior women’s health services researcher) collaborated with the PBRN Director and the Site Leads to ask regional leadership if the PBRN sites in their respective regions could participate in the trial. All agreed to participate, but one region dropped out prior to study launch and was replaced by another.

Twelve VHA medical centers (VAMCs) were randomly assigned to EBQI or control (PACT implementation as usual) in an unbalanced 2:1 ratio with each region. Randomization was conducted by the study biostatistician, who used www.randomization.com with a random permutated block of three and a seed of 15,356 to start the random allocation sequence [24].

Key stakeholder sample

The purposeful sample was designed to include key stakeholders, defined as individuals who are “responsible for…healthcare-related decisions that can be informed by research evidence” [30]. Eligible roles included the following: regional leadership (directors, chief medical officers, regional service line chiefs, regional Women Veteran Program Managers, chief quality officers, chief informatics officers); facility leadership (directors, chiefs of staff, facility-level primary care/PACT service line chiefs); facility-level clinical leaders such as WH Medical Directors; facility-level Women Veteran Program Managers (a mandated staff role at each VHA medical center, usually filled by a clinician); and PBRN Site Leads. In many cases, individuals held more than one role, e.g., some PBRN Site Leads also hold clinical and/or leadership roles. Individuals in these roles were identified using publicly available information as well as lists provided by the Site Leads. Using a snowball sampling approach [31], during the course of interviews, additional individuals in the eligible roles were recommended by key stakeholders due to their women’s health and/or PACT expertise.

In total, 91 individuals were recruited across the 12 sites, and 87 (96%) completed individual interviews, with an average of five participants per site (range, 3–8 participants per site). Interviews were conducted with: regional leaders, n = 25; facility leaders, n = 17; clinical leaders, n = 20; and other staff, n = 25. Some roles are minimally represented across sites, typically because the roles did not exist in all sites or because the roles were not filled at the time of the interviews.

Study design and procedures

This analysis utilizes two sources of qualitative data: notes and reports from the stakeholder meetings and individual semi-structured interviews with the key stakeholders described above.

Notes and reports from the stakeholder planning meetings (activity 1)

The PI convened four in-person inter-professional stakeholder planning meetings (one per region). The meetings took place between May and August 2014. Prior to each meeting, stakeholders received background reading and a pre-panel rating form on the importance and feasibility of (1) senior leader involvement in women’s health, (2) tailoring access to care to meet women Veterans’ needs, (3) tailoring PACT to meet women Veterans’ needs, (4) delivering comprehensive care for women Veterans (including gender-specific preventative care and managing care for women with MST histories), and (5) gender-sensitive providers, staff, and care environments. Responses were collated prior to each planning meeting.

At each meeting, the PI presented a summary of research on women Veterans’ health and health care, regional data on gender differences in ratings of care and local data about women’s health practice arrangements, and characteristics of the local women Veteran patient population. Then the PI facilitated a discussion of the pre-panel rating results. Regional QI roadmaps were generated on the basis of the stakeholder panel meeting results; these roadmaps were reviewed, revised, finalized, and distributed to the sites. All planning meetings were recorded and transcribed with the expressed permission of the attendees.

Key stakeholder individual interviews (activity 2)

Following completion of each planning meeting, key stakeholders from the participating sites (many of whom attended the stakeholder planning meetings) were recruited via email for voluntary interviews. All interviews were conducted by phone by the lead author [AH], an anthropologist and expert qualitative methodologist, and at least one team member. Interviews were conducted between May and December 2014. On average, interviews lasted between 45 and 60 min. With participants’ verbal consent, all interviews were recorded and transcribed verbatim. All participants provided verbal consent to be recorded.

The semi-structured interview guide included questions about the structure of WH and primary care, the evolution of PACT in general and in women’s health, past and current research and QI efforts, the use of data and performance metrics to guide change efforts, and ideal care for women Veterans.

All procedures were reviewed and approved by the VHA and UCLA Institutional Review Boards.

Analysis

Planning meeting minutes and reports based on the transcripts were prepared by [IC] and integrated into the overall analysis by the lead author.

Interview transcripts were summarized using a template of interview domains, and summary points were put into a spreadsheet for matrix analysis [32]. Transcripts were then imported into ATLAS.ti (version 7, Scientific Software Development GmbH, ©2001–2016). An initial codebook was developed based on the interview guide, the matrix analysis, and the independent open coding of three transcripts by two researchers. Coding was compared for consistency and as appropriate, the codebook was revised to clarify construct definitions or better highlight critical themes [33]. All transcripts were independently coded by at least two members of a four-person research team including [AH, JB, CC, EC] using the final codebook. Memos were written in ATLAS.ti to document analytic observations. Any coding discrepancies were discussed until consensus was achieved. This analysis focuses on interview narratives related to QI efforts and the use of data to guide those efforts, particularly related to women’s health or gender disparities. Codes associated with those interview domains were analyzed to identify differences by role, facility, and region, using the Document Family tool in ATLAS.ti, which facilitates comparisons by categories.

RESULTS

Stakeholder planning meetings

During each planning meeting, stakeholders were encouraged to respond and react to the data pertaining to gender differences in patient access to care and the overall lower ratings of women Veterans compared to male Veterans. They consistently remarked that they had not been exposed to the breakdown of patient ratings for their own region by gender, and were impressed and typically alarmed by the findings of gender disparities, e.g., related to satisfaction with care. The PI’s presentations of the data to these inter-professional groups had a palpable impact on stakeholders’ engagement in the process of determining priorities for QI care targets and requesting technical assistance (see Table 1), as well as buying in to the EBQI approach that the intervention sites would be using.

Table 1.

Priority QI target selected and technical assistance requested, by region

	Region 1	Region 2	Region 3	Region 4
Priority QI target	Improving women’s preventive health care	Improving access to care for women Veterans	Improving cardiovascular risk reduction among women Veterans	Increasing PACT women’s health medical education
Technical assistance requested	Integration of women’s mental health expertise	Management of high-risk, complex women Veteran patients	Care coordination for women Veterans	Care coordination for women Veterans

Key stakeholder interviews

General QI readiness

Interviews revealed that stakeholders were generally familiar with a variety of QI efforts and mechanisms, with higher level leaders (e.g., regional leaders, facility directors) being aware of multiple regional governance and inter-disciplinary committees, as well as efforts to foster organizational change, including patient-centered medical home (PCMH) transformation. QI efforts typically involved reliance on performance measures for determining the focus of change or improvement efforts: a regional leader said, “You probably are aware from both your interview process and reading the newspaper that VA is a very data intense, data-rich healthcare system. We have, if anything, too many measures.” Another regional leader was familiar with several different committees at the regional level, which all “rolled up” into the highest regional level of quality management oversight. A Facility Director explained that her facility has a “variety” of committees: “That’s what they do—quality and performance improvement, primary care issues, women Veterans committee, women’s health committee—so there’s numbers of committees where the PACT model, primary care access, primary care processes would be discussed. And then that would raise up to leadership.” Facility leaders described processes of (1) facility-level executive leadership having managers transmit performance information to their clinical areas (a “top-down” approach) or (2) clinical areas taking on performance improvement efforts that did not necessarily reach the executive leadership level (a “bottom-up” approach). A Facility Director characterized the top-down approach: “We expect managers to take the [performance measure] information out there to their areas.” Another Facility Director exemplified the bottom-up approach, explaining that there are a “ton” of QI teams and groups at his facility: “Every section has multiple performance improvement projects going on at any given time, some of them a result of an RCA [Root Cause Analysis], and then we have to follow up…but not all of it comes to the Executive Leadership Board, but yeah, there’s a ton of performance improvement stuff going on.”

Specific to PACT, a Chief of Staff noted that part of their PACT initiative has been to allow teams to set aside time for quality improvement, specifically Plan-Do-Study-Act (PDSA) cycles which are foundational to QI: “I certainly support [setting aside time for QI] and allowed them to carve out that time, but the teams themselves have actually again seized the opportunity and they’ve been using that time and they’ve developed their [projects], so we have a lot of these PDSAs going.” As described below, some stakeholders described lack of time as a barrier to consistent engagement in QI.

Range of women’s health-related QI initiatives

While QI efforts were common, facility-level stakeholders typically described general QI efforts, rather than women’s health QI efforts. This “generic” approach was often attributed to a desire to address the needs of all patients, not just one subpopulation or clinic. A primary care/PACT lead explained:

Women’s health is part of primary care…In most cases our QI topics are pretty generic, I mean they’re not gender-specific. Primary care includes women. The VA has struggled with that concept, but we try not to have them stand separate. It’s all about taking care of your patients, and there’s women everywhere. I mean, if there’s a specific project that really has to address the women’s health clinic, for example, then of course, they will work on that. But for the most part, the initiatives that we do are broader and not so gender-specific, and they’re more applicable.

A Facility Director mentioned having “instituted Lean [34] here to do rapid process improvement workshops,” but, he said, “I think that a lot of things happen without it being formal…I haven’t felt a need to stop the presses and do a big push to change things in the women’s clinic.” Similarly, a PBRN Site Lead in a different region noted, “We haven’t done any [research or QI] that’s directly related to women’s health,” and another PBRN Site Lead said, “There are lots of QI projects…but not necessarily concerted efforts through women’s health.”

Although most stakeholders described more general QI efforts, some sites had engaged in women’s health-related projects, with the most common focus being decreasing gender disparities in lipid control (a topic of national focus). A Regional Director said, “This year we’ve been working on lipids…We felt that our women’s health lipids management was an opportunity for improvement.” Similarly, a Woman Veteran Program Manager described: “We have done several different types of things, like this past year we took one of the performance measures and we’ve been really targeting it…it was the [low-density lipoproteins] for diabetic and ischemic heart disease in women.” Several sites had worked on increasing timely turnaround of test results (e.g., Pap smears) to their patients.

Some stakeholders who were working on local WH-related projects described keeping their efforts informal and “in-house.” A Primary Care Physician Lead said, “We’re very blessed in working together as a team. It’s not like it needs to go up through Central Office [VA headquarters] in order for us to institute something. We just do it at our level and follow it.” A Women’s Health Medical Director said that her team had been doing “very effective” things to address clinical concerns, but that the efforts were “to be honest, driven by our own assessment of clinical need rather than data per se.” Women’s health stakeholders described “informal efforts.” For example, a Woman Veteran Program Manager said that they “evaluate data and do something, but not a formal project…not a formal design.” There was a sentiment that keeping these informal efforts local to the WH clinic or team also kept them simpler.

Challenges in QI engagement

Three main challenges with routinely engaging in QI were noted mainly by facility- and clinic-level stakeholders (providers, staff) rather than regional leaders, who seemed to consider involvement on multiple committees and in multiple efforts as part of their roles and responsibilities. For providers and staff, challenges included the following:

1. Lack of coordination and communication across clinical service lines due to “silos”: A Primary Care Physician Lead said, “Everyone’s in their own little silo, and there isn’t necessarily good communication across the board…” Similarly, a Women’s Health Medical Director said, “We [women’s health] are like in a silo and it creates a tough work environment, especially for the providers.” A Primary Care Physician Lead similarly shared, “We have three silos…the providers, the nursing staff, and the clerical associates, all in their own silos, and it creates challenges in trying to all be aligned doing the same thing together, and trying to have everybody see the mission together.”

2. Fluidity of staffing (including changing clinical, facility, and regional leaders): Several stakeholders noted multiple changes in roles, especially changing leadership and leaders being detailed to other facilities and regions. Some noted the challenges of having several “acting” (temporary) leaders with whom it was difficult to establish relationships. A PBRN Site Lead stated, “I can’t even keep track of the key players.” Similarly, a Primary Care Physician Lead said, “This is such a huge healthcare system with so many different moving parts and so many different people playing so many different roles that it’s kind of hard to keep up with what everyone is doing.”

3. Lack of time for clinicians to engage in QI: A PBRN Site Lead noted that clinicians in women’s health do not have protected time, but they are enthusiastic, especially as indicated by their “willingness and even eagerness to participate in this project.” Similarly, another Site Lead expressed the challenge related to protected time: “It’s hard to get protected time to do something that is considered above and beyond your job, so if you have that protected time and consider that to be an integral aspect of your clinical practice—because leadership supports it—that would go a long way.” Lack of protected time may have contributed to another challenge: lack of commitment to QI. A Women’s Health Medical Director noted, “I have to really push these [QI] things and I wish that more people on the team were interested in QI.” Notably, a facility-level leader pointed out how important “time set aside” can be for PACT implementation: “I think we do better than many facilities in terms of our implementation. We do have weekly dedicated PACT meetings where they work on case management and practice improvement, the sorts of things that often fall by the wayside when you’re dealing with a busy practice. We do have time set aside and we do have PDSA cycles and our teams really are doing very well that way.”

DISCUSSION

The goal of this analysis was to examine processes of engaging stakeholders in early implementation of tailoring VHA’s medical home, PACT, to the primary care needs of women Veteran VHA users. This minority but growing patient population has unique primary care needs that necessitate a tailored QI approach which recognizes and addresses the gender disparities in care, the need for gender sensitivity, and the unique contexts in which women’s health care is delivered in the healthcare system. In order to prepare sites and the research team for the partnership entailed in EBQI [35], we needed to engage key stakeholders by (1) educating and raising awareness about women’s experiences of VHA care and (2) discussing with them their past and current experiences with QI. Together, these activities provided a critical foundation for subsequent efforts to work with sites, design their QI training, and select and execute QI projects in women’s health primary care.

The in-person regional stakeholder planning meetings provided opportunities for stakeholders from multiple levels to come together and learn about gender differences in patient ratings of care and ways to ameliorate those disparities using EBQI. The fact that these data were presented and explained in person by a nationally esteemed “change agent” [36]—the study PI—contributed to the collaborative selection of priority QI targets and to the identification of technical assistance needs that the PI was in a position to meet. This study’s approach of using data to inform high-level EBQI planning discussions has been used successfully by others [18, 37]. In our study, it was important to hold these meetings in-person [23] and very early in the study timeline (prior to EBQI training), because senior leadership awareness and support was critical to site-level stakeholders’ confidence with pursuing QI specifically in women’s health primary care [38]. In contrast, Salyers and colleagues [39] found that kick-off meetings were more appropriate after individuals were trained in the intervention in order to avoid generating enthusiasm prematurely. It will be important to follow up on the timing of each EBQI activity with respect to the range of innovations considered and the success of implementation [40].

Key stakeholder interviews revealed general QI readiness at the sites in the sense that stakeholders were at least familiar with QI efforts and many had direct experience with QI projects. Women’s health QI readiness was less consistently observed across the sites. Some facility-level stakeholders explicitly expressed that their sites’ QI efforts were more general and broadly applicable, relevant to all of primary care. Women’s health QI efforts seem to have been more local in the sense that they were not always formal initiatives and therefore were not known by leaders outside of women’s health or primary care; this finding helped to steer our WH-based EBQI efforts toward capitalizing on local (i.e., WH team) momentum, as well as helping WH teams to communicate their successes to leadership. Berry and colleagues [41] found that small primary care practices (akin to VHA WH clinics) serving disadvantaged populations achieved aspects of the PCMH model through flexible, informal strategies. Recent research suggests that global improvement initiatives may yield less than efforts targeted to specific organizational units (e.g., WH primary care) and tailored to those units’ specific strengths and weaknesses [42]. Future analyses in our study will assess the extent to which WH-specific QI efforts became formalized as a result of EBQI, and whether formalization resulted from—or resulted in—shifts in multilevel stakeholder engagement.

The finding that some facility-level stakeholders did not necessarily perceive a need for WH-specific QI speaks to a persistent issue of the extent to which organizational stakeholders perceive a need for gender-specific primary care. A qualitative study of mental health service delivery for women Veterans similarly found a lack of consensus around the need for gender-specific care [43]. Moreover, it speaks to the importance of understanding institutional norms when embarking on implementation of initiatives that focus on clinical subpopulations [20]. Tannenbaum and colleagues [44] have specifically argued that addressing institutional gender norms is critical to implementation efforts, particularly in understanding “how…institutionalized gender influence[s] the way in which an implementation strategy works, for whom, under what circumstances and why?” We see our multisite study as a prime opportunity to examine the evolution of institutional gender norms—and the influence of those norms—over the course of implementation.

By bringing stakeholders together and following up with individual interviews, we surmise that the stakeholder planning meetings and qualitative interviews served as platforms for intensive education and engagement. Future analyses will investigate the mechanisms by which these activities generated multilevel support [45] for using EBQI to address women’s health needs. We also capitalized on the opportunity to advance the intervention sites in their own QI capabilities as PBRN sites, postulating that these capabilities would extend well beyond the life of the study and would therefore be a worthwhile investment to support future research and QI efforts [46]. Future studies of QI readiness might benefit from additional methods to look systematically at key constructs such as relative commitment and implementation capability [29], including the value and importance of protected time to engage in QI [47, 48]. In-depth investigation of the ways in which long-term stakeholder-engaged research fosters a “cycle of trust” among researchers and organizational stakeholders is also warranted [49].

The analysis described herein is limited in two primary ways. First, although all key stakeholders were asked about QI efforts and the use of performance measures, this was only one of several sections of the interview guide, thus potentially limiting the depth of information we gathered about these topics. This is a recognized trade-off with semi-structured interviews that are designed to provide more breadth rather than depth of topic coverage [31]. Second, all interviews were conducted by telephone rather than in person, which potentially hindered the probing that might have more readily occurred in face-to-face interactions [50].

Several reviews of QI success indicate that stakeholder engagement is critical in all phases of improvement work [45, 51, 52], and a growing number of implementation studies also confirm the importance of an engaged approach to implementation [53]. This study represents the marriage of a stakeholder- and data-driven implementation strategy—EBQI—with the inherently partnered PBRN infrastructure, which together are intended to generate improved quality in VHA women’s health primary care, with all of the medical home features known to result in favorable health outcomes and patient experiences of care [22].

Compliance with ethical standards

Study funding sources

Support for this research was provided by VA HSR&D Service, Veterans Health Administration through the CREATE initiative (CRE 12-026) and the VA HSR&D Center for the Study of Healthcare Innovation, Implementation, and Policy (CIN 13-417). Development and implementation of the EBQI intervention is being supported through annual Memoranda of Understanding with funding from VA Women’s Health Services (WHS) in the Office of Patient Care Services, Veterans Health Administration. Dr. Yano’s effort was funded by a VA HSR&D Senior Research Career Scientist Award (Project #RCS 05-195).

Conflicts of interest

The authors declare that they have no conflicts of interest.

Reporting and control of data

The findings reported have not been previously published and the manuscript is not being simultaneously submitted elsewhere. The data has not been previously reported. The authors have full control of all primary data and agree to allow the journal to review their data if requested. The views expressed within this study are solely those of the authors and do not necessarily represent the views of the Department of Veterans Affairs or the U.S. Government.

Ethics approval and consent to participate

This study protocol has been approved by the VA Greater Los Angeles Healthcare System IRB-B Subcommittee (IRB Chair, Paul Shekelle, MD, PhD) (approval number 2013-040589). Individual study components, including key stakeholder interviews, were reviewed and approved by IRB-B as serial project modifications to an umbrella IRB application under the study title. For each study component, we obtained IRB approval for waivers of documentation of consent for all data collections. Consent was provided verbally and recorded in the case of interviews. The VA Central IRB (Washington DC) also determined that the participating VA medical centers (n = 12) were not engaged in research and that the intervention (or implementation strategy, evidence-based quality improvement [EBQI]) was QI. Documentation of IRB approval (and in VA settings, Research and Development Committee approval as well) and Central IRB determination of non-research for the implementation strategy are available on request. The UCLA Institutional Review Board also reviewed and approved the key stakeholder interview component. No individual person’s data in any form are or will be included in this or subsequent manuscripts. Only aggregated data will be reported. For key stakeholder data, non-attributed or identifiable quotes may be reported in addition to thematic summaries.

Statement on welfare of animals

No animals are or will be involved in this study protocol. Similarly, use of animal or human data or tissue is not applicable to this study protocol.

Helsinki statement

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.