Innovative Implementation Studies Conducted in US Safety Net Health Care Settings: A Systematic Review

Abstract

Little is known about dissemination and implementation in safety net settings. The authors conducted a literature review of innovation/implementation studies in US safety net health care settings between 2008 and 2017. Each article was coded for (1) intervention characteristics, (2) implementation stage, (3) internal versus external ownership, and (4) prespecified implementation outcomes (eg, acceptability and fidelity). Twenty studies were identified; the majority were implemented within community clinics or integrated safety net systems (15 articles), most involved care process improvements (13 articles), and most were internally developed (13 articles). The internally developed innovations reported fewer barriers to acceptability among staff/providers, higher leadership involvement and organizational alignment, greater amounts of customization to the local setting, and better sustainment. Future work should harness the high levels of alignment and acceptability in implementation research within safety net settings, with an eye toward maintaining fidelity to facilitate dissemination across sites.

Innovation is often defined as “the implementation of a new or significantly improved product (good or service) or process, a new marketing method, or a new organizational method,”¹ often to replace older processes altogether.² Alongside the concept of innovation, translating evidence into practice requires changes in current practices and is known to occur variably and slowly.^3–6 The concepts of innovating and translating evidence into practice are often discussed separately, but especially in underresourced health care settings, the delivery of new and improved care processes is often intertwined with the implementation of practices known to work in other health care settings.³

Implementation of innovations in the health care safety net is particularly important because these systems face additional challenges in order to meet the demands of providing high-quality care to diverse, low-income populations, under conditions of significant resource constraints.^7,8 Specifically, safety net settings care for patients who have higher levels of both medical and social complexity, yet they operate primarily on federal and state funds, which reimburse health care services at lower rates than private insurance. Therefore, implementation in safety net health systems often requires specific efforts to tailor or adapt practices.^3,7,9

There have been several overarching frameworks that attempt to provide a means to understand the complex process of implementing successful new interventions and programs into health care settings.^10,11 Various factors, such as organizational leadership support and resource availability, affect the successful spread of practices in health care,^3,12,13 and much has been written about these factors.^10,14–20 However, few studies directly compare or review implementation processes across multiple programs.

The study team completed a review of articles reporting on implementation of interventions or programs in US safety net settings. The objectives of this review were to both describe the landscape of this body of literature as well as to examine how specific implementation outcomes (such as fidelity and sustainment) are addressed within these studies.

Methods

Search Strategy

The study team searched both PubMed and Google Scholar databases for “safety net” or “low income” in combination with “innovation”/“innovate” and “implement”/“implementation” between January 2008 and April 2017, to be sure to capture articles post health care reform. The team also searched the reference lists of included articles to identify additional articles describing different elements of the same program implementation.

Inclusion Criteria

First, in a joint meeting, 2 coauthors (GG, CRL) collaboratively screened 40 abstracts from the initial search to generate the inclusion criteria categories and definitions. Then, the remaining abstracts were divided among coauthors to eliminate articles that did not meet these criteria, which included full-text review of any articles that could not be categorized from the abstract alone. Any uncertainty was resolved by discussion between at least 2 coauthors.

Articles were included as follows: (1) original research (ie, not commentaries/editorials), (2) US focus (ie, not global/international focus), (3) presence of empirical data (eg, not case studies without qualitative or quantitative outcome assessments), (4) implementation of a specific program or intervention (ie, not broad processes covering multiple programs), (5) conducted within safety net clinical health care settings (ie, based at least partially at a health care system serving predominantly low-income or publically insured patient populations), (6) containing staff/provider- or organizational-level data (ie, not solely patient outcome data), and (7) focused on local programs/interventions (ie, not evaluating the impact of a broader health care policy).

Because the criteria excluded articles without empirical data or intervention descriptions, this ensured that the articles met the Standards for Quality Improvement Reporting Excellence (SQUIRE) criteria.²¹

Article Coding

For each article, 4 coauthors (CRL, MAH, PW, MW) then summarized (1) the main intervention (a summary of the intervention, including whether it was an evidence-based practice), (2) study design (qualitative, quantitative, or mixed methods, as well as a study sample description), and (3) implementation setting.

The study team also summarized the stage of implementation. The vast majority of studies included data collection in an active implementation phase. An additional set of planning studies encompassed research that had a “protocol” focus that explained future implementation data collection or pre-implementation data to inform a future program or intervention.

Studies were categorized as “internal” versus “external.” Externally developed programs were defined as partnerships/collaborations with outside institutions (such as interventions proposed by external researchers). Internally developed programs were described as resulting from a specific internal health care system need. Further specified within the internal category were any studies that were implemented because of an internal motivation but funded by an existing grant opportunity (“internal with grant support”) or conceptualized and executed through an equal partnership between the clinic and an outside entity (“cocreated” projects).

Implementation Outcomes.

Finally, a set of established implementation outcomes were used to explore major themes within the active implementation studies: acceptability (at the staff/provider level in the context of their existing work/roles), appropriateness (at the level of both organizational alignment and fit/relevance), fidelity (in terms of customization vs standardization of the program implemented), cost (unforeseen financial barriers to implementation), and sustainment (program continuation in the future).²² The study team coded whether this implementation outcome was mentioned/discussed in the article and, if mentioned, how this implementation outcome was perceived as a barrier or facilitator to success.

The team also examined each of the implementation outcomes by the internal versus external development categorization because previous literature has documented improved success based on internal motivation and ownership of the program.^23–27

Results

In total, 346 articles were identified using the search terms; 27 were included in the final sample: 20 active implementation and 7 planning articles (Figure 1).

Figure 1.

Manuscript selection flow diagram.

Active Implementation Articles

Of the 20 active implementation studies (Table 1), 19 programs are summarized because 2 of the articles discussed implementation of the same program.^28,29 Eleven articles employed mixed methods.^30–40 This included in-depth interviewing of health care system leaders and/or staff as the most common methodology,^{28–30,32,33,36,41–46} followed by descriptive data collection involving observations,^39,40,42 document review,³³ and/or detailed case studies of implementation.^{31,34,35,37,38}

Table 1.

Overview of Active Implementation Studies^a.

Author	Year	Evaluation Methods	Intervention/Program	Clinical Setting	Implementation Stage
Bolin JN	2013	Mixed methods (patient surveys, key stakeholder interviews, quantitative usage tracking)	Patient education, technology (touchscreen diabetes education kiosk)	2 community clinics, 1 community center, and 2 pharmacies in a predominantly Latino community in South Texas	During, Post
Borkowski N	2013	Mixed methods (case study combined with quantitative analyses of membership and cost data)	Care processes (clinic-wide efficiency improvements)	Community clinics in Jackson Health System in Miami-Dade County, Florida	Pre, Post
Braun PA	2013	Mixed methods (n = 13 qualitative interviews with staff/leaders combined with parent surveys)	Co-location (dental hygienist based in primary care practice)	Community clinics serving disadvantaged children in Colorado	Post
Breslau ES	2010; 2015	Qualitative (n = 24 qualitative interviews with leaders)	Community engagement, care processes (state and community partnerships for increased screening for cervical and breast cancer)	Clinic and community clinic partnerships in underserved counties in Southern rural states with high breast and cervical cancer mortality	Pre, During, Post
Clark NM	2014	Mixed methods (Document review combined with n=60 qualitative interviews with leaders)	Care processes (enhanced diabetes care and coordination)	5 regional sites (Camden, NJ; Chicago, IL; Dallas, TX; Memphis, TN; Wind River, WY) with partnerships between community clinics and community organizations (applied for grant support)	Pre, During, Post
Dennehy P	2011	Mixed methods (case study combined with quantitative provider/staff survey data)	Capacity building, technology (EHR installation and optimization)	2 nurse-managed health centers serving low-income primary care patients (Glide Health Services in California and Campus Health Center Detroit in Michigan)	Pre, During, Post
Drainoni ML	2016	Qualitative (n = 50 participants, 7 focus groups, 6 individual interviews)	Care processes (improvement in naloxone prescribing implementation in emergency department)	Emergency department staff at Boston University Medical Center	Post
Garg SK	2016	Qualitative (n = 8 in-depth interviews of leaders/staff across sites)	Care processes, technology (clinic-wide patient text messaging programs)	8 California community clinics or integrated safety net health care systems (applied for grant support)	Pre, During, Post
Johnson TL	2015	Mixed methods (case study combined with quantitative analyses of the high-utilizing patient population)	Care processes, technology (patient risk stratification for new care models)	Urban integrated safety net health care system in Denver, Colorado	Pre, During, Post
Labbok MH	2013	Mixed methods (quantitative staff/provider survey assessments combined with qualitative interviews)	Care processes (hospital-based breastfeeding support program)	6 hospitals serving low-income women in North Carolina	During, Post
McMullen CK	2013	Qualitative (n = 9 in-depth interviews with leadership and clinic observations)	Capacity building, care processes (patient-centered medical home implementation)	6 community clinics in Portland, Oregon	Pre, During
Novick G	2015	Qualitative (n = 24 qualitative interviews with providers, staff, and leaders)	Care processes (group prenatal care visits)	6 urban community clinics or hospitals serving low-income, minority women	During
Ramos-Gomez FJ	2014	Mixed methods (case study combined with data measurement of program uptake)	Co-location, community engagement (oral care program for children ages 0–5)	Community clinics and community sites such as WIC and Head Start	During
Samaan ZM	2016	Quantitative (clinic- and provider-level data measurement)	Care processes (bundled infant preventative services delivery)	Cincinnati Children’s Hospital Medical Center pediatric clinic sites (community and hospital)	Pre, Post
Smith MG	2017	Qualitative (n = 73 interviews with various pharmacy staff members)	Care processes, community engagement (use of clinic- and community-based pharmacies to manage medication review for high-risk patients)	Network consisted of 123 community and health center pharmacies in North Carolina	Post
Steiner BD	2008	Mixed methods (case study combined with quantitative cost estimates)	Capacity building, care processes, community engagement (establishment of community health primary care network)	Community Care of North Carolina (including hospitals/clinics, social service agencies, and public health departments serving mostly rural Medicaid recipients)	During, Post
Sugarman JR.	2014	Mixed methods (observations of implementation processes at each site combined with surveys of providers/staff); n = 65 sites	Capacity building, care processes (patient-centered medical home implementation)	65 clinics including homeless clinics, private practices, residency training centers, and other safety net practices	During, Post
Wagner EH	2014	Qualitative (n = 7 qualitative interviews with leaders)	Capacity building, care processes (patient-centered medical home implementation)	3 diverse community clinic locations (urban site in Oregon, rural site Colorado, multisite urban and rural in Idaho)	Post
Waitzkin H	2011	Mixed methods (quantitative intake surveys completed with patients, combined with n = 35 qualitative interviews with patients, providers, and staff, as well as observations)	Care processes (use of lay health workers to facilitate depression care)	2 community clinics in New Mexico	During, Post

Abbreviations: EHR, electronic health record; WIC, Special Supplemental Nutrition Program for Women, Infants, and Children.

^aThe 20 active implementation phase articles include during- or post-implementation data, 2 of which are the same author describing the same intervention and are therefore presented jointly.

The vast majority of studies were implemented within community clinics or integrated safety net health care systems (16 articles), while the other 4 articles focused on hospital care. A total of 6 articles also discussed explicit linkages between the health care setting and other organizations, such as community-based organizations and pharmacies.

Fifteen active implementation articles focused on care process improvements covering a range of topics, such as patient-centered medical homes, group prenatal visits, bundled infant preventive care, naloxone prescribing, or diabetes care delivery. Five articles described capacity building techniques to foster internal support and skill building to implement the new program, 4 articles reported community engagement strategies to partner with external organizations, and 2 articles addressed implementing co-location of care (such as dental care into existing primary care). Five of the articles also examined technology implementation, and 2 focused on patient education.

In all, 6 external^{28–30,32,36,40,43} and 13 internal programs were studied,^{31,33–35,37–39,41,42,44–47} and the specific implementation outcomes assessed are summarized in Table 2.

Table 2.

Implementation Outcome Assessment Among Active Implementation Studies.

Author	Year	Evidence-Based Practice?	Implementation Outcomes of Interest
			Acceptability	Appropriateness	Fidelity	Implementation Cost	Sustainment
External studies
Bolin JN	2013	No	Clinics and pharmacies did not have existing staff person dedicated to monitor kiosks	Leadership lightly involved in MOU process and evaluation interviews	Standardized—everyone used same kiosk	Resource/$ limitations to pay for Wi-Fi, etc	Potential continuation for some sites
Braun PA	2013	No	Challenge to keep regular hygienists in this new position	Leadership involved in implementation and evaluation	Not mentioned	Resource/$ limitations in getting reimbursed for dental care	Ongoing for 5+ years
Breslau ES	2010; 2015	Yes	Understaffed—felt that this new initiative was extra work on top of other responsibilities	Engagement of community partners was challenging; leadership was a clear barrier for community organizations/clinics feeling invested in projects	Option to customize—each local partnership chose their own evidence-based practice	Resource/$ limitations for clinics to provide cancer care to more patients in need	Sustained in only a few cases; success stemmed from relationship building
Labbok MH	2013	Yes	Challenge in relying solely on lactation consultants to do all the work and lack of self-efficacy among other staff members	Leadership buy-in present, but not aligned with existing health system culture	Standardized 10-step breastfeeding program	Expense of baby-friendly designation as a barrier	Not mentioned
Novick G	2015	Yes	Understaffed to handle the more challenging group scheduling process	Some sites aligned with organizational culture (especially with a great staff champion) while others faced major cultural barriers and apprehension about the program	Standardized group prenatal visit program among intervention sites	Resource/$ limitations, mainly major organizational financial shortfalls	Sustained at only half of sites able to address implementation barriers
Waitzkin H	2011	Yes	Understaffed in times of staff turnover	Structural bureaucracy/leadership limits in terms of professional roles of promotoras vs other staff; Some leadership buy-in	Standard approach to promotora work with patients identified with depression	Not mentioned	Not mentioned
Internal studies
Borkowski N	2013	No	Not mentioned	Innovation aligned with the shared health system culture of the coalition; leaders very engaged in establishing a coalition with change management strategies	Option to customize—each primary care site determined their own redesign projects	Not mentioned (beyond context for the program)	Sustainment bolstered by monthly data reports and scheduled meetings over first 2 years of program deployment
Clark NM	2014	No	Not mentioned	Aligned with organizational culture—greatest success when diabetes programs aligned with other programmatic goals and with substantial leadership buy-in	Option to customize specific diabetes programmatic activities	Not mentioned	Striving toward sustainment with necessary policy changes
Dennehy P	2011	No	Lack of existing sufficient computer skills themselves	Aligned with health system culture—huge emphasis on the partnership/leadership model to ensure long-term success	Option to customize their EHR implementation plans	Resource/$ limitations—one institution removed IT funds shortly after program got under way	Sustained with another grant as a part of an ongoing community-based research network
Drainoni ML	2016	Yes	Strong acceptability by all staff	Strong associated appropriateness of intervention at leadership level	Standardized—original policy deployed	Not mentioned	Not mentioned
Garg SK	2016	No	Lack of existing staff IT expertise	Leadership and staff buy-in for the overall goals of the texting program; structural bureaucracy/leadership limits, particularly around HIPAA regulations	Option to customize, each clinic chose their own texting use case	Not mentioned (beyond grant funding)	Not mentioned
Johnson TL	2015	No	Appropriate staffing/tech support to build the new algorithm to identify high-utilizer patients	Innovation aligned/meeting organizational needs across multidisciplinary teams; leadership buy-in to create the program	Not mentioned	Not mentioned (beyond grant funding)	Sustained use of algorithm system-wide
McMullen CK*	2013	No	Not mentioned	Innovation aligned with health system culture; leadership buy-in, especially in the inspiration/motivational phase of the work	Option to customize the specific primary care redesign process	Provided modest financial incentives	Sustained due to “incubator” experience shared by organizational leaders
Ramos-Gomez FJ	2014	No	Not mentioned	Program worked because dental students were required to do rotations in this community-based dental program, and the community partners already had relationships with underserved patient populations to refer	Standardized trainings and use of standardized forms	Limited funds for program, but free labor because dental students provided the care as a part of training	Sustained by integrating program into covered prenatal care bundle
Samaan ZM*	2016	Yes	Appropriate staffing/tech support, especially for the QI consultant to manage the day-to-day work and paid protected time for some staff to participate	Innovation aligned with health system culture (multidisciplinary team represented all relevant stakeholders); leadership buy-in on steering committee	Customized over time and then became a standard process	Internal funds provided to launch and sustain program	Sustained by developing standardized orientation and training process
Smith MG	2017	No	Not mentioned	Appropriate local primary care providers and pharmacy leaders collaborated to build partnership	Standardized protocol requested from all pharmacies and care teams	Not mentioned	40% had difficulty maintaining consistency of medication reviews month to month
Steiner BD*	2008	No	Early physician buy-in; case managers and physicians need more time to meet together	Innovation aligned with health system cultures, largely because of the local control of the work; leadership buy-in	Option to customize as each site focuses on their own specific QI projects	Existing payment models not sustaining work for highly complex patients, but Medicaid funding key to overall program coordination	Sustained through ongoing support by state office of rural health
Sugarman JR	2014	Yes	Understaffed, particularly with high staff and provider turnover and low staff morale in some instances	Innovation aligned with health system culture; leadership buy-in	Standardized the 8 change concepts for all clinics to work on	Barrier to getting full insurance reimbursement for a new model of care	Sustained but facing ongoing financial constraints
Wagner EH	2014	Yes	Not mentioned	Innovation aligned with health system culture; leadership buy-in as central to mission of organization	Standardized the 8 change concepts for all clinics to work on	Not mentioned	Sustained through established trust in the improvement process

Abbreviations: EHR, electronic health record; HIPAA, Health Insurance Portability and Accountability Act; IT, information technology; MOU, memorandum of understanding; QI, quality improvement.

^*These studies provided key examples of note, especially in combining various implementation methods.

Implementation Outcomes.

1. Acceptability was conceptualized as the ability of staff to take on implementation related activities in their daily work. This is particularly relevant in safety net health care settings that are often resource limited with respect to staffing. Among the 19 studies included in this review, 13 discussed acceptability, 9 of which classified it as an inhibitor or barrier to widespread success.

The most common reason for lack of acceptability of the intervention within the health care system was understaffing and high rates of staff turnover. For example, Bolin et al³⁰ mentioned that the new educational kiosk was not able to be monitored by existing staff members in the midst of their busy days, and therefore might go offline without anyone realizing the problem. In fact, it was common throughout the studies that staff and providers were expected to take on the new intervention/program in addition to their existing work responsibilities, which set up an inherent challenge to engagement by frontline workers. Another aspect of reduced acceptability was the need for staff to have specific technical assistance or skills to implement the program. For example, in the study by Garg et al⁴¹ on implementing texting systems to communicate with patients, one of the barriers was a lack of existing staff technological expertise to manage vendors or troubleshoot glitches.

Of note, there were 4 examples of high acceptability among providers and staff. In the study of implementing bundled infant preventive services by Samaan et al,⁴⁷ the authors described how hiring a dedicated quality improvement consultant, coupled with paid protected time for frontline staff to work on the design of the program, led to high engagement. In addition, with the implementation of improved naloxone prescribing procedures for patients on opioids,⁴⁶ very high acceptability was noted because of staff and leadership alignment with the public health mission.

All of the external programs mentioned acceptability as a barrier, while only 3 of 13 internal programs did. The other internally developed programs either did not mention acceptability explicitly (6 studies) or discussed it as a strength of implementation (4 studies).

2. The study team assessed appropriateness as the fit, suitability/relevance, and practicality of the intervention/program based on the existing work and culture of the organization. Sixteen studies discussed leadership as a positive element of their implementation process. The level of leadership engagement varied somewhat, with some studies such as Waitzkin et al⁴⁰ evaluating leadership perspectives through interviews, while other studies such as Dennehy et al³⁴ described a deep collaborative/partnership process with leaders in the health care system. Four of 6 externally developed programs had leadership buy-in, compared to 12 of 13 internally developed programs.

The team also evaluated appropriateness as alignment of the intervention/program with the culture or mission of the health care system. A total of 16 studies discussed alignment: 11 described it as contributing to the success of their program, while another 5 described inadequate alignment. Alignment was critical for the new program to be seen as adding value to existing priorities and workflows, given that there were a number of new projects at any given time. Borkowski et al³¹ describe a process in which the entire health care system coalition agreed on a shared vision for primary care, and the specific implementation of efficiency improvements tied directly to that shared vision. Alternatively, Breslau et al^28,29 describe community organization integration into the implementation process too late to give input on the specific project goals. There also were more structural types of barriers that made alignment more challenging, such as Waitzkin et al⁴⁰ documenting the lack of formalized job responsibilities for lay health workers that made workflows challenging.

Of note, all 6 of the externally developed programs either did not discuss appropriateness of the program or reported challenges with this process. This was in contrast to 10 of the 13 internally developed studies in which organizational alignment was mentioned as a strength.

3. Next, the study team examined the fidelity of the program implementation, or the degree to which implementation was tied to an external standard (such as a detailed protocol for program components) versus allowed to be customized by the setting. A total of 9 studies mentioned standardization, while 8 studies explicitly mentioned the ability of local sites to customize or choose the program elements they would implement. Types of standardization included a network of safety net clinics in Wagner et al⁴⁵ that all implemented the same change concepts to work toward primary care redesign. This coordinated approach can be juxtaposed against the description presented in Steiner et al,³⁸ in which each rural primary care site focused on its own quality improvement projects supported by a centralized state administrative office.

Of the externally developed programs, 4 of 6 studies described standardized implementation, while 7 of the 13 internally developed programs were customizable and 5 were standardized (one study did not mention fidelity).

4. With respect to implementation cost, the team assessed whether financial resources were considered during program rollout. A majority of studies mentioned finances related to implementation (11 studies in total, 8 of which described financial barriers and 3 that described appropriate internal resource allocation). The financial challenges were primarily medical care reimbursement challenges (eg, Sugarman et al³⁹). Other examples of unmet financial needs included lack of funds for supplies/services (such as Wi-Fi access in Bolin et al³⁰) or certification processes (such as paying to become designated as a “baby friendly” hospital in Labbok et al³⁶).

Five of the 6 external programs mentioned financial barriers to implementation. This was in contrast to 3 of 13 internally developed programs that mentioned successful financial planning, another 3 that described financial barriers, and 7 that did not mention implementation costs.

5. Finally, it was noted whether each article discussed the sustainment plans for the program. Overall, 14 articles mentioned sustainment. Assessment of sustainment was somewhat less strict than the other implementation outcomes because in most of the articles not enough time had elapsed since program launch to definitively comment about sustainment. However, it was noted if they mentioned concrete plans or first steps toward maintaining the program. In the case of Braun et al,³² the authors had 2 years of evaluation data and more than 5 total years of implementation experience co-locating dental hygienists into community clinics. Similarly, Dennehy et al³⁴ described their ability to obtain additional grant funding to maintain a community-based practice research network to pull data from electronic health records that were implemented during the initial project. In contrast, the diabetes improvement efforts described in Clark et al³³ mentioned plans for spread without evidence of sustainment.

Three of 6 externally developed programs described continuation of the intervention at limited sites and only 1 article mentioned widespread maintenance. Of the internal articles, 9 of 13 studies described sustainment positively, with examples such as shared vision and trust for continuing the program, plans for ongoing staff training and engagement, and intermediate billing mechanisms to ensure reimbursement.

Articles of Interest.

Several included studies provided key examples of note, especially in combining various implementation methods, highlighted with an asterisk in Table 2.

Planning Articles

Of the 7 planning articles (Table 3), 4 described the collection of qualitative pre-implementation data to plan an implementation,^48–51 while the other 3 were protocols.^52–54

Table 3.

Planning Studies^a.

Author	Year	Methods	Intervention/Program	Clinical Setting	Implementation Stage	Internal vs External
Angier H	2015	Mixed methods (quantitatively assess health insurance coverage continuity and utilization of health care services, assess implementation process with staff)	Care processes; technology (identify and outreach to uninsured patients)	4 community clinics in Oregon	Protocol	Internal
Coker TR	2014	Mixed methods (expert panel ranking of models, and focus groups of WCC stakeholders)	Community engagement, care processes (redesign structure of pediatric care)	Multisite community clinic system and 2 independent practices serving Medicaid-insured children in Los Angeles area	Pre	External
Cole AM	2015	Qualitative (n = 13 interviews to assess facilitators and barriers to implementing program)	Care processes (mail-based colorectal cancer screening)	7 community clinics in Washington, Wyoming, Alaska, Montana, and Idaho practice-based research network	Pre	External
Gold R	2015	Mixed methods (baseline surveys of study clinic characteristics, intervention uptake rates, document review, and qualitative observations/interviews)	Care processes (diabetes quality improvement intervention)	29 community clinics through OCHIN, Inc. national network	Protocol	External
Nguyen OK	2015	Qualitative (n = 50 interviews with health care and social service providers to determine feasibility)	Community engagement; care processes; technology (electronic data exchange between health care systems and social service departments)	Urban integrated safety net system in Dallas County, Texas	Pre	Internal
Patterson ME	2016	Qualitative (n = 6 providers in semistructured interviews)	Care processes; technology (develop an EHR tool to categorize high-risk heart failure patients)	Urban safety net hospital in Kansas City	Pre	Internal
Quanbeck AR	2014	Mixed methods (interrupted time-series analysis of treatment attendance, staff interviews to assess adaptations to implementation protocol)	Care processes; technology (electronic communication for substance abuse treatment)	3 community clinics (one affiliated with University of Wisconsin, one independent rural site, and one urban site)	Protocol	External

Abbreviations: EHR, electronic health record; WCC, well-child care.

^aThe 7 planning articles encompassed research that had (1) a protocol focus or (2) pre-implementation data collection to inform a future program or intervention.

Four planning-phase articles were conducted within community health centers, and all of the studies described programs that were focused on care process in terms of implementing new communication modalities (eg, health information exchanges), care management programs (eg, risk stratification for heart failure patients), and patient tracking systems (eg, insurance coverage tracking). The implementation outcomes among these planning articles were not examined, as they did not include data from the actual implementation phase.

Discussion

This review aimed to directly compare studies by evaluating implementation outcomes in a systematic way. This study found that internally developed programs within safety net sites were more likely to be aligned with overall organizational priorities, and demonstrated higher rates of acceptability and local customization by staff and providers. Moreover, internally developed interventions included more up-front planning efforts for system-wide integration and sustainment. However, the findings also highlight common challenges faced by safety net health care organizations, including diffusing successful programs across systems, and working with external collaborators/networks.

For internally developed innovations, lack of standardization across sites may impede the innovative practices from functioning as intended. Further shortcomings common among internally developed programs are understaffing and high staff turnover. This near-universal barrier must be addressed in order for safety net health care systems to be able to innovate in meaningful and sustainable ways. Finally, a focus on generating innovative practices internally may result in “self-siloing,” a failure to look outward for innovations, resulting in delayed uptake of novel evidence-based practices that have been developed in other health care settings. This may contribute to the slow translation of evidence into practice.⁵⁵ Ideally, safety net health systems would regularly survey health system evidence that is relevant to their patient populations to identify innovative practices to implement and would have structures in place to judge the benefit of, and incorporate, these innovations.

Limitations

Despite its strengths, this study does have some limitations. Relevant studies may have been overlooked in the search process, despite adherence to guidelines for systematic reviews.⁵⁶ Because interventions were included based on the setting—the US ambulatory safety net health system—rather than a specific clinical area, outcomes could not be combined across studies. The study team believes that there are shared lessons in implementing new practices across a variety of clinical problems across safety net delivery systems, and therefore chose to structure the review to encompass diverse interventions.

Conclusions

The findings suggest that there are unrealized opportunities for sharing innovations across like-minded institutions and for bidirectional engagement between external stakeholders and frontline staff to enhance acceptability and permit internal customization. Because external stakeholders often bring expertise and resources, enhancing collaboration between them and safety net health centers has the potential to catalyze innovation. Practice-based research networks serve as a model for this type of engagement.⁵⁶

Implications.

Future work on safety net health system innovation should examine how programs and novel interventions are “made” or “remade” internally in order to benefit from the advantages of internally developed programs, while maintaining some standard features that would make their outcomes comparable across sites/settings. This may require a closer alignment with teams that incorporate implementation science as well as practice-based research methods.^52,57 Both external stakeholders and safety net leadership will need to change the way they partner on innovation. Key strategies that fostered internal innovation in this review—such as emphasis on local ownership, scope for customization, and planning for sustainment—could be combined with continued emphasis on standard benchmarks for both key processes and outcomes.

Moving forward, additional evidence will be needed to determine if safety net organizations are able to sustain long-term health improvements from their implementation, as well as how implementation research can be used to optimize how innovative practices are incorporated into clinical settings. Systematic discussion of acceptability, appropriateness, fidelity, cost, and diffusion/sustainment would advance the study of health system improvement and permit more rapid spread and scale of innovations.