Linking Sustainability Research to Intervention Types

Abstract

Researchers, funders, and managers of health programs and interventions have become concerned about their long-term sustainability. However, most research about sustainability has not considered the nature of the program to be sustained. Health-related interventions may differ in their likelihood of sustainability and in the factors likely to influence continuation. I suggest a framework for analyzing the sustainability of 6 types of interventions: (1) those implemented by individual providers; (2) programs requiring coordination among multiple staff; (3) new policies, procedures, or technologies; (4) capacity or infrastructure building; (5) community partnerships or collaborations; and (6) broad-scale system change. Hypotheses for future research and strategies that program managers might use to achieve sustainability also differ by program or intervention type.

Researchers and funders have expressed great interest recently in learning more about program sustainability. Do programs and other interventions that receive short-term funding continue after that investment ends? Can researchers show evidence for strategies that enhance the sustainability of health and human service programs? Questions about sustainability often arise for programs initially funded from an external government or foundation grant or contract. Since 2007, the National Institutes of Health (NIH) has fostered these interests by sponsoring annual conferences about research on dissemination and implementation, which include sessions on studying sustainability, especially of interventions developed and tested with NIH funding.¹

No agreed-upon definition exists for the term “health program sustainability.” Most sources agree that it refers to the continuation of activities or benefits for target recipients after an initial period of funding ends or following the initial implementation of a new program or procedure. According to one definition, “Sustainability is the continued use of program components and activities for the continued achievement of desirable program and population outcomes.”^2(p2060) Terms such as “institutionalization” or “routinization” refer to embedding a new set of activities into the normal operations of an organization. Other authors have used the terms “continuation,” “maintenance,” and “durability” or their opposites, such as “discontinuation.”

Sometimes sustainability is included as a subconcept under the broader concept of implementation,³ but that framework does not emphasize the longitudinal perspective needed for sustainability research. Furthermore, factors that influence successful initial implementation are not necessarily the same factors that enable continued implementation into longer-term sustainability. A more detailed discussion of concepts affecting research on sustainability, as illustrated by 6 different definitions of outcomes or dependent variables for sustainability research, is available elsewhere.²

Here I focus on sustainability of programs or interventions that have already been implemented in the target locations. A program is defined as a specified set of activities intended to address one or more identified outcomes, especially in the human services arena.⁴ The focus of the sustainability study may also be termed an intervention, especially when it involves intervention into some aspect of the lives of recipients, to address their needs and enhance opportunities for productive lives and healthy behaviors. These programs and interventions take place in both clinical and community settings, although some are more likely to occur in a clinical health care setting, such as interventions delivered by individual providers; other programs more often occur in a community setting, such as community partnerships or broad-scale system change.

In contrast to my usage here, sometimes the term “program” refers simply to a funding stream, such as a line item designating a program in the federal budget that allocates money for a broad purpose. Those programs may encompass many different types of activities in different locations, rather than a specified set of activities or components. Use of “program” to signify a diverse set of local activities makes it more difficult to assess local-level sustainability when no well-defined entity serves as the focus for research. In addition, when broad-scale systemic change is intended, the intervention strategy might include multiple programs focused on diverse forces interacting in an environment.^5–7

Some researchers define sustainability as the processes that lead to continuation of program activities after initial funding ends.^8,9 They appropriately suggest that later continuation of program components is influenced by processes that take place during program adoption decision-making and initial implementation. However, defining sustainability as this set of longer-term processes is not conducive to cumulative research measuring both sustained outcomes and the influences that lead to those outcomes.

Several streams of research and research methodology have expanded the topic of sustainability in recent years:

• Systematic reviews of empirical research results about sustainability^10–13;

• Development of frameworks and conceptual models about factors likely to affect sustainability^10,13–16 (e.g., Schell et al., Center for Tobacco Policy Research, engaged experts in conceptual mapping to identify 8 factors believed to foster sustainability capacity¹⁷);

• Discussion of methods needed to strengthen empirically based research about sustainability^2,13,18;

• Focus on sustainability as a set of processes likely to enhance the durability of programs and projects^8,9;

• Advice to funders and practitioners about the importance of planning early for sustainability and suggested strategies to enhance its likelihood^15,19;

• Development of related concepts, especially institutionalization and routinization^11,20–23;

• Numerous studies about the sustainability of specific programs or projects (reviewed by Scheirer¹¹ and by Stirman et al.¹³);

• Discussions of why sustainability is important and the need for more research about sustainability.^12,18,24

These streams overlap within and across specific articles; many sources contribute to several of these topics.

To date, writers about health program sustainability have not focused on the nature of the program, project, or intervention to be sustained. Precisely what is being evaluated in studies of the extent of and processes underlying sustainability? Are factors influencing sustainability likely to differ for different types of interventions? For research about sustainability to contribute evidence-based propositions about how to sustain effective new programs, should the research differentiate among different types of interventions? Perhaps one size does not fit all for this research topic. To date, sustainability research and evaluation have treated the focus intervention as a black box whose contents are not expected to affect the research conclusions or the strategies needed to enhance sustainability. A similar weakness hampered evaluative studies of program impacts in the early days of evaluation research.^25,26 The subsequent widespread use of theory of change or logic models, along with measures of program implementation, now provide details about expected program components in most efficacy and effectiveness studies.

I propose that sustainability research and advice to program managers should differ by type of program or intervention. The framework describes 6 types of interventions or programs distinguished by their structure: what they are and how they are implemented. They can be expected to differ in the processes and resources required to sustain them after initial funding ends. Hypotheses differ about key factors likely to foster the sustainability of each type of program or intervention; similarly, advice for program managers who seek to sustain their implemented programs should be tailored to the type of program. Advances in knowledge and practice about sustainability are more likely if researchers carefully define the nature and requirements of the entity to be sustained and then assess hypotheses appropriate to that type of intervention.

The terms “program,” “project,” “intervention,” and “strategy” are often used interchangeably to refer to the focus of the sustainability research or practical advice. In some initiatives, a typical project may include several types of interventions, such as a community coalition that is both working toward school food policy changes and advocating for interventions by primary care providers to reduce childhood obesity. For these multicomponent projects, the likelihood of sustainability, and the factors affecting it, may differ considerably among those multiple components of a single project or site. Therefore, researchers and practitioners should plan appropriate data collection approaches and sustainability strategies for the diverse components of their projects.

Legitimate questions may arise about the origin of this typology. It is not directly derived from previous theory, but rather developed from my reflections about recent analyses and advice to practitioners about health program sustainability. For example, the very helpful Program Sustainability Assessment Tool developed by Luke and colleagues contains rating scales for 8 factors believed to influence sustainability capacity, derived from the literature and a concept-mapping exercise.¹⁴ (I served on an advisory committee aiding its development.) Yet no attention is given in that tool or its explanatory material to the possibility that these factors may relate differentially to different types of programs. For example, is the factor labeled political support equally important for individual physicians to continue prescribing a new medication and for the continuing work of community coalitions? A major new review of sustainability studies distinguishes among medical, mental health, and public health interventions but does not further characterize the structures of the interventions in the studies reviewed.¹³

Previous theories and conceptual work about research translation processes have not developed specific typologies about program sustainability. For example, the new synthesis book, Dissemination and Implementation Research in Health: Translating Science to Practice,²⁷ provides no conceptual framework of intervention types, even for furthering research about initial implementation, which might have different predictors than does sustainability. Existing typologies focus most often on how specific program components might bring about behavioral change in target recipients, rather than illuminating diverse ways of sustaining interventions.^28–30 The frequently used diffusion of innovations framework includes characteristics of the intervention, such as complexity, relative advantage, and adaptability, but these are characteristics as perceived by specific stakeholders, not conceptualized as inherent structures of the intervention being considered.³¹

The intervention typology has roots in congruence and open-system theories of organizational behavior, which emphasize the importance of fit among organizational and environmental components when assessing organizational change,^32–34 but no theory has applied these frameworks to sustaining health-related interventions. This typology is also related to the 6 types of sustainability outcomes or dependent variables suggested recently,² but does not imply a one-to-one correspondence between intervention type and intended outcomes.

Space does not permit discussion of the full range of processes in a comprehensive approach to the dissemination and implementation of evidence-based practices to a large number of sites or individuals.^3,27,35 The broader topic of research-to-practice translation, or spread, of interventions requires attention to at least the following subcomponent processes after an intervention has been carefully developed and tested for efficacy: (1) dissemination of information about the intervention to potential adopters, who may be individuals or organizations working in either a clinical or a community context (or diffusion occurring by other methods, without an identified dissemination or marketing effort); (2) decision processes by potential adopters to start using the program or intervention; (3) implementation processes for adopters and adopter organizations to learn to use the intervention and to make other complementary changes, for example, in organizational processes needed to implement or adapt it appropriately; and (4) sustainability, the continued use of that intervention after an initial implementation period. Here I focus on the fourth process: how to characterize, research, and enhance the sustainability of beneficial interventions after initial implementation or short-term funding ends. Sustainability is likely to be influenced by all the earlier processes, but in different ways for different types of interventions or programs.

I also do not consider fully here the environmental setting or context of the focus program, which is likely to have major influences on all these processes. For example, Kegler et al. describe several contextual factors that influenced the planning and implementation phases of community-based health promotion coalitions in California (they did not study sustainability), including history of collaboration, community demographics, economic conditions, geography, and community values and norms.³⁶ Other researchers found that support for implementing evidence-based practices differed between private and public mental health agencies.³⁷

Different intervention types often occur in different organizational contexts. For example, interventions implemented by individual providers might be found primarily in clinical settings, although these settings could range in scope from single-provider primary physician offices to much larger health care organizations. Future analyses might usefully identify and characterize different types of environments, to enable contingency research models for sustainability that consider both type of intervention and the favorability of environmental influences.³⁸

TYPES OF INTERVENTIONS

The framework differentiates 6 types of interventions primarily according to their structure—who and what are required for continued delivery—rather than to their health content area. Each type might include programs that focus on diverse content areas, such as cancer education, prevention of smoking, or other educational or community services. When most research efforts about health program dissemination into practice are organized within health content areas, it is difficult for research about process topics such as sustainability to accumulate findings across content areas.

Interventions Implemented by Individual Providers

Some innovations are for procedures or practices that can be prescribed or implemented by individual providers, essentially working alone.^18,39 Examples abound in the classical diffusion of innovations literature,³¹ such as primary care providers prescribing a new medication, substance abuse counselors using a new therapeutic technique, classroom teachers using a new curriculum, or farmers adopting a new type of seed. In the health field, this type is often delivered in clinical settings. Disseminating such innovations and ensuring their initial correct implementation by those individual providers may be complex and problematic. But after the individuals learn how to use them and are motivated to continue using them, continuing sustainability may be less complex than for other intervention types. This may be a reason that traditional diffusion of innovations research has not placed much emphasis on the sustainability of innovations after their initial diffusion and implementation.

Hypotheses for the sustainability of this type of intervention include

• Interventions implemented by individual providers are likely to have high rates of sustainability compared with other types of interventions, if they have been implemented appropriately before sustainability is assessed. However, a recent literature review found that in the few studies that employed independent fidelity ratings to assess sustainability at the provider level, fewer than half the providers sampled continued the practice or intervention at high fidelity.¹³

• The sustainability of interventions implemented by individual providers is likely to be strongly influenced by whether payment for the individual’s delivery is included within normal streams of financial support, such as fee-for-service medicine or the normal classroom of a public school teacher. If no continuing payment (or profit) is available for those individuals to continue providing the intervention, then the likelihood of sustainability will be influenced by the same factors that affect interventions implemented by organizations.

• The sustainability of interventions implemented by individual providers is likely to be strongly influenced by their motivation to continue the new practice.

When managing interventions implemented by individuals, program administrators may require less external support for maintaining them than for programs implemented by organizations, after initial training and feedback about appropriate use. Enhancing the motivation of the providers may promote sustainability, by continuing positive feedback about their successful use or continued networking among providers about the feasibility and benefits of continued use.

Interventions Requiring Coordination Among Multiple Staff

Many innovative health promotion programs are implemented by organizations or in community settings and require the coordinated work of several different types of staff. For example, a community project to provide diabetes education and screening in African American churches might require not only health educators who work directly with the church members, but also procedures for assessing the screening tests, for feedback of test results to members, and for coordinating with church administrators, as well as leadership and support for the project by the implementing agency.

Many innovations that take place within a single organization involve coordinated efforts among several different kinds of staff, such as nurses, health educators, and outreach workers; therefore, the organization’s leadership and administrative support become key factors in both implementation and later sustainability.^40,41 Furthermore, full implementation may involve training for several different types of staff and ensuring that the new work fits within their job roles (rather than conflicting with other duties or appearing to contradict their professional or personal norms). To identify this type of intervention, researchers should inquire how the intervention is implemented and whose work is involved. Previous research about sustainability often addressed this type of program, when it became clear that continued implementation after initial funding was not automatic or easily achieved.¹¹

Hypotheses about the sustainability of this type of intervention include

• Sustaining interventions requiring coordination among multiple staff members is likely to be strongly influenced by factors within the organizational context, such as administrative support, the work of project champions, the intervention’s congruence with the implementing organization’s underlying mission and culture, and its fit with other organizational procedures and programs.

• Sustaining interventions requiring coordination among multiple staff members is likely to be strongly influenced by the availability of continued financial resources for supporting the staff members and administrators involved. If initial project funding covered the salaries of key implementing staff, then new financial resources are likely to be needed for continued support of those staff members, either from the organization’s normal budget sources (institutionalization) or by securing new external resources for financial support, such as grants or donor contributions.

• Sustaining interventions requiring coordination among multiple staff members is likely to be enhanced by externally provided training and technical assistance to organizational leaders for the organizational processes and planning required.

For program management, funders and leaders of an externally funded program that involves multiple staff roles should start planning early for sustainability—at least a year before the initial funding expires. What resources would be needed to continue staff salaries currently supported by the external grant? Can these costs be transferred to the organization’s normal ongoing sources of support? Could the program’s services be supported by a fee-for-service arrangement? What external sources might provide new funding? Can partnerships or networking in the community uncover ways to gain new financial support or volunteer effort? Furthermore, strong internal administrative support is likely to be required to continue coordinating the work of multiple types of individuals, especially if there is turnover among key implementing staff.

New Policies, Procedures, and Technologies

Sometimes an initiative focuses on securing changes in specified policies or standard procedures or on implementing a new technology. For these interventions, securing the policy change or managing the initial implementation is likely to be complex, but sustainability may be less problematic. For example, an advocacy effort might aim toward establishing no-smoking policies in restaurants in the community, or a hospital might change its standard operating room procedures to avoid new infections. New technologies are continually coming into use in health systems, including imaging systems, information technology, and so forth. For these changes, major efforts may be devoted to obtaining initial decisions to change the policy or procedure or to purchase the technology,⁴² and substantial efforts may be needed to ensure full initial implementation or enforcement. However, maintenance following full implementation may be less problematic, if additional resources are not required.⁴³

Hypotheses about the sustainability of this type of intervention include

• Interventions for new policies, procedures, and technologies are likely to have high rates of sustainability, once the change in policy, procedure, or technology is in place and fully implemented.

• The sustained use of new policies and procedures may be strongly influenced by continued efforts to monitor and enforce the intended new policy. If no one pays attention to the continued implementation of the new policy or procedure, its use may gradually decline.⁴⁴

• After some new technologies are in place and fully implemented, it may be impossible to revert to the previous system; therefore, at least some continued use is likely. For example, after implementing an electronic health records system, individual staff members could not revert to paper records. However, inadequate implementation or lack of technical support may hamper the effectiveness of that new technology.

When managing an initiative focusing on changing an ongoing policy or procedure, or on installing a new technology, program administrators need to go beyond the initial decisions for change, to ensure that the policy or procedural change is fully implemented. This may require training for all workers involved, analysis of the work flow to secure compatibility among diverse procedures, and ensuring that supervisors and all types of workers are on board with the changes. But after the changes are fully implemented, they are likely to be institutionalized into the organization’s standard operating routines, and may need little continuing effort for sustainability. Continued monitoring of appropriate implementation and assessing the fit of the focal policy, procedure, or technology within the context of additional new policies and procedures are also likely to enhance continued sustainability of these types of change.

Capacity or Infrastructure Building

Some funders of health-related programs focus on improving the capacity or infrastructure of their target recipients or other agencies, for example, by leadership training, training for board development or strategic planning, or helping agencies to develop new information systems. For instance, the Division of Oral Health of the US Centers for Disease Control and Prevention has had a program for several years to enhance the capacities of state governments’ oral health agencies.⁴⁵ Similarly, foundations may have initiatives for capacity development among grantee agencies or potential grantees, with or without funding for specific program activities.⁴⁶ The sustainability question becomes, What happens to the enhanced state-level or agency capacities after the federal or foundation funding ends? Do the agencies continue to use the new skills acquired during the capacity-building period?

Hypotheses about the sustainability of this type of intervention include

• Sustaining capacity or infrastructure building efforts depends strongly on the continued presence of those trained during capacity building. If turnover of key individuals is high, then the new capacities may be lost, unless procedures are also in place for transferring the new skills to new leadership or staff members.

• Sustaining capacity- or infrastructure-building efforts does not depend as heavily on new sources of financial support as does sustaining some other types of interventions, but these efforts depend strongly on the political and financial climates affecting the recipient organization. If the political or financial climate is turbulent, then agency administrators’ attention may be diverted from continuing to implement the new skills or processes developed during the capacity-building initiative.

• Capacity or infrastructure building that focuses on changes in technology or standard operating procedures (e.g., organizations’ financial management procedures) is more likely to be sustained after full implementation than is capacity building that focuses on training individuals, assuming that these procedural changes outlast staff turnover.

Funders working toward capacity or infrastructure development might enhance sustainability by focusing on installing new technologies, such as new information systems or new operating procedures, rather than training individuals who may be transient. If individual training is the capacity development needed, a program that includes both current political appointees and longer-term stable employees in that training may increase the likelihood that changes in capacity will survive individual turnover to continue internal administrative support.

Collaborative Partnerships or Coalitions

Some initiatives involve developing long-term collaborative relationships among organizations, such as coalitions, collaborations, or partnerships. For example, community coalitions were fostered by federal agencies to work on community stop-smoking campaigns⁴⁷ and by foundations to coordinate asthma intervention providers with recipient families.⁴⁸ They also have a long history in substance abuse prevention programs.⁴⁹ Many partnerships and coalitions also develop program-focused activities involving individual providers, more complex coordinated programs, or advocacy for new policies. However, the coalition or partnership itself may be worth sustaining, even if its programmatic activities change or decline.⁵⁰ Sustaining the partnership could lead to new joint activities with other sources of funding, to better coordination among community agencies, or to joint advocacy for needed policy change in the targeted content area.

Hypotheses about the sustainability of this type of intervention include

• Formal coalitions or partnerships developed during a funded initiative are more likely to be sustained than are the activities delivered during the funded period, if partnership members continue a commitment to meeting and working together.

• Sustaining coalitions or partnerships beyond the initial funded period may enable them to develop new activities, win new grants, or otherwise continue to address the focus problem area, perhaps by modifying their initial activities.

• Sustaining coalitions or partnerships beyond their initial funded period may not require new external funding sources; coalition leadership and partners’ perceptions of the value of continued affiliation are more influential than additional external funding.

Programs that develop coalitions and partnerships may have longer-term outcomes or benefits for their participants other than continuing to provide the program activities initially addressed. Yet the continued existence of the partnership or coalition does not guarantee that it is still addressing the needs that initially led to its formation. To foster continuation of coalitions, funders and managers might focus on developing strong, responsive coalition leadership, so members feel their time is well spent when continuing the coalition after initial funding ends. Organizations participating in coalitions might enhance the sustainability of their partnership work by ensuring that all members’ contributions receive favorable reception, so that members remain motivated to continue participating.

Broad-Scale System Change

Some initiatives go beyond implementing a specific set of program activities within the existing system of organizations in a targeted content area. These system-change initiatives aim to rearrange the system itself: the many ways that organizations, practitioners, financial sources, and other factors in a setting interact to affect the problem area. They involve change of, rather than within, the system. For example, the Maine Health Access Foundation funded a multiyear initiative to integrate primary care and behavioral health for Maine beneficiaries. Beginning in 2008, this initiative involved (1) grants to provider organizations to locate both primary care and behavioral health providers in the same offices, (2) training for several types of staff to foster changes in their roles and procedures, and (3) extensive networking among grantees to foster mutual learning about how to make these changes. After organizational case examples illustrated how to implement integrated care, the foundation began disseminating these models of integrated care to other, nongrantee providers and encouraging financial agencies toward policy changes to sustain the new provider roles for integrated care. The initiative was also supported by multimethod evaluation and ongoing feedback of evaluation results.⁵¹

Broad-scale system change often requires longer-term efforts than does implementation of a specific program, and the needed program activities and components cannot always be defined in advance. If the system change is successful, sustainability then becomes a property of the system as a whole, rather than the outcome of a specific intervention.⁵² Researchers have called for greater attention to the nature and evaluation of health system change, but literature on health program sustainability has not often focused on this broader type of system change.^5,7,53–55

Hypotheses about the sustainability of this type of intervention include

• Broader-scale system change is likely to require a long period of continuing and diverse efforts to achieve the desired outcomes; these continuing responses to the change environment are more likely to occur than a scenario of implementing and sustaining a pre-specified set of program activities.

• Sustaining efforts toward broader-scale system change is likely to require continued financial support for a long time (e.g., 10–20 years), rather than the typical 3- to 5-year grant period. In the United States, securing financial support from insurance sources for new ways of promoting health may be a key influence for broader processes of system change.

• Environmental contexts are likely to be especially influential for sustaining changes in a broader health system: for example, whether financial support is available for continued dissemination and training for providers’ changed roles, whether patients or recipients recognize the value of new methods and advocate for the changed system, and how financial and political turbulence affects the system.

Funders and system-change managers need to be aware of the complexity of broad-scale change, to avoid expecting multicomponent innovations to be easily implemented and sustained. Implementing and sustaining an intervention delivered by individual providers working alone (the first type of intervention in this framework) is quite different from initiating broad-scale system changes that involve coordinated changes by many individuals and organizations. It may be especially important to include ongoing evaluation within system-change initiatives, to provide feedback about what is occurring on the ground, and to look for short-term outcomes expected to lead to longer-term and broader-scale changes, rather than to design research to assess the efficacy of the total initiative. Each broad-scale change effort is likely to be unique to its environment and its set of players; therefore, replication of specific systemic changes in other locations may not be feasible.

IMPLICATIONS

The 6 types of interventions are likely to have quite different implications for both research and practice concerning sustainability. Although some interventions may have characteristics that overlap or shade between these types, distinguishing the major features of the focal program will clarify thinking and research about influences on sustainability. These intervention types may also have implications for earlier phases of broader research-to-practice processes, but those implications are beyond the scope of this article. Certainly, research about dissemination and implementation processes would be clarified by including attention to the type of intervention being addressed and to its environmental context. Furthermore, the role of continued fidelity of implementation for ensuring continued benefits to the target audience may differ among these intervention types: programs with community partnerships or multiple components by different actors are most likely to be modified over time, but interventions implemented by individual providers or involving a policy change may need to be sustained with fidelity to be effective.¹³

Another question is whether intervention type is related to the aims or importance of the intervention. This question is worth exploring as research accumulates about sustainability, but it appears not to be a necessary relationship. For example, if smoking cessation is the intended health outcome, it might be addressed by interventions delivered by individual providers, such as physicians’ brief counseling and advising nicotine replacement; by change in no-smoking policies or cigarette tax rates; by community coalitions mounting a social marketing campaign against smoking, or by other strategies. Which of these types of interventions is most likely to be sustained beyond an initial implementation period is subject to the hypotheses outlined here. Which type of intervention is most likely to be effective in smoking cessation, in itself or in combination with other interventions, is the focus of comparative effectiveness research and beyond the focus of this article.

It might seem that some intervention aims can only be addressed by individual providers and therefore always require the first type of intervention, such as a new surgical procedure implemented by appropriately trained surgeons. However, if that new surgical procedure also requires role changes among other staff members (e.g., operating room staff), then it may actually be a more interpersonally complex, second type of intervention.

The hypotheses suggested for each type of intervention are propositions about what might be expected from future research that differentiates among these intervention types. These hypotheses are not known empirical findings from currently available studies, because few researchers studying sustainability have incorporated clear descriptions of intervention type when designing studies or reviewing other studies. These hypotheses are intended to encourage researchers to explicitly collect evidence to test them, so that guidance for program managers about how to increase sustainability will eventually be based on stronger evidence. Additional hypotheses are desirable to extend and clarify this framework of intervention types.

Appropriate guidance for efforts to increase sustainability will also differ by the type of change being implemented. Technical assistance by external trainers may be particularly desirable for programs requiring continued planning, administrative support, and often, additional financial resources (e.g., interventions requiring coordination among multiple staff and broad-scale system change). Other types of interventions may benefit more from continuing evaluation that provides monitoring and feedback, for example, to assess continued intervention delivery by individual providers or to monitor enforcement of policy changes. Efforts to increase sustainability must also take into account the political and financial environments in each organizational context. It may be unrealistic to expect a particular program to be sustained if the host organization is struggling for financial survival or is implementing other complex interventions.

Research and evaluation about sustainability will be enhanced by tailoring data collection designs to the type of intervention being addressed. For example, when the intervention is primarily implemented by individual providers, data collection for assessing sustainability should focus on those individuals’ behaviors and on their motivations and resources for continuing or abandoning the intervention. By contrast, data collection about sustaining interventions requiring coordinated organizational actions from several types of staff members will benefit from multiple streams of data about organizational processes, as well as about the sustained and coordinated actions of individuals. For both these intervention types, collecting data about actual services and benefits to the target recipients would also be highly desirable. Several intervention types are likely to need continuous data collection during the intended sustainability period, to monitor enforcement or continued use (e.g., for policy or procedural changes) or to assess intended short-term outcomes during system-change efforts or after a capacity-building effort.

The randomized controlled trial, favored for assessing the underlying efficacy of interventions, is not likely to be an appropriate research design for assessing the sustainability of those interventions.⁵⁶ Even if an effort to enhance sustainability is being evaluated (e.g., external technical assistance), a randomized trial alone is not likely to provide the most useful knowledge. For these sustainability enhancement efforts, local organizational context and numerous environmental influences are likely to be more powerful influences on sustainability than externally provided technical assistance. For many interventions, processes involved in sustainability involve interactions over time among multiple influences at a local level, including leadership and champions; ability to secure additional funding, if needed; capacities and stability of internal staff and external partners; and the use of feedback data about delivery and effectiveness.^11,13 A randomized trial would not illuminate how the multiple influences interact to foster sustainability or allow program abandonment.

Sustainability research and its resulting advice to practitioners has previously not differentiated among diverse types of interventions. Findings and advice derived from some types may not apply to other types. In addition, research methods that are appropriate for a specific type of intervention may yield disappointing results when applied to sustainability assessment. Sustainability research and practice will both benefit from careful consideration of the type of program to be maintained—one size does not fit all!