The development of the sustainability measure for healthcare using a modified Delphi process

Abstract

Background

Valid and reliable measures for assessing the sustainability of complex, multicomponent, and interdisciplinary healthcare interventions are lacking. The study objective was to develop a multidimensional instrument for use to assess the sustainability of complex, interdisciplinary, healthcare interventions implemented in acute care settings.

Methods

Content experts participated in a modified Delphi study of electronic REDCap^® measures. Round 1, composed of 49 structured and unstructured questions, was analyzed using descriptive statistics and content analysis. In rounds 2 and 3, experts rated items derived from round 1 to provide evidence of sustainability on a 5-point Likert-type scale. Questions rated by > 75% of the experts as important were retained for the final measure.

Results

Ten experts representing areas of quality improvement, sustainability, and implementation science participated in rounds 1 and 8 experts in rounds 2 and 3, respectively. Round 1 statements with a median value of < = 6 on the 10-point Likert scale or < = 3 on the 4- or 5-point Likert scale were retained. The items retained, modified, and added in Round 2 included 53 items. Questions rated by > 75% of the experts as important were retained for the final measurement; conversely, questions rated by ≤ 25% of the experts as important were discarded. Twenty-five items with associated %s expert ratings of “important” (between 25% and 75% from Round 2) were included in Round 3. The modified Delphi process resulted in a final 37-item scale.

Conclusions

Using a modified Delphi technique, experts reported varying perceptions of sustainability. However, commonalities in key areas were successfully translated into the Sustainability Measure for Healthcare for assessing the sustainability of complex, multicomponent interventions.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-025-13291-7.

Contribution to the literature

• This paper demonstrates how the electronic Delphi process can be used successfully to develop a measurement tool in health services research.

• This paper fills an important knowledge gap for a measure that is not context or intervention specific, which limits their broad application to assess the sustainability of evidence-based interventions.

• This paper reports on the development of the Sustainability Measure for Healthcare, which is a context-generic measurement tool that can be used in different settings, populations, and interventions.

Background

Sustainability is commonly thought of as a set period of time during which interventions, programs, or implementation strategies are continually delivered [1]. However, the current literature lacks a consistent formal definition of sustainability [2], with definitions including concepts such as “continued program activities”, “continuous health benefits”, “capacity built”, “adaptation”, and “evidence-based intervention” [3–5]. Sustainability has been conceptualized in terms of institutionalization and routinization, continuation, and maintenance. Institutionalization and routinization refer to the embeddedness or integration of program components into organizational processes to the degree that they become standard of practice [6, 7]. Others conceptualized routinization as the ongoing use of the principles and practices of an organizational routine while institutionalization refers to contextual adaptations that occur over time to accommodate new work practices [8]. These conceptualizations, however, does not reflect the complexity of the overall construct of sustainability [4].

A recent review of the sustainability of programs [2] indicated that Scheirer’s definition of sustainability, namely as the continuation of the components and activities of programs in order to achieve the intended program and population outcomes [9] is most often used. Their work was informed by the work of Shedeiac-Rizkallah and Bone [10] who clarified sustainability as a dynamic process that encompasses the continuation of programs in various potential forms or formats [10]. Stirman and colleagues noted sustainability as a multidimensional global term that includes the continuation of program benefits and program activities but included program capacity in their definition [10]. Chambers and colleagues’ recent definition of sustainability also aligned with that of Scheirer, as they integrated the concepts of continued intervention delivery over time, institutionalization within settings, and capacity building as part of their definition. Moore et al. identified five concepts as encompassing sustainability and their definition can be summarized as the period during which the program continues to be delivered while individual behavioral change is maintained or evolves/adapts and produces benefits for individuals or systems [4].

The lack of consensus and ensuing ambiguity of the definition of sustainability may be attributed to the fact that sustainability is measured as an outcome in some studies but measured as a process in others [11, 12] Additionally, the difference between measuring predictors of sustainability as an outcome versus sustainability as a process is not well characterized [1, 12–14]. Predictors of sustainability as an outcome are related to organizational and contextual strategies [15, 16]while the outcomes of sustainability as a process are associated with improved targeted results and quality improvement goals. A recent systematic review of sustainability measures identified 11 measures [17]. The authors found none to be applicable across settings nor could any be tailored for specific evidence-based practices (EBPs). Moreover, none of the identified measures were pragmatic or psychometrically sound, none were generalizable across varying interventions, and all contained questions that were difficult for frontline providers to answer [17].

Furthermore, there is a gap in the current knowledge of what and/or how certain processes and factors affect sustainability [1]. Often, there is an attempt to quickly fix a “problem” if program metrics are not being met rather than taking the time to understand deeper problems that may impede sustainability. Thus, a valid and universal measure specifically for sustainability could help stakeholders and funders more precisely detect and address processes or factors that inhibit the long-lasting sustainability of beneficial interventions [18]. Including a measure of sustainability in ongoing process evaluations and continuous assessments of how the local context supports an intervention [3, 19] highlight contextual problems, provide essential information about ongoing needs of the intervention, and aid in continued feedback [3, 20]. To our knowledge, no measure exists that captures the multidimensionality of the sustainability of complex interventions.

This paper responds to the call by Moullin et al. [17] for a generic measure that captures the various domains of sustainability and can be adapted to assess the sustainability of different EBPs in healthcare settings. This paper answers the call for more rigorous tools to consistently and reliably assess the sustainability of complex healthcare interventions in acute care settings.

Methods

Study objectives

The study objective was to develop a multidimensional instrument for use to assess the sustainability of complex, interdisciplinary, healthcare interventions implemented in acute care settings. We posited that the Delphi process would best capture the multidimensionality of sustainability and enhance the clarity of the instrument.

Study context

The eDelphi study was conducted in the context of a larger AHRQ K01-funded study (HS025486) to examine the implementation and sustainability of a Medication Reconciliation (MedRec) toolkit (known as the MARQUIS Toolkit) which aimed to reduce the number of unintentional medication discrepancies in hospital systems [21–23].

Ethical considerations

The Vanderbilt University Medical Center Institutional Review Board (IRB # 170736) approved this study.

Use of the modified Delphi process to build consensus

The Delphi consensus approach is an iterative qualitative methodology consisting of repeated questioning, anonymity, and controlled feedback until consensus is reached [24, 25]. With the modified Delphi process investigators generate initial ideas through literature or document review, interviews, or focus groups [25]. The Delphi process typically comprises three rounds of review and feedback; however, the number of rounds may vary depending on the difficulty of the questions and thus how quickly consensus is reached [26]. The level at which consensus is determined to have been achieved varies between studies, with reported consensus values ranging from 51 to 100%, with 70% agreement being common [25].

In this study, an electronic modified Delphi process was used consisting of web-based questionnaires specifically developed for this study (see Supplemental File) and executed using the secure Research Electronic Data Capture (REDCap^®) [27] web platform, with each round of survey results and analysis informing the next round. The REDCap^® platform provides a mechanism for building secure research databases and online surveys [28]. The Delphi process has been successfully implemented using electronic platforms in prior studies, and the process provides the benefits of collecting data from geographically diverse experts, affordability, and anonymity [24, 29, 30].

Literature search

Under the guidance of a mentoring team, the first author conducted a search of the literature to identify elements of sustainability as captured in definitions and measurement items. These elements were then used to formulate statements and questions to be included in round one of the modified Delphi method. PubMed and Google Scholar databases were initially searched for English full-text articles published in the last 10 years that included the term “sustainability” in the title or abstract. Because other terms are often used in the literature to refer to sustainability, the following terms were also included in the search: institutionalization, routinization, continuation, and maintenance [6, 7]. These terms emphasize the embeddedness or adaptation of programs to standards of practice or organizational routines [6–8]. Because sustainability is often used to refer to environmental sustainability, titles of articles in the results were reviewed to ensure the term sustainability were used to refer to program/intervention sustainability. After an initial cursory scan of titles and abstracts, articles with explicit definitions (operational or theoretical) and measures of sustainability were reviewed. Under the guidance of the mentoring team, the first author used existing sustainability definitions and measures to develop statements for inclusion in the eDelphi round 1. For example, from Rabin et al.’s [31]. sustainability definition, which includes “. intended benefits over an extended period of time after external support from the donor agency is terminated.” led to statements of “The program delivers the intended health benefits over an extended time period.” and “the program is continued after external funding ceased”.

Panel selection

Following IRB approval, letters were sent to sustainability and implementation science experts, as well as quality improvement experts, inviting them to participate in the study. The number of experts recommended for content validation varies, and some recent studies have indicated panel sizes ranging from 12 to 110 [25]. Others suggest that between five and ten experts suffice [32]. Practical considerations when choosing a panel size include resources to analyze the data and provide feedback to participants [25]. For practical purposes and to allow for potential attrition between Delphi rounds, 10 to 15 experts were proposed for this study. A pool of experts in implementation science, sustainability, and quality improvement was generated. Experts in implementation science and/or sustainability were researchers who had extensively published or presented on the topic or had successful grant applications in the area and had some background in quality improvement. Quality improvement (QI) experts held positions related to QI or had published extensively on the topic. Individuals were recruited using emailed letters containing a link to the REDCAp^® survey. Once the experts clicked on the link, they were presented with an IRB-approved consent document, and once they consented to participate, they gained access to the survey document.

Round 1

The goal of round one was to obtain experts’ evaluations of the initial set of items, as well as to garner qualitative comments on sustainability to develop additional items and further enhance the robustness of the final instrument. The survey consisted of an introduction, the initial list of sustainability statements developed from the literature review, and some demographic questions. The sustainability statements were divided into three sections. In Section A, experts ranked the importance of 32 statements on a Likert-type response options. Because we were concerned about low sensitivity and variation when using a 5-point Likert scale for questions related to indicators of sustainability, we instead used a 9-point Likert Scale (10th item was not applicable) on questions we deemed particularly important to informing round 2 questions (19 questions in total). A response of ‘1’ indicated “Most Important”, and the highest number indicated “Least Important” regardless of the range of responses. In Section B, experts were asked to respond in textual format to five questions about their perspectives on sustainability. For example, participants were asked to define the term “sustainability” and whether the context of care should affect its measurement. Section C contained 12 multiple-choice questions to gain participant perspectives on sustainability concepts identified in the literature, such as which program components to be continued, the timepoint at which to measure sustainability, and the role of program benefits in a measure of sustainability. After the demographic questions were answered, a text box was provided for additional comments. Responses to the Likert response statements were tabulated using frequency distributions and median values. The qualitative responses were independently reviewed line by line by two research staff members and summarized into item statements. The two researchers then met and discussed the summary statements and agreed on one statement capturing the panelists’ perspectives.

Round 2

Recalling that a lower rating indicated greater importance from the experts, round 1 statements with a median value of < = 6 on the 10-point Likert scale or < = 3 on the 4- or 5-point Likert scale were retained in either their original or modified format for Round 2. These statements, along with additional statements derived from the qualitative feedback in Round 1, resulted in the set of 51 questions included in Round 2. As in Round 1, Round 2 panelists were asked to indicate the importance of each item for measuring sustainability on a Likert-type scale ranging from 1 (strongly disagree) to 5 (strongly agree). Panelists were also provided with prompts that asked for clarification of qualitative themes that emerged from round one regarding the timing of sustainability measurement, diffusion as sustainability and the role of context, program complexity, and the interprofessional nature (IP) of programs in measures of sustainability. Frequency distributions were used to summarize the experts’ responses. Questions rated by > 75% of the experts as “important” (4 or 5 on a 5-point scale) were retained for the final measure; conversely, questions rated by ≤ 25% of the experts as important were discarded. Those statements rated between those values were included in Round 3 of the modified Delphi process (n = 26 questions, Table 3).

Table 3.

Summaries of responses to Delphi rounds 2 and 3 statements (N = 8 experts)

Statement	(R2) % Agree	(R3) % Agree
Outcome Focused (19 items)
Program implementation achieved the intended health/administrative benefits.	75
Program implementation achieve continued benefits for the intended clients.	100
The program delivers the intended health/administrative benefits over an extended time period.	100
The program is delivered at a sufficient level or fidelity or intensity to yield the desired health outcomes.	75
Lasting improvement in positive outcomes occurred.	63	75
The program aims are achieved.	50	63
Long-term solutions to the goals of the program are continued.	63	63
The program demonstrates the desired health/administrative outcomes over multiple time points.	88
The program is continued for the timepoint set forth by those implementing the program to meet organizational or regulatory requirements.	13
The program is continued when any extra-organizational supports- financial or otherwise- are removed from the implementation setting.	88
Program implementation achieved the desired outcomes.	50	63
The program has a stable, secure source of funding.	38	38
The program continues after funding sources are terminated.	88
Program benefits endure beyond program implementation.	100
Program benefits endure more than 5 years beyond program implementation.	63	38
The program has achieved a level of organizational ownership with less support from outside organizations.	88
The program is used as intended or when indicated.	38	63
The program is continued at least at the same level/percentage as at baseline.	25
The program is used proficiently and consistently over time by targeted organizational members.	50	63
Maintenance Focused (7 items)
The program has become institutionalized.	88
The program has become routinized.	88
The program has become both institutionalized and routinized.	88
The program is being maintained.	88
The program is fully integrated into existing organizational structures or processes.	88
Neither program institutionalization nor routinization is relevant to program sustainability.	25	0
The program is fully integrated in the unit(s) or organization(s) where the program was adopted and implemented.	75
TBC Focused (7 items)
Core components of the program, which remain recognizable from other programs.	50	50
All components of the program.	13
Successful components of the program are continued, and unsuccessful elements are discarded, regardless of whether they were core components of the program.	50	75
New/changed practices, procedures, and policies to support the program are continued.	88
The adaptations to components that were made during program implementation are continued, even if it reflects a lower level of the program components, as indicated by achieving outcomes.	75
The components necessary for effective outcomes are continued and can be either the original or adapted program components.	75
The program components that are required by factors such as policy, best practices, or regulatory requirements are continued.	63	75
Assessment/Measurement Focused (18 items)
The standards that define the program	63	75
A small set of outcome, process, and balancing measures.	50	75
Adaptations when benchmarks are not met and it becomes necessary to tailor more specific content to the program.	63	63
Institutional support is present.	63	88
Leadership support is present	50	75
Fidelity to the protocol	50	50
Program progression in development and implementation	50	38
Economic, environmental, and social measures are in place.a	13	13a
Changes in process measures as the program is adapted over time	63	88
Documentation to support the program and its use is in place	75
Education to support the use of the program is provided on an ongoing basis.	63	75
Program metrics are in place and used regularly to assess outcomes.	75
Feedback on program metrics is provided to stakeholders at regular intervals.	63	75
Program related policies and procedures are in place.	75
Staff has knowledge and skills related to the program	88
A program vision and mission are in place.	50	63
Accountability systems to facilitate program use	75
The program has been in place more than 1 year after implementation.
The program has been in place more than 1 year after funding ended.
Structures are in place to support ongoing program assessment and improvements.b	63	100b

^a Question modified after Round 2: “The program and structures and processes that support the program are continued (may include collaborative structures).” New/modified from Round 1

^b Question unintentionally included in Round 3

Round 3

This round included only the questions without consensus of importance from Round 2, with modifications or additions from the qualitative queries included in that previous round. The expert participants were informed of each item statement and associated percentage of experts rating it as “important” in Round 2. The experts were asked to again select the strength of their agreement with the importance of the statement as an indicator of sustainability on a 5-point Likert-type scale (1 = strongly disagree to 5 = strongly agree). Two “Select All That Apply” questions (e.g., when sustainability should be measured and if diffusion indicated sustainability) with response items were included in Round 3 to clarify the responses received during Round 2. In this case, the question was stated followed by the response options, and for each response option, the % agreement that was previously achieved. Frequency distributions were once again used to assess the % agreement among the experts on the importance of each item.

Results

Modified Delphi panel selection

Over a period of 6 weeks, a total of 10 out of 20 experts responded to our invitation to participate in the eDelphi process. Of these, all 10 participated in round 1, eight participated in round 2, and eight participated in round 3. Participants varied in areas of expertise including quality improvement, medication reconciliation, and leadership with all rating their level of expertise as moderate to high. See Table 1 for details.

Table 1.

Modified Delphi panel expert description (N = 10)

Age, years, median (IQR)	50 (43,59)
Profession	n (%)
- Physician	3 (30)
- Registered Nurse	3 (30)
- Quality Improvement Specialists or Health Services Researchers	4 (40)
Time in current role
- At least 5 years	10 (100)
- More than 10 years	7 (70)
Expertise
- Medication Reconciliation	1 (10)
- Program sustainability	4 (40)
- Other aspects (leadership, program design and implementation)	5 (50)

Round 1

A review of the literature resulted in a total of 49 structured and unstructured questions for inclusion in Round 1 (Table 2). Of the 49 questions, the experts rated the importance of 32 questions on either a 10-, 5-, or 4-point Likert scale (1 = Most important; 4, 5 or 10 = Least Important) and 12 questions asked participants to indicate their perspective on when a program can be considered sustained and were asked for each item stem, to select one best option from up to 5 potential response choices. For example, for the question “The program components continued are: (select one) response options were 1 = all components; 2 = core components; 3 = other, and 4 = not applicable to sustainability. Another question asked “A program can be considered sustained when the benefits from the program implementation:” with response options of “1 = endure beyond program implementation; 2 = are achieved during/at the end of program implementation; 3 = other; and 4 = not applicable to sustainability”. An additional five open-ended questions asked participants their viewpoint regarding the definition of sustainability and the role of the care context, complexity, and interprofessional nature of a program in the measurement of sustainability,

Table 2.

Summaries of responses to Delphi round 1 set of questions (N = 10 experts)

Statement	Median
Outcome Focused (1 = Most Important, 10 = Least Important)
Program implementation achieved the intended health benefits.a	4.0
Program implementation achieve continued benefits for the intended clients.a	3.0
The program delivers the intended health benefits over an extended time period.a	2.0
The program is delivered at a sufficient level or fidelity or intensity to yield the desired health outcomes.a	5.0
Lasting improvement in positive outcomes occurred.a	2.5
There is a continued perception of program accomplishments.	9.0
Long-term solutions to the goals of the program are continued.a	6.0
Program activities perform at the same level than at the time of the initial implementation.	7.5
Program activities perform at a higher level than at the time of the initial implementation.	8.0
Program services are delivered at a higher percentage in follow-up audits when compared to baseline.	8.0
Maintenance Focused (1 = Most Important, 10 = Least Important)
The program has become institutionalized, that is, the organization's context has adapted, including its structures and processes, to the new program.a	3.0
The program has become routinized, that is, it has become part of the habitual practices of organizational members.a	5.0
The program has become both institutionalized and routinized evident by contextual adaptations and routine use of the program by organizational members.a	2.5
The program has become so institutionalized and/or routinized that it has reached obsolescence in the organization.	7.5
The program is being maintained, that is, the program is adapted as needed after implementation to improve its performance, remove obsolete aspects no longer in use, or to add functionalities to continue to achieve intended program outcomes.a	3.5
The program is fully integrated into existing organizational structures or processes. That is, the program is no longer a stand-alone entity and it cannot be differentiated from other programs.a	3.0
The program is integrated to some degree into organizational structures or processes and can be differentiated from other programs.	7.0
Attention to the issues that are addressed by the program implemented is maintained.	7.0
The program has achieved permanence in the implementation setting.	7.5
TBC Focused (1 = Most Important, 4 = Least Important)
Core components of the program is maintained/routinized/institutionalized and remain recognizable from other programs.a	3.0
All components of the program are maintained/routinized/institutionalized.	4.0
Successful components of the program are continued, and unsuccessful elements are discarded, regardless of whether they were core components of the program.a	2.1
New/changed practices, procedures, and policies to support the program are continued/maintained/routinized.a	2.0
Time Focused (1 = Most Important, 4 = Least Important)
The program is continued after external funding ceased. a	2.0
Funding to support the program is secured.	4.0
The program is in place more than 1 year after implementation. a	3.0
The program is in place more than 1 year after research/project-funding period ends.a	2.0
Diffusion Focused (1 = Most Important, 5 = Least Important)
The program diffuses to other settings/units/organizations. a	2.5
Collaborative efforts to support the program spread to other settings/units/organizations.a	3.0
The program has achieved a level of organizational ownership with less support from outside organizations.a	3.0
The capacity of a collaborative structure, such as a coalition, is maintained.	3.5
The organization uses the intervention effectively, that is, use it as intended and when indicated.a	2.5

^a Statements were retained in original or modified format for Round 2

Round 1 informed the development of 49 items for inclusion in Round 2 in several ways. First, noting that a lower rating indicated greater importance from the experts, round 1’s 32 question statements with a median value of < = 6 on the 10-point Likert scale or < = 3 on the 4- or 5-point Likert scale were retained in either their original or modified format for Round 2. Items meeting the criteria for inclusion in Round 2 are indicated in Table 2. Summaries of the experts’ ratings of importance for the 32 Likert response items are shown in Table 3. In addition, responses to the 12 additional questions specific to program sustainability were summarized. If a majority (> 50%) of participants indicated that the question referred to a concept applicable to sustainability, the question was reformatted as an additional item statement for inclusion in Round 2. For example, the majority of participants (n = 7; 70%) indicated they considered a program as sustained when the core components of a program are continued even if as an adapted program or include lower levels of the program components. The following item stem was then generated from that question, “The components necessary for effective outcomes are continued and can be either the original or adapted program components”. Participants who indicated “Other” to the questions were asked for additional comments. Those qualitative comments were analyzed and main themes were also reformatted as item statements for inclusion in Round 2. Participants highlighted the need to maintain core program components to achieve the desired outcomes and the longevity of program outcomes once implementation or outside support stops. One participant noted, “Maintenance of core program components deemed necessary to achieve outcomes over a prolonged period of time without outside/research staff involvement and support.”

Qualitative definitions of sustainability

Qualitative comments from Round 1 provided information on how participants perceived sustainability and whether the organizational context should inform the measurement of sustainability. Key themes emerging from the comments included sustainability as maintenance of program fidelity, support from organizational resources, over a defined period of time, and benefitting from the program. As one participant noted, “The process owners of a program (a) perform the key processes in nearly every indicated instance (b) at high fidelity to the protocol (c) to the desired effect (d) indefinitely (e) supported by the needed institutional resources.”

Round 2

The retained, modified, and added items included in Round 2 are shown in Table 3. Using a Likert-type scale ranging from 1 (strongly disagree) to 5 (strongly agree), participants were asked to indicate the extent to which they agreed that each item measured sustainability. Summaries of the combined response options “agree” and “strongly agree” were used to indicate “important” and are shown in Table 3. Questions rated by > 75% of the experts as important were retained for the final measurement; conversely, questions rated by ≤ 25% of the experts as important were discarded. Statements rated between those values were included in Round 3.

Panelists in Round 2 were also provided with several open-ended clarifying questions from rounds one and 20 about themes related to the timing of sustainability measurement, diffusion as sustainability and the role of context, program complexity, and the interprofessional nature (IP) of programs in measures of sustainability. Responses to those additional more qualitative questions, as well as to the items with inclusive importance ratings, were submitted to the expert participants in Round 3.

Round 3

During Round 3, each of the items with importance ratings of 25–75% from Round 2 were distributed to the expert participants, along with its respective rating of importance from Round 2. Participants were again asked to rate the importance of each item for measuring sustainability on a Likert-type scale ranging from 1 (strongly disagree) to 5 (strongly agree). Summaries of the resulting percentage of experts providing an ‘Agree’ or ‘Strongly agree’ response from that final round are also shown in Table 3. Those items with at least 75% agreement were included in the final pool of potential items, along with those retained from Round 1, for the new measure.

Discussion

This study used a modified Delphi process with experts to develop a set of items for assessing the sustainability of healthcare interventions. The iterative nature of the Delphi technique, with its cycles of feedback and revision, enables the refinement of measurement concepts and the identification of the most salient and comprehensive set of items for inclusion in the final instrument.

The modified Delphi process is widely used in research related to healthcare delivery and interventions to build consensus and in developing guidelines in health care, identifying and setting research priorities, and improving the implementation of research findings [33–36]. This technique has been effectively used to establish consensus for the diagnosis and treatment of patients with different diseases, to establish research priorities [37, 38]select outcome measures [39]and establish clinical guidelines [40]. Furthermore, its use spans across various disciplines including nursing, medical, and health services [41] and its effective use for developing instruments has also been demonstrated [42]. For example, Oliver and colleagues [43] successfully used the modified Delphi process to garner the input from multiple stakeholders including patients, clinicians, and researchers, to develop a set of outcomes in inflammatory bowel disease. Jacob et al. similarly used the modified Delphi process to develop assessment criteria for the evaluation of eHealth tools [44].

With the availability of Internet resources, conducting the Delphi technique by leveraging technology is feasible and provides a practical, cost-effective alternative to traditional paper-based and in-person Delphi techniques [45]. Using these technologies, we were able to include experts across the U.S. in our study to systematically clarify sustainability and its measurement.

Some of the highest priority challenges set out by Proctor and colleagues as the sustainability research agenda [14] were the identification of key or core elements of sustainability and methodological advances in sustainability research. The results from the current study can help individuals move the field toward both of those goals by identifying potential core domains of sustainability in health care research and by creating a measure for assessing those domains. Using a panel of experts constructed by the Delphi method, the core factors and measure of sustainability identified in this study are cross-sections of the input of many experts, highlighting the commonality of their definitions of sustainability and the importance of the sustainability construct.

The Sustainability Measure for Healthcare is context generic and can be used to assesses the sustainability of various evidence-based interventions implemented in different acute care contexts. It therefore extends the work of Palinkas et al. [13] who developed a measure of sustainment that is context specific. It focuses exclusively on prevention programs with grant support from the Substance Abuse and Mental Health Services Administration (SAMHSA). This work also addresses a critique by Moullin and colleagues [17] who called for psychometrically sound generic measures that are not context specific.

The differentiation of sustainability as a process and sustainment as an outcome likely affects how it is measured. In the development of their measure, the Sustainment Measurement System Scale, Palinkas et al. conceptualized sustainability as the final process of implementation, referring to it as the continued use of an innovation in practice [13, 46].In this measure, they included both determinants of sustainability and sustainment as an outcome. The conceptualization of sustainment used by Palinkas et al. aligns with the conceptual model developed by Aarons and colleagues who sees sustainment, referring to the continued use of an innovation in practice, as the final process in implementation [13, 46]. In contrast and in alignment with our prior work [16, 47, 48]in this current study, we used the term sustainability to reflect an outcome following the implementation of an intervention. Our conceptualization aligns with Proctor and colleague’s conceptualization of sustainability as an important implementation outcome resulting in a sustainability outcome measure [14, 49]. In sum, this study addresses recent calls for clear definitions and methods to bring clarity to sustainability and methodologically advance the sustainability research field by developing a tool to assess the sustainability of healthcare interventions.

Implications for future research, study strengths and limitations

The development of the Sustainability Measure for Healthcare stems from the input from experts in the domains of quality improvement, health services research and sustainability and in the acute care domain of healthcare delivery. We focused specifically on acute care settings and the generalizability of the Sustainability Measure for Health Care is therefore limited to acute care settings. Initial pilot testing of the measure with a small sample has been completed (not reported here) but further rigorous work is needed for a robust evaluation of the psychometric properties of the tool. Future work is also needed to evaluate the transferability of this measure to other settings and populations, including community settings, primary care settings, and in services outside of healthcare delivery. Work in this area has begun with the use of this measure to assess the sustainability of an intervention to improve patient safety. We are also planning to refine the measure for use in complex interventions implemented in community settings.

Our study has several strengths and limitations. We used a robust process to develop and test our new measure by drawing on experts in the field of implementation, sustainability, and healthcare improvement. The experts in our study represented a diverse group of professionals, allowing for heterogeneity in the responses and suggestions. The expert panel consisted of participants from only one area of the United States and therefore the perspectives of experts in the wider United States and international arena are not represented. As we noted earlier, we have started additional work on the measure to test it in other settings and including a diverse group of individuals representing the broader US society.

Methodological limitations are also inherent in the use of surveys and interviews. Such methods increase the risk of social desirability responses. We mitigated that risk by ensuring participant confidentiality, formulating questions without bias, and asking questions in a neutral tone to avoid leading the participants to a particular response. Attrition was another limitation of our study, as there was a drop-off in the modified Delphi method participants between rounds one and two.

Additional work is planned to address the current limitations. One limitation is the risk of survey response burden, as the measure is somewhat lengthy. Developing a shorter version would reduce this risk and likely enhance its use in future studies. Another limitation is the evaluation of the measure for participant understanding of the survey instructions and questions. To address these limitations, additional work is being conducted, including cognitive interviews, to further clarify the questions and response options. Cognitive interviewing is an important methodology for evaluating the design of questionnaires, as well as for refining both survey instructions and items [50, 51] Additional testing of the updated tool is planned, including developing a shorter version to limit participant response burden in future assessments to enhance the potential for use in research and practice. Testing of the measure in various settings and populations would facilitate ongoing refinement. Despite these limitations, the researchers believe that dissemination of the current measure is important for making the tool available to other researchers for use and further testing and adaptation in other settings and for advancing the field of and measurement of sustainability.

This study utilized a robust Delphi process to develop a novel Sustainability Measure for Healthcare that could be used to assess the sustainability of various evidence-based interventions across acute care settings. While the current tool is limited to acute care, future work is planned to refine and test the measure’s transferability to other healthcare settings, including community-based and primary care environments. Overall, this newly developed sustainability assessment tool represents an important step forward in addressing key priorities for sustainability research and measurement in healthcare.

Abbreviations

QI

Quality improvement

MARQUIS2

Implementation of a Medication Reconciliation Toolkit to Improve Patient Safety Study

MedRec

Medication Reconciliation

Authors’ contributions

DPS conceptualized, designed and executed the study; DPS and MSD determined the methodology; DPS conducted the qualitative analysis; and MSD conducted the quantitative analysis. ACJ drafted the manuscript outline; all the authors contributed to the original manuscript and reviewed and edited all the drafts.

Funding

Dr. Stolldorf was funded by a grant from the Agency for Healthcare Research and Quality (K01HS025486; 75% effort; $424,270), and the use of the REDCap platform was supported by an institutional grant (UL1 TR000445) from NCATS/NIH. The authors are solely responsible for this document’s contents, findings, and conclusions, which do not necessarily represent the views of AHRQ or official views of the National Institutes of Health. Readers should not interpret any statement in this product as an official position of AHRQ or of the U.S. Department of Health and Human Services. None of the authors has any affiliation or financial involvement that conflicts with the material presented in this product.

Data availability

The datasets analyzed during the current study are available within the scope of reasonable requests as deidentified data and with written request to the corresponding author and with a data sharing agreement between institutions in place.

Declarations

Ethics approval and consent to participate

The Vanderbilt University Medical Center Institutional Review Board (IRB # 170736) approved this study. Informed consent was obtained for participation in the modified Delphi technique and survey. This study adhered to the Declaration of Helsinki.

Consent for publication

No patient information was included in this manuscript, and patient consent was not needed.

Competing interests

The authors declare no competing interests.