Reliability and validity of a Spanish-language measure assessing clinical capacity to sustain Paediatric Early Warning Systems (PEWS) in resource-limited hospitals

Abstract

Background

Paediatric Early Warning Systems (PEWSs) improve identification of deterioration, however, their sustainability has not been studied. Sustainability is critical to maximise impact of interventions like PEWS, particularly in low-resource settings. This study establishes the reliability and validity of a Spanish-language Clinical Sustainability Assessment Tool (CSAT) to assess clinical capacity to sustain interventions in resource-limited hospitals.

Methods

Participants included PEWS implementation leadership teams of 29 paediatric cancer centres in Latin America involved in a collaborative to implement PEWS. The CSAT, a sustainability assessment tool validated in high-resource settings, was translated into Spanish and distributed to participants as an anonymous electronic survey. Psychometric, confirmatory factor analysis (CFA), and multivariate analyses were preformed to assess reliability, structure and initial validity. Focus groups were conducted after participants reviewed CSAT reports to assess their interpretation and utility.

Results

The CSAT survey achieved an 80% response rate (n=169) with a mean score of 4.4 (of 5; 3.8–4.8 among centres). The CSAT had good reliability with an average internal consistency of 0.77 (95% CI 0.71 to 0.81); and CFAs supported the seven-domain structure. CSAT results were associated with respondents’ perceptions of the evidence for PEWS, its implementation and use in their centre, and their assessment of the hospital culture and implementation climate. The mean CSAT score was higher among respondents at centres with longer time using PEWS (p<0.001). Focus group participants noted the CSAT report helped assess their centre’s clinical capacity to sustain PEWS and provided constructive feedback for improvement.

Conclusions

We present information supporting the reliability and validity of the CSAT tool, the first Spanish-language instrument to assess clinical capacity to sustain evidence-based interventions in hospitals of variable resource levels. This assessment demonstrates a high capacity to sustain PEWS in these resource-limited centres with improvement over time from PEWS implementation.

Strengths and limitations of this study.

• The Clinical Sustainability Assessment Tool (CSAT) was translated to Spanish and edited for clarity and syntax by regional group of experts, ensuring cultural appropriateness.

• The CSAT was administered to the multidisciplinary Paediatric Early Warning System (PEWS) implementation leadership teams of 29 diverse paediatric cancer centres in Latin America implementing PEWS, adding to generalisability of results.

• Analysis focused on establishing reliability (psychometric, confirmatory factor analysis) and initial validity of the CSAT, as well as describing the clinical capacity to sustain PEWS in participating centres.

• Study strengths include the mixed-method design, large sample size, high response rate and diversity of participants and centres across multiple countries, allowing for evaluation of individual and hospital characteristics associated with clinical capacity for sustainability and feedback on the CSAT report.

• Limitations include the risk of bias to the study, including from the predominant inclusion of PEWS implementation leaders and the use of the CSAT to assess a single intervention (PEWS).

Background

Paediatric Early Warning Systems (PEWSs) are evidence-based bedside assessment tools to identify early clinical deterioration in hospitalised children.¹ PEWSs are particularly useful for hospitalised children with cancer, who frequently experience clinical deterioration resulting in preventable mortality,² especially in resource-limited hospitals.^3–6 PEWS implementation improves patient outcomes through promoting interdisciplinary communication between nurses and physicians, leading to quicker identification of clinical deterioration and prompt intervention to address complications.^6–11 In 2017 Proyecto Escala de Valoración de Alerta Temprana (EVAT), a quality improvement collaborative of Latin American paediatric oncology centres, was formed to improve survival of hospitalised children with cancer through PEWS implementation.^{6 12} Currently, Proyecto EVAT has 60 centres in 19 countries; to date, 37 have successfully implemented PEWS.¹³ Despite the potential benefit of PEWS over time, the long-term sustainability of PEWS is not yet established.

Many clinical interventions like PEWS lack evidence demonstrating their sustainability. Sustainment, or the maintenance of an intervention over time, is critical to maximise benefits of evidence-based interventions,^14–17 particularly in low-resource settings.^{18 19} Theoretically, sustainment is associated with greater hospital clinical capacity to maintain an intervention, including skilled staff, adequate finances and engaged leadership.^{20 21} Implementing new interventions is costly, and if they are not sustained, then initial investments are lost.^{22 23} Premature abandonment of effective interventions may lead to staff frustration, damaged relationships with patients and a loss of general capacity to provide services.²⁴ For these reasons, understanding sustainment of evidence-based interventions is extremely important, particularly in low-resource settings where resources available for intervention implementation are constrained.²² Unfortunately, there is little empirical evidence about factors that contribute to intervention sustainment in hospital settings, in part due to a lack of measurement tools.²⁵

In this study, we describe the development and testing of a Spanish-language version of the Clinical Sustainability Assessment Tool (CSAT), a theoretically informed, reliable measure of clinical capacity for intervention sustainment.^{26 27} Currently available in English, the CSAT can be completed either individually or as a group in 15 min or less. On completion, a complementary CSAT report is generated that summarises the responses to the tool and helps identify areas for improving clinical capacity.²⁸ The CSAT is being used increasingly in a variety of clinical settings and used to evaluate diverse interventions, indicating it is a useful measure of sustainability for researchers, evaluators and clinicians. The goal of this study was to evaluate the usability, reliability and validity of a Spanish-language CSAT and demonstrate its use to evaluate capacity for sustaining PEWS in resource-limited hospitals participating in Proyecto EVAT.

Methods

Proyecto EVAT

EVAT is a Spanish-language PEWS composed of a five-component scoring tool (neurologic, cardiovascular, respiratory, staff concern and family concern) associated with an escalation algorithm that guides the clinical team in the treatment of a deteriorating patient.⁹ Through collaboration between St. Jude Global at St. Jude Children’s Research Hospital (SJCRH) and local stakeholders, Proyecto EVAT was formed in 2017 with the goal to improve survival in hospitalised paediatric oncology patients through implementation of PEWS.^{12 29} As part of Proyecto EVAT, each hospital assembles a local PEWS implementation leadership team, adjusting the size to account for local needs. In collaboration with the EVAT Steering Committee, a 26-member multidisciplinary team of nurses and physicians from 11 hospitals in 8 countries in Latin America, Proyecto EVAT provides hospital-to-hospital mentorship through a three-phase implementation process, including planning, piloting and outcome assessment, to help establish the resources and processes necessary to implement and maintain PEWS. A centre is considered to have completed PEWS implementation when they achieved an error rate, as indicated by incorrect calculation of PEWS or algorithm use, of less than 15% for at least 2 months.¹³

In the first 3 years of the programme, we successfully supported PEWS implementation in 37 hospitals of varying resources.¹³ Of these, 29 centres from 14 countries completed implementation prior to June 2020, when this study was conducted (see figure 1). These centres represent diverse hospital organisation, including paediatric multidisciplinary, dedicated oncology and general (adult and paediatric) hospitals with private, public and mixed (public–private) funding structures, including hospitals with or without dedicated paediatric oncology units. Together, these hospitals manage approximately 4300 annual new diagnoses of childhood cancer annually (online supplemental table 1). While all hospitals are mentored through the same three-step phase implementation process, our previous experience suggests that centres face a variety of sustainability challenges, including turnover of PEWS leadership teams, variable hospital leadership support for the programme, and human and material resource limitations. No formal assessment of PEWS sustainability, however, has been conducted.

Figure 1.

Participating centres (n=29) map depicting 29 Proyecto EVAT collaborating paediatric oncology centres participating in the pilot of the Spanish-language CSAT with centre characteristics. CAST, Clinical Sustainability Assessment Tool; EVAT, Escala de Valoración de Alerta Temprana; PEWS, Paediatric Early Warning Systems.

CSAT adaptation

We adapted the CSAT, a brief and reliable instrument consisting of 35 items within seven domains to assess an institution’s capacity for sustaining a clinical practice.^{10 12} These domains include Engaged Staff & Leadership, Engaged Stakeholders, Organisational Readiness, Workflow Integration, Implementation & Training, Monitoring & Evaluation and Outcomes & Effectiveness.^{26 27 30} Each domain includes five items that are scored on a Likert scale from 1 to 7 (35 questions total), where 7 indicates an individual believes their institution has that domain to a great extent. CSAT development and testing demonstrated excellent internal consistency and several trends towards discriminant validity.²⁷ For this survey, we also include several validating questions taken from existing measures including the organisational readiness to change assessment³¹ and the Change Process Capability Questionnaire,³² which bear conceptual overlap with the CSAT and include indications of intervention implementation as well as quality improvement. The survey was translated to Spanish by bilingual staff at SJCRH, iteratively edited for clarity and syntax, and back-translated to confirm accuracy. Based on feedback from the EVAT Steering Committee and the research teams’ experience with prior surveys in Latin America, the Likert scale was adjusted from the 7-point original scale to a 5-point scale. This is modification is consistent with literature demonstrating that reduction of Likert anchors resolved ambiguities and improved validity, reliability and response rates among Latino populations within the USA.^{33 34} An electronic version of the Spanish tool was reviewed by 19 members of the EVAT Steering Committee, representing countries in Mexico, Central and South America, to establish baseline acceptability within the context of Proyecto EVAT. During testing, survey completion took 10–15 min. Feedback was integrated into the final version of the Spanish CSAT tool. The Spanish CSAT is available at https://sustaintool.org; the English version used in this study is included in online supplemental figure 1.

CSAT pilot testing and data collection

After initial translation and testing, the adapted Spanish CSAT was distributed through an anonymous electronic survey on Qualtrics³⁵ to PEWS implementation leadership teams of 29 centres completing implementation by July 2020 (online supplemental table 1). The PEWS implementation leadership team consists of the core individuals (physicians, nurses, and administrators) responsible for implementing and maintaining PEWS at their centre, as identified by each site leader working with our programme. This included an average of 7 (range 4–15) individuals per centre, with a total 210 participants surveyed. The assessment consisted of the CSAT (35 questions) as well as demographic questions about the individual, intervention and organisation (online supplemental figure 1). The survey remained open for 1 month following distribution and weekly reminders were sent to participants. Centre-specific data were collected from PEWS implementation site leader at each participating centre. Time since implementation of PEWS was calculated from the date of implementation completion (as defined above) to when the survey was completed (1 August 2020). After completing the assessment, each participant was provided a Spanish-language centre-specific CSAT report summarising results from their hospital (see online supplemental figure 2 for English example).

Data management and analyses

The survey data were analysed using R (V.4.0.4). Initially, descriptive analyses were used to explore the responses. Psychometric and multivariate analyses using R package Lavaan were conducted to assess CSAT reliability and validity characteristics. Reliability analyses focused on the internal consistency (ie, Cronbach’s alpha) for each of the seven CSAT domains. Internal consistency measures the extent to which individual items in a subscale measure the same underlying construct.³⁶ Confirmatory factor analysis was conducted to confirm the subscale structure and assess the fit of the items within their hypothesised latent domains. In the confirmatory factor analyses, we used traditional fit indices to assess the adequacy of the CSAT structural model, including the comparative fit index (CFI), the root mean square error of approximation (RMSEA) and the standardised root mean square residual (SRMR),³⁷ as well as the Vuong model fit test.³⁸ Analysis of variance (ANOVA) was used to assess initial validity of the measure. This was conducted using demographic characteristics of the setting, PEWS intervention and participants. ANOVA was used to study the association between centre and respondent characteristics and CSAT results.

Focus groups

Following survey administration and distribution of centre CSAT reports, all participants were invited to participate in focus groups to discuss the CSAT tool and report, focusing on usability. From those volunteering to participate, three focus groups were organised to be homogeneous by discipline: nursing, paediatric floor physicians and intensivists. This focus group structure encouraged open and honest discussions. The focus groups were semistructured using a facilitator guide to assess participants’ ability to interpret the CSAT report and to elicit constructive feedback to improve its usability. The facilitator guide was developed based on the components of the CSAT report along with questions regarding the participant’s report use. This guide was translated into Spanish by bilingual members of the research team (CV, AA and MP-T). A pilot focus group was conducted including sseven participants from four countries. The focus group guide was revised based on feedback from this pilot session and finalised. The finalised guide was used for all focus groups (see online supplemental figure 3).

As participants were from hospitals all over Latin America, and the study occurred during the COVID-19 pandemic, focus groups were conducted over WebEx, a secure online platform familiar to all participants and accessible from a computer or mobile device. Participants were asked to engage using their videos and microphones to approximate an in-person discussion. An experienced bilingual native Spanish speaker not otherwise involved in PEWS implementation facilitated all focus groups (CV).

Focus groups were audiorecorded, then simultaneously translated to English and transcribed by a professional service. Transcripts were subsequently deidentified, and uploaded into MAXQDA qualitative software (VERBI, Berlin, Germany) for analysis. Each participant response was used as the unit of analysis. As the focus group aimed to address specific questions regarding the report, an a priori codebook was deductively developed by the research team based on the facilitator guide to analyse data regarding the interpretation of the report, specific report components, and general feedback (negative and positive) about participants’ experience with the CSAT (see online supplemental table 2). A single coder (AA) coded all transcripts. Thematic analysis was conducted on coded data to describe the participants overall perspectives on the report, its usability, as well as specific recommendations for improvement.

Results

Descriptive statistics

The CSAT survey achieved an 80% response rate (total n=169 responses from 29 centres, centre median response rate 83.3%, see online supplemental table 1 for centre details). Respondent demographics are described in table 1. The overall mean CSAT result was 4.4 (out of 5; range per centre 3.8–4.8) (table 2).

Table 1.

Respondent demographics and association with CSAT result

Category	Subcategory	n (%) n=169	Mean CSAT	P value
Individual demographics
Profession	Nurse	75 (44.4)	4.46	0.510
	Physician (Peds HO, Peds ICU, other physician)	86 (50.9)	4.40
	Other (management, administration, coordinators, other)	8 (4.7)	4.56
Role in PEWS	PEWS implementation leader	115 (68.0)	4.48	0.006
	Clinical staff	40 (23.7)	4.25
	Other (hospital admin, data admin, other)	14 (8.3)	4.54
Years worked since professional degree	Less than 5 years	46 (27.7)	4.47	0.290
	From 6 to 10 years	32 (19.3)	4.33
	More than 10 years	88 (53.0)	4.45
	NA	3
Gender	Male	37 (21.9)	4.47	0.576
	Female	132 (78.1)	4.42
Age	Less than 30 years	12 (7.1)	4.54	0.382
	30–40	77 (45.6)	4.39
	40–50	59 (34.9)	4.43
	More than 50 years	21 (12.4)	4.54
Centre characteristics
Type of hospital	General (adult and paediatric)	67 (39.6)	4.38	0.386
	Oncology (adult and paediatric)	44 (26.0)	4.49
	Paediatric multidisciplinary	58 (34.3)	4.45
Hospital funding structure	Public	117 (69.2)	4.47	0.087
	Private or public/private partnership	52 (30.8)	4.34
Annual new cancer diagnoses	1–75	68 (40.2)	4.45	0.845
	76–150	49 (29.0)	4.43
	>150	52 (30.8)	4.41
Paediatric oncology unit structure	No paediatric oncology unit (integrated with paediatrics or other unit)	19 (11.2)	4.39	0.602
	Separate paediatric	150 (88.8)	4.44
Time since implementation of PEWS	1–12 months	67 (39.6)	4.27	<0.001
	12–24 months	66 (39.1)	4.53
	>24 months	36 (21.3)	4.55
No of staff working in centre	0–249	24 (14.2)	4.51	0.398
	>249	145 (85.8)	4.42

CSAT, Clinical Sustainability Assessment Tool; HO, Hematology-Oncology; ICU, Intensive Care Unit; NA, not applicable; PEWS, Paediatric Early Warning System.

Table 2.

CSAT subscale and overall descriptive statistics

Domain	Mean	SD	Low	High	Cronbach’s alpha
Engaged staff and leadership	4.55	0.48	2.80	5.00	0.77
Engaged stakeholders	4.33	0.55	2.80	5.00	0.71
Organisational readiness	4.08	0.66	2.00	5.00	0.80
Workflow integration	4.49	0.52	2.50	5.00	0.76
Implementation and training	4.37	0.56	2.40	5.00	0.79
Monitoring and evaluation	4.48	0.53	2.20	5.00	0.81
Outcomes and effectiveness	4.75	0.41	2.80	5.00	0.78
Overall	4.43	0.42	2.74	5.00	0.77

CSAT, Clinical Sustainability Assessment Tool.

Psychometrics and CSAT structure

The Spanish version of the CSAT shows acceptable to excellent reliability—an average Cronbach’s alpha of 0.77, ranging from 0.71 to 0.81 across the seven subscales (table 2). Given the hypothesised structure of the seven CSAT domains, confirmatory factor analysis was used to assess how well the data fit this structural model. The results suggest acceptable fit with a CFI of 0.825, RMSEA and SRMR scores of 0.067 (table 3). Traditional guidelines suggest that CFI scores of >0.80 indicate acceptable fit, and RMSEA and SRMR scores of <0.08 also indicate acceptable fit.³⁹ Importantly, these results show that the seven-subscale model fits significantly better than the single factor model. More specifically, the smaller AIC value, the smaller relative χ² statistic and the significant Vuong model fit test (LR=424.5, p<0.001) all show that the seven factor model fits the data better than the single factor model.³⁸ This reiterates the idea that the translated version of the CSAT retains an important subscale structure to measure clinical capacity.

Table 3.

Confirmatory factor analysis (CFA) results of baseline and final Spanish CSAT instruments

Phase	Subscales	Items	Chi/df	CFI	RMSEA	SRMR	AIC
Baseline	1	35	2.31	0.685	0.088	0.080	10529.6
Final	7	35	1.76	0.825	0.067	0.067	10147.1

Total n=169; CFA model fit with robust maximum-likelihood. Vuong’s test of model distinguishability demonstrated that the final seven domain model was a significantly better fit to the data than the baseline single domain model (Likelihood ratio=424.5, p<0.001).

CFI, Comparative Fit Index; CSAT, Clinical Sustainability Assessment Tool; RMSEA, root mean square error of approximation; SRMR, standardised root mean square residual.

Initial validation analyses

The survey included questions to assess the respondents’ perception of PEWS, the PEWS implementation process and current use in their centre, and general questions describing the institutional culture and implementation climate. Respondents who reported a higher strength of evidence supporting PEWS use, a stronger PEWS implementation leadership team, more frequent use of PEWS in the clinical setting, and generally collaborative hospital climate that is receptive to changes for quality improvement also scored higher on the CSAT assessment, suggesting good construct validity. Of participants, 43.1% somewhat or strongly agreed that their centre’s resources were too tightly limited to improve quality of care, however, this was not associated with the total CSAT result (table 4).

Table 4.

Validation questions influencing CSAT result

Validation question	Response category	n (%) n=169	CSAT mean	P value
Perceptions of PEWS
Please rate the strength of the scientific evidence supporting PEWS implementation.	Weak +neither weak nor strong	7 (4.2)	4.05	<0.001
	Strong	56 (33.3)	4.31
	Very strong	105 (62.5)	4.53
	NA	1
How important is PEWS to provide quality care to your patients?	Not very important (neither important nor unimportant +somewhat important)	17 (10.1)	3.95	<0.001
	Very important	152 (89.9)	4.49
PEWS Implementation process and use
How difficult was the implementation of PEWS in your hospital?	Very difficult	16 (9.6)	4.21	0.054
	Somewhat difficult	77 (46.1)	4.44
	Neither easy nor difficult	41 (24.6)	4.40
	Somewhat easy	24 (14.4)	4.54
	Very easy	9 (5.4)	4.68
	NA	2
Our PEWS implementation team understands and uses quality improvement skills effectively.	Neither agree nor disagree	6 (3.6)	3.27	<0.001
	Somewhat agree	59 (34.9)	4.23
	Strongly agree	104 (61.5)	4.61
Regarding patients under my care, how often is PEWS used in their care?	None of the time +some of the time	7 (4.1)	4.17	0.002
	Most of the time	22 (13.0)	4.19
	All the time	140 (82.8)	4.49
Centre culture and implementation climate
Our resources (personnel, time, financial) were too tightly limited to improve care quality.	Strongly disagree	16 (9.8)	4.50	0.764
	Somewhat disagree	41 (25.0)	4.38
	Neither agree nor disagree	34 (20.7)	4.44
	Somewhat agree	50 (30.5)	4.41
	Strongly agree	23 (14.0)	4.50
	NA	5
Our clinical team has changed or created systems in the hospital that make it easier to provide high quality care.	Neither agree nor disagree	20 (12.0)	4.14	<0.001
	Somewhat agree	79 (47.3)	4.29
	Strongly agree	68 (40.7)	4.68
	NA	2
We choose new processes of care that are more advantageous than the old to everyone involved (patients, clinicians, and our entire clinical team).	Strongly disagree +somewhat disagree	3 (1.8)	3.98	<0.001
	Neither agree nor disagree	17 (10.1)	3.98
	Somewhat agree	75 (44.6)	4.31
	Strongly agree	73 (43.5)	4.68
	NA	1
The working environment in our clinical team is collaborative and cohesive, with shared sense of purpose, cooperation, and willingness to contribute to the common good.	Strongly disagree +somewhat disagree	8 (4.7)	4.16	<0.001
	Neither agree nor disagree	17 (10.1)	4.19
	Somewhat agree	73 (43.2)	4.30
	Strongly agree	71 (42.0)	4.66
Our clinical team has greatly improved quality of care in the past 12 months.	Somewhat disagree +neither agree nor disagree	14 (8.3)	4.03	<0.001
	Somewhat agree	66 (39.1)	4.26
	Strongly agree	89 (52.7)	4.63

CSAT, Clinical Sustainability Assessment Tool; NA, not applicable; PEWS, Paediatric Early Warning System.

Individual and centre characteristics were evaluated for associations with the CSAT result (table 1). Front-line clinical staff rated sustainability lower than PEWS implementation leaders and other administrative staff (p=0.006). We found no other significant differences by other individual demographics. Similarly, hospital characteristics, such as type and size of centre, volume of paediatric cancer patients or paediatric oncology unit structure, was not associated with the performance on the CSAT assessment. However, respondents at centres with a longer time since achieving PEWS implementation, rated sustainability of PEWS significantly higher (p<0.001). This relationship was significant across all domains except organisational readiness (online supplemental table 3). Similar association was seen in centre-level analysis, although not statistically significant (p=0.085, (online supplemental tables 3 and 4). The relationship between time from implementation of PEWS and mean CSAT total and domain results are further described in online supplemental figures 4 and 5.

CSAT report and tool usability

To evaluate the CSAT report, we conducted three focus groups (see online supplemental table 5 for focus group participant demographics), with a total of 22 participants (7–8 per group) from 10 countries. Participants generally found the CSAT and its report useful to assess the clinical capacity for sustainability of PEWS at their centre; ‘This tool provided a lot of information…we got a high score, however, we still have some items to improve. Yes, I love this tool because it tells us what items we need to pay attention to.’ (Intensivist). Specifically, participants felt the tool allowed them to identify areas of strength and opportunity for improvement ‘I consider that it is a valid tool, understandable in terms of this process that we can first see how we are numerically, but then it gives us an analysis to be able to say what we are doing well and what things we should improve’ (Nurse) and ‘This is very positive because at the end they are giving me tips on the things that I should do within everything related to sustainability and all my weaknesses that it evaluates; it is telling me to focus on this and work on this.’ (Nurse). They also saw utility in using the tool to advocate for institutional support and resources to improve clinical capacity to sustain PEWS: ‘Well, obviously this type of tool, CSAT report, is a tool to advocate; I mean, with this report I can talk to my foundation boss and tell her, we have a team that wants to work, a proper work flow, a very good score in monitoring and evaluation, but we lack organizational readiness, how easily can you through the foundation keep supporting us based on this report? That would be what I would do to advocate.’ (Oncologist)

Participants recommended shortening the introductory text, reducing redundancy in reporting domain results and including visual and descriptive cues to allow for easier interpretation of each centre’s strengths and weaknesses. They also suggested providing more guidance on how to use the report and next steps: ‘I strongly agree to add a conclusion, or steps to follow in the last part of the second page, it would be a summary of what we have to do, actions to be taken’ (Nurse).

Discussion

The CSAT is the first reliable Spanish/English bilingual instrument to assess clinical capacity to sustain interventions and adding to the existing data of the CSAT’s performance in high-resource settings.²⁶ The Spanish CSAT performed well across a heterogeneous group of respondents from diverse hospitals in Latin America with variable resource limitations, indicating good reliability. Our findings demonstrate that the respondents’ perception of the clinical capacity to sustain PEWS at their centre was associated with a positive perception of the intervention, its implementation process and use, and the centre’s culture and implementation climate, suggesting initial validity in this setting. We also demonstrated the utility of the CSAT report for helping clinicians understand their capacity for sustaining interventions and presented potential ways the report might be improved. These proposed changes will be integrated into the next version of the CSAT report.

Sustainability is the least-studied aspect of the implementation continuum for evidence-based interventions,^{25 40 41} and presents a challenge across a range of interventions and settings,¹⁷ potentially limiting the long-term impact of effective interventions on patient outcomes. These problems are magnified in resource-limited settings, where investments for implementation and maintenance of interventions are more constrained.²² Tools such as the CSAT are needed to aid clinicians and hospitals seeking to assess their organisation’s clinical capacity to sustain interventions. The translation of the CSAT to Spanish for this study further allows for the broad use of this measure, as lack of multilingual measure has been a notable scientific barrier for other instruments.^{42 43} Now available in both English and Spanish, the CSAT and its associated centre report provide an opportunity for clinicians and researchers to assess institutional factors contributing to intervention sustainability, thus improving the sustainment of evidence-based interventions and maximising their benefits for patients. Additional research lending validity to the CSAT, including its application to different clinical settings and interventions, would be valuable to confirm or contradict the relationships demonstrated in this study.

This study leveraged an international collaborative to evaluate the clinical capacity to sustain one intervention, PEWS, across a variety of respondents and hospital settings. This allowed us to not only evaluate the CSAT, but also study these hospitals’ capacity to sustain PEWS. The clinical capacity to sustain PEWS was rated higher by respondents directly involved in PEWS implementation than others (clinical staff and hospital administrators), suggesting that implementation leadership may feel more enthusiastic about the intervention than other hospital staff. The CSAT results did not vary, however, by other respondent demographics or centre characteristics, indicating these factors did not influence a centre’s ability to sustain PEWS over time. A longer history of PEWS use (longer time from implementation) was associated with a higher clinical capacity for sustainment of PEWS as rated by the CSAT. Together, these findings suggest that centres build capacity to sustain improvement interventions over time.^{21 43} This outcome is promising, indicating hospitals are able to maintain the necessary infrastructure to sustain PEWS, regardless of hospital characteristics or resource level. Such results are important for clinicians, hospital authorities, and funders as they indicate that investment in implementation of PEWS is likely to result in sustained improvements in patient outcomes over time. However, while these preliminary results are positive, future research to prospectively evaluate the relationship between hospital clinical capacity, assessed by a broader group of hospital staff, PEWS sustainment, and impact on clinical outcomes is needed to strengthen the evidence for this relationship. Lastly, the CSAT and the CSAT report helped hospitals identify specific limitations in clinical capacity for sustainment, creating an opportunity to grow capacity though targeted improvements. In the qualitative portion of our study, several participants noted the CSAT report could aid them to advocate for needed resources to hospital stakeholders, potentially giving clinicians a tool to actively build their institution’s capacity over time.

This study has several limitations. Our respondents sample included primarily PEWS implementation leaders, who generally rated the sustainability of PEWS higher than other participants and are likely more invested in maintaining PEWS use at their centres. This may have introduced bias to our study, resulting in overestimates of the clinical capacity to sustain PEWS and contributing to the observed restriction of range. A more broad sample of participants, including clinical staff using PEWS, is needed in future work. Similarly, there is a potential risk of non-response bias, as we do not have demographic details for the 20% non-responders to compare with study participants. However, these limitations should not impact our ability to assess the reliability and validity of the CSAT instrument, which was our primary goal of this study. Similarly, while the current work evaluated the use of the CSAT to assess clinical capacity to sustain a single intervention (PEWS), prior diverse experience with the CSAT suggests it has a potential for broad utility to evaluate sustainability of clinical interventions globally.

Conclusion

The CSAT is a Spanish/English bilingual instrument to assess the clinical capacity to sustain evidence-based interventions in hospital settings of variable resource levels. The CSAT report summarises survey results to help clinical teams interpret their performance and identify areas of opportunity. This assessment of diverse hospitals in Latin America implementing PEWS suggests that that clinical capacity for sustainment grows over time. Future work should explore this finding to further evaluate how centres build clinical capacity to sustain PEWS during initial implementation and how maintenance of capacity relates to sustainability of PEWS use and its impact on patient care over time. To promote use of the CSAT, the English and Spanish version are currently available publicly at https://sustaintoolorg/. Broader dissemination and adoption of this sustainability assessment tool for clinical interventions and quality improvement efforts in hospitals of all resource levels will help ensure sustained improvements in patient outcomes from maintenance of critical evidence-based practices.

Data availability statement

Data are available on reasonable request. Deidentified data from this study are available upon reasonable request to the corresponding author.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

This study was approved by the St. Jude Children's Research Hospital (SJCRH) Institutional Review Board (IRB) as exempt research under category 2 (research that includes interactions involving survey procedures and minimal risk) as study number 20-0501. Per this determination, no written informed consent was required from participants; consent was inferred from their agreement to complete the anonymous survey or participate in the focus group.