Process evaluations undertaken alongside randomised controlled trials in the hospital setting: A scoping review

Abstract

Background

There is increasing recognition of the importance of undertaking process evaluations alongside implementation of health interventions by examining mechanisms of impact and contextual factors. However, a comprehensive synthesis of process evaluations undertaken alongside clinical trials in hospital settings is lacking. We undertook a scoping review to address this gap.

Methods

This review was guided by the methodological framework for scoping studies. Studies were identified using four databases; Ovid Medline, EBSCO CINAHL, EMBASE and Scopus. Two authors independently screened all titles and available abstracts, with a third author available to adjudicate. Studies were eligible for inclusion if they described a process evaluation undertaken alongside a randomised controlled trial in the hospital setting. Data were abstracted by one author and checked by two others and analysed both descriptively and using inductive content analysis.

Results

Data were extracted from 30 articles reporting on 15 trials, most of which were cluster randomised trials (c-RTs) (n = 12). The most common data collection methods used in process evaluations were interviews, questionnaires or surveys, and records or logs. Data analysis revealed three themes relative to how authors: use process data to interpret, understand and explain trial outcomes; evaluate responses to the intervention; and consider the implementation context.

Conclusions

Findings from this review demonstrate the complex nature of intervention implementation in the hospital setting. Overall, there is need for standardised reporting of process evaluations and more explicit descriptions of how authors use frameworks to guide their evaluation.

1. Introduction

The importance of undertaking process evaluations alongside implementation of health interventions is increasingly recognised [1]. Clearly, testing an intervention's effectiveness alone overlooks important information about how or why it worked, the extent to which it was implemented as intended and its relevance and reproducibility in other settings [1,2]. Process evaluations address these limitations by examining the quality and quantity of what is being implemented, the causal mechanisms underpinning an intervention, and contextual factors influencing intervention outcomes [2].

Hospitals are unique and complex organisations. In the hospital context, intervention implementation and uptake can be influenced by many factors such as resource requirements, organisational culture, motivation to change, and structural elements including staff turnover, workload and time constraints [1]. To fully understand uptake, it is important that researchers capture and describe these factors when evaluating interventions in this setting; undertaking a parallel process evaluation is one way to achieve this [2].

Several published reviews have investigated practices in process evaluations of health interventions, providing valuable information about key frameworks and research methods used [[3], [4], [5], [6], [7]]. However, these reviews have focused on specific settings or study populations, limiting their generalisability and wider applicability (3–7). For example, previous reviews have focused on process evaluations of interventions delivered to patients with chronic disease in the primary care setting [3], complex interventions for patients with musculoskeletal disorders [4], public health interventions undertaken in low and middle income countries [5,6] and, interventions to change healthcare professionals’ clinical practice behaviour [7]. A comprehensive synthesis of process evaluations undertaken alongside interventions tested in the hospital contexts, however, is lacking.

To address this gap, a scoping review was undertaken to examine what is known about process evaluations undertaken alongside hospital-based interventions tested in RCTs and c-RTs. This review may provide guidance for other researchers planning to undertake process evaluations when designing randomised controlled trials (RCTs) by identifying relevant frameworks and methods to address their research questions.

2. Methods

A scoping review was undertaken guided by Arksey and O'Malley's framework [8]. This framework, first described in 2005 and expanded by Levac and colleagues in 2010 [9], describes five key steps: (i) identifying the research question; (ii) identifying relevant studies; (iii) selecting the studies; (iv) charting the data; and (v) collating, summarising and reporting the results.

This method was selected as it is appropriate for synthesising studies with diverse methods and broad research questions in terms of the target population or health problem [8,9]. Further, scoping reviews are helpful for highlighting research trends and important gaps by examining when, where and by whom studies are being published and the types of methods being employed [8,9]. Thus, findings will support future researchers in planning and undertaking a process evaluation; an important justification for conducting this type of review [8,9]. This paper follows the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for scoping reviews checklist (PRISMA-ScR) [10].

2.1. Step 1: identifying the research question

Subsumed under the overarching aim (i.e. what is known about process evaluations undertaken alongside RCTs and c-RTs in hospital settings) were the following research questions:

• 1.What are the study characteristics of RCTs and c-RTs and their related process evaluations, undertaken in acute hospital settings?
• 2.What theoretical or conceptual frameworks underpin process evaluations of RCTs and c-RTs in the acute hospital setting, and which are more commonly cited?
• 3.What process evaluation components are assessed in hospital settings?
• 4.What data collection and analysis methods are used to undertake process evaluation components in hospital settings?
• 5.How do the authors use process data to interpret, explain or better understand trial outcomes?

2.2. Step 2: searching for relevant studies

Search terms were developed by referring to key texts, browsing database medical subject headings and discussion among the research team experienced in undertaking process evaluations. Four electronic databases – Ovid Medline, EBSCO CINAHL, Scopus and EMBASE-were independently searched with support from a university health librarian who assisted with the search strategy. Following the database searches, all references were exported to EndNote X9 for removal of duplicates and then imported to Rayyan [11], a web-based application designed for systematic reviews. The reference lists of included articles were searched to identify accompanying articles (including main trial outcomes and/or additional process evaluation findings) that may have been missed in the database search. Supplementary file 1 presents the full electronic search strategy for one of the four databases.

2.3. Step 3: selecting the studies

Two authors (IS and RW) independently screened all titles and available abstracts against the eligibility criteria in Rayyan [11]. A third author (BG) was available to resolve any discrepancies that arose.

Studies were eligible if they:

• •Explicitly identified as a process evaluation or clearly aligned with the MRC [1] definition of a process evaluation, i.e. a primary study evaluating a specific intervention, which can be qualitative, quantitative or mixed-methods, and aims to understand the functioning of an intervention by examining implementation, mechanism of impact, and contextual factors [1]. Where multiple process evaluation papers were published from the same trial, additional results were extracted;
• •Undertaken alongside an RCT or c-RT testing the effectiveness of an educational, treatment, diagnostic or prevention intervention;
• •Conducted in acute hospital settings, i.e. inpatient, intensive care units, emergency departments, medical or surgical units;
• •Focussed on addressing a health condition or problem for hospital inpatients (studies were also eligible if participants include healthcare professionals, provided the outcomes focussed on patients); and
• •Peer reviewed primary research articles published in any year (i.e. no date restrictions) in English.

We excluded studies undertaken in rehabilitation, paediatrics, obstetrics and mental health services, as well as studies where the intervention was delivered in a mix of both acute hospital and other settings. Pilot and feasibility studies and Phase 0 to II clinical trials were excluded as the nature of the process evaluation typically focuses on feasibility and piloting in contrast to effectiveness trials where process evaluations aim to examine fidelity, quantity and quality of what was delivered, generalisability and context [1]. Protocols, reviews, methodological papers, opinion pieces, commentaries, books and book chapters were excluded; only peer reviewed journals were included as these enhance credibility of included studies and facilitate comparisons across the literature. Reviewing non-English language was beyond the expertise of authors and because the review was unfunded, we did not have the resources for translation.

In accordance with Arksey and O'Malley's methodological framework [1], eligibility criteria were refined as familiarity with the literature increased. For example, after reviewing full texts the team decided to only include studies where the entire intervention was undertaken in acute hospital settings. Further, on close examination, some terms which initially appeared synonymous with “process evaluation” did not in fact align with the Medical Research Council's (MRC) definition (e.g. “program evaluation” and “process of care measures”) [1]. Eligibility criteria were subsequently amended to reflect this.

2.4. Steps 4 & 5: charting the data and collating, summarising and reporting the results

A data extraction form was developed to ensure standardisation of information collected on each study [8]. Development of the instrument was an iterative process involving input from all authors; as articles were reviewed and understanding of the topic deepened, new types of data were extracted. Authors maintained a log of decision-making to document this process.

One author (IS) extracted data from eligible articles with two authors (BG and RW) independently verifying that extracted data were consistent with the original article. For some types of information (e.g. author, year, country), data were extracted and displayed in a summary table.

To answer the review question how do the authors use process data to interpret and explain trial outcomes? the results sections of process evaluations were extracted and analysed using inductive content analysis [12,13]. Data were initially divided into meaning units, which were further condensed, and codes were subsequently derived. During these initial stages, authors remained close to the data with minimal interpretation in keeping with recommendations [12,13]. Once all texts were coded, categories were derived by grouping codes that appeared to belong together, with minimal interpretation [13]. The final stage involved developing themes; this level of abstraction for reporting results is considered appropriate if the intention is to answer questions relating to ‘who, what, when, where and how?’ as was the case with this review [13]. Research team meetings were held throughout data analysis to ensure consensus regarding the evolving categories and themes (i.e. researcher triangulation). We also documented analytical decision-making of each step of data analysis and developed a table to provide key examples of the data analysis process to clarify how codes, categories and eventual themes were derived from the raw data/meaning units (refer to Supplementary file 2) [13].

3. Results

A total of 1451 articles were identified from all sources after duplicate records were removed. Of the 1451 articles screened, 30 articles (reporting on 15 trials) were included in this review, presenting process evaluation findings (n = 14) [[14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27]], main trial outcomes (n = 11) [[28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38]], or process evaluation and main trial outcomes together (n = 5) [1,[39], [40], [41], [42], [43]] (Fig. 1). In total, the two reviewers disagreed on the eligibility of five (0.4%) articles, all of which were resolved with input from the third reviewer.

Fig. 1.

PRISMA flow diagram.

4. Characteristics of included articles

4.1. Trials

Most trials were c-RTs (n = 12) [[28], [29], [30], [31], [32], [33], [34],[36], [37], [38], [39], [40]], conducted in the UK (n = 5) [[32], [33], [34],39,41], Australia (n = 4) [28,37,38,40] and the Netherlands (n = 3) [29,35,36], with one undertaken each in India [30], China [31] and the USA [43]. Participant sample sizes ranged from 133 [35] to 31, 411 [28]; the median (interquartile range) was 1861 18,063. Some only reported the number of patient admissions [29,36] or clusters [33] rather than number of participants. All articles were published between 2011 and 2020.

Included trails focused on a wide range of health problems or topics. Interventions focused on improving patient safety in the areas of pressure injury [38], falls [33], suspected ventilator associated pneumonia [41], and audit and feedback for patient safety in general [28]. Three interventions were targeted at patients with cardiovascular disease, including ischemic heart disease [29], acute coronary syndrome [31] and stroke [37]. Interventions were targeted at surgical patients undergoing emergency open abdominal surgery [32], general, orthopaedic or gynaecological surgery [34], and patients recovering from total knee replacement [30]. Three trials focused on improving care processes in the ICU (e.g. pain management, sedation and analgesia management) [36,39,40]. One intervention was targeted at patients with cancer [35] and another on patients with substance misuse on medical inpatient units [43].

5. Process evaluation objectives and aims

Process evaluation aims and objectives were broad and varied. Overall, process evaluation papers focused on examining the implementation process [15,17,[19], [20], [21],24,41], including implementation fidelity [23,25,39,42,43] and barriers and facilitators to implementation [[14], [15], [16],39]. While most studies explored responses to the intervention [15,[17], [18], [19],22,24,25]; others examined several aspects of the implementation process. For instance, participant and end-user perceptions and experiences [18,19], how the intervention was received [24,25], understood [17], accepted [15] and level of engagement with the intervention [22]. A few studies also aimed to better understand how effects of the intervention were brought about (i.e. mechanism of impact) [14,18,25].

6. Process evaluation frameworks

Twelve of the 15 included trials referenced a framework or guidance in the published process evaluation [14,15,[17], [18], [19],[22], [23], [24], [25],39,[41], [42], [43]]. The most widely cited frameworks were Grant's framework for process evaluations of c-RTs (n = 5) [14,15,19,23,40], the MRC's guidance for process evaluations of complex interventions (n = 3) [15,39,41] and Carroll's framework for implementation fidelity [22,42]. Linnan and Steckler's framework for process evaluations of public health interventions [25], Hulscher's process evaluation framework for quality improvement studies [18], the Promoting Action on Research Implementation in Health Services (PARIHS) framework [24], and the Clinical Performance Feedback Intervention Theory (CP-FIT) [14] were also cited. While most studies referenced a framework or guidance, some did not clarify how they were used [14,17,39,40]. There was also considerable variation in the number of domains authors chose to evaluate in their process evaluation methodology. Several process evaluations clearly mapped framework domains with their research questions, methods and data collection tools [19,25]; Roberts and colleagues [19] evaluated all domains of Grant's framework [19], and Tamminga and colleagues [25] described how each of the key process indicators outlined by Linnan and Steckler were defined and measured [25].

7. Process evaluation methods

Most process evaluations used qualitative methods (11/14), and all but one [22] reported using quantitative methods. The process evaluation published by Sheard and colleagues [22] identified a “qualitative process evaluation”, however the authors did use quantitative methods to evaluate fidelity in the main trial paper [33]. The process evaluation undertaken alongside the QASC Trial identified as purely quantitative [23]. Seven of the 14 trial process evaluations clearly identified as mixed methods [14,15,17,19,31,34,39].

The most common methods used were interviews (10/14) [[14], [15], [16], [17],19,22,24,40,41,43] (9/14 trials) questionnaires or surveys (10/14 trials) [15,17,18,24,25,[39], [40], [41], [42], [43]] and records (e.g. meeting minutes/records, screening logs) (8/14 trials) [14,[17], [18], [19],23,25,39,42]. Less common methods were focus groups (3/14) [18,24,39], field notes (3/14) [22,25,41], observational data (3/14) [17,19,42] and document analysis (2/14) [17,19]. Most process evaluations used more than one method, with one reporting using five different methods [17]. The extent to which methods were reported varied. Some authors reported the process evaluation and main trial outcomes together in the one publication [[39], [40], [41],43], limiting description of process evaluation methods when compared to other studies that published trial results and the process evaluation separately.

8. Integrating process evaluation and trial outcomes

Most included studies captured rich data to describe participant responses, contextual factors, and the implementation process. These data guided discussions regarding potential relationships between process evaluation measures and trial outcomes. However, few studies tested hypotheses about associations between process evaluation measures and trial outcomes (e.g. via statistical analyses) Table 1 describes the characteristics of included studies.

Table 1.

Characteristics of included articles in chronological order.

Author and year	MAIN TRIAL		PROCESS EVALUATION
	Setting and participants	Design and intervention	Aims	Framework and components	Methods
a Hellyer et al., 2020 [41]	Setting: 24 ICUs (17 hospitals in England, Scotland, and Northern Ireland) Participants: n = 210 patients admitted to ICU with suspected VAP.	Design: multicentre RCT Intervention: Biomarker-guided recommendation on Antibiotics- concentrations of IL-1β and IL-8 were rapidly determined in bronchoalveolar lavage fluid - if concentrations were below a previously validated cut-off, clinicians were advised that VAP was unlikely and to consider discontinuing antibiotics Acronym: VAPrapid-2 Trial	To understand clinical behaviours and implementation of the trial protocol.	Framework(s): Cited the MRC's guidance for process evaluations [1]. Component(s): •Compliance with the trial intervention (i.e. implementation fidelity) •Intervention quality •Attitudes to the trial (i.e. response to intervention) •Barriers or facilitators to successful trial delivery •Local factors determining recruitment (i.e. context, recruitment)	Participants: Pre-trial: n = 22 PIs Within-trial: n = 22 research nurses Late-trial: n = 9 PIs, n = 13 research nurses, n = 9 ward managers, n = 5 doctors, n = 4 lab technicians. Data collection and analysis: •Interviews •Field notes •Questionnaire
b Singh et al., 2019 [15] c Huffman et al., 2018 [30]	Setting: 63 hospitals (Kerala, India). Participants: n = 21,374 Patients admitted with IHD ischemic heart disease	Design: pragmatic, step-wedged c-RT Intervention: toolkit included: (1) a monthly audit and feedback reporting system, (2) standardised admission and discharge order checklists, (3) patient education materials related to healthy lifestyle (4) access to free online quality improvement. Acronym: ACS QUIK Trial	To present development, implementation, acceptability, sustainability, facilitators, barriers and context to understand key findings from the trial from the perspective of physicians	Framework(s): MRC's guidance for process evaluations [1]. Components: •Implementation of intervention/Implementation fidelity •Acceptability of intervention •Context •Intervening mechanism (interaction between context and underlying mechanisms to support the trial results) (i.e. mechanism of action/effect) •Facilitators and barriers •Sustainability	Participants: n = 22 physician site investigators (surveys), n = 28 physicians (interviews) from 27 hospitals Data collection and analysis: •Online surveys •Physician Interviews
a McDonall et al., 2019 [40]	Setting: 3 surgical wards (1 metro teaching hospital in Melbourne, Australia) Participants: n = 240 Surgical patients recovering from TKR	Design: cluster randomised cross-over trial Intervention: Bedside multimedia intervention (MyStay) presented in a chapter-based format combining text, sound, graphics and animation packaged for iPad presentation. Two interacting components: (1) information tailored to each day of recovery to enhance patients' understanding of the goals of recovery and their role in their own recovery, and (2) opportunity for patients to achieve their recovery goals through clinician Acronym: MIME Trial	To determine if there were any differences in patient activation between IG and CG patients and whether patient outcomes related to pain intensity may have been attributed to differences in prescribed and/or administered analgesics between groups.	Framework(s): None cited. Components: •Conduct of the trial •Difference in patient activation between IG and CG	Participants: n = 240 participants. Data collection and analysis: •Survey
b Gude et al., 2019 [14] c Roos-Blom et al., 2019 [29]	Setting: 21 ICUs, Netherlands Participants: n = 21 clusters (i.e. ICUs), n = 25, 141 patient admissions, n = 253, 530 patient shifts.	Design: pragmatic 2-armed c-RT. Intervention: A&F intervention informed by Control Theory. Key component was an online dashboard that provided detailed performance information using trend charts, indicator descriptions and patient subgroup analyses. 11 ICUs randomised to feedback 10 to feedback with toolkit Acronym: A&F Trial	To understand the mechanisms through which A&F with action implementation toolbox facilitates action planning by ICUs to increase A&F effectiveness.	Framework(s): Theoretical framework (CP-FIT) for designing, implementing, and evaluating feedback in health care. Components: •Experienced Barriers and facilitators •Feedback factors •Recipient factors •Context factors	Participants: HCPs interviews and patient data. (number unclear) Data collection: •Records/logs •Semi-structured interviews
Martino et al., 2019 [43]	Setting: 13 general medical inpatient services at one university affiliated teaching hospital, USA. Participants: n = 38 providers (physicians, physician assistants, nurse), n = 1173 patients.	Design: Type 3 hybrid effectiveness-implementation RCT [44] Intervention: Implementation strategies included (1) a continuing medical education workshop on detection of substance misuse and provision of a motivational interview; (2) workshop plus bedside supervision (apprenticeship condition); and (3) a workshop plus ability to place a medical order for an interview from a consultation-liaison service (consult condition) Acronym: Nil	To determine the effectiveness of three strategies for implementing motivational interviewing for substance misuse with general medical inpatients	Framework(s): Not reported/unclear Components: •Percentage who receive interview •Integrity of interview •Amount of change talk by patients	Participants: n = 38 providers (physicians, physician assistants, nurse), n = 1173 patients. Data collection: •Questionnaires/surveys •Interviews
b Stephens et al., 2018 [17] b Martin et al., 2017 [26] c Peden et al., 2019 [32]	Setting: 93 hospitals (surgery, anaesthesia and critical care disciplines), UK Participants: n = 15,873 patients receiving emergency open abdominal surgery.	Design: Multicentre, stepped wedge c-RT Intervention: quality-improvement programme to improve uptake of a 37-point pathway for patients undergoing emergency abdominal surgery. Focuses on 6 key QI strategies: (1) stakeholder engagement, (2) building a QI team, (3) analysing local data collected for NELA, (4) using run-charts to inform progress and feed back to colleagues, (5) segmenting patient pathway to make change more manageable, (6) use of PDSA cycles to support change process. Acronym: EPOCH Trial	To describe how the EPOCH intervention was planned, delivered and received, at both cluster and local hospital levels [17] and focus on the way the model pathway was apprehended, adopted and adapted by the teams in the participating hospitals: the local work to make the pathway work.	Framework(s): Grant's framework [45], MRC's guidance for process evaluations. Components: •Delivery to clusters •Response of clusters •Delivery at site level •Response of site level/response by QI leads •Level of engagement •Fidelity •Barrier and facilitators •Intervention implementation •Context •The way the model pathway was apprehended, adopted and adapted	Participants: n = 77 QI leads completed exit questionnaire. n = 15 face-to-face 1-day cluster activation meetings, n = 15 follow-up meetings (attendees were research nurses, theatre nurses, trainees in surgery and anaesthesia). Ethnographic study: n = 53 interviews (with pathway implementers, medical, surgical and nursing colleagues). 216 h of observation. Data collection: •Meeting logs •Online exit questionnaire •Ethnographic sub-study – interviews, •observation of site-level meetings and forums, and documentary analysis.
b Sheard et al., 2017 [22] b O'Hara et al., 2016 [27] c Lawton et al., 2017 [33]	Setting: 33 general medical wards across 5 hospitals in the UK Participants: 33 wards (i.e. clusters). Do not specify sample size.	Design: c-RT Intervention: PRASE uses two theoretically informed and validated tools to collect patient feedback about the safety of care as a means of achieving patient-centred service improvement. patient feedback is then collated and presented to each ward as part of a multidisciplinary meeting during which ward staff are supported to agree a set of ward-specific actions to address areas of patient concern. Acronym: PRASE Trial	To understand staff engagement across the 17 intervention wards (main a priori research question: ‘where does the intervention work, how and why?‘)	Framework(s): Grant's framework for c-RTs [45]. Carroll's framework for implementation fidelity [46]. Components: •Knowledge of intervention site culture •Level of engagement with the intervention (staff approaches and attitudes toward intervention) •Intervention fidelity •Feasibility and acceptability	Participants: 2 hospitals, 17 wards. Participants included study facilitators and APM leads. Data collection: •Facilitator's field notes •Analysis of taped discussions •Telephone interviews •Intervention fidelity
b Roberts et al., 2017 [19] b Roberts et al., 2017 [21] b Roberts et al., 2016 [20] c Chaboyer et al., 2016 [38]	Setting: 8 hospitals (medical and surgical wards), Australia Participants: n = 1600 adult patients at risk of pressure injury.	Design: c-RT Intervention: PIP care bundle aimed at both the individual (patient) and cluster (hospital). Included three main messages: (1) keep moving; (2) look after your skin; and (3) eat a healthy diet. Delivered to patients through a brochure, poster and DVD. Acronym: INTACT Trial	To evaluate process underpinning implementation of the intervention and explore end-users' perceptions of it to give a deeper understanding of its effects [19].	Framework(s): Grant's framework for c-RTs [45]. Components: •Recruitment of clusters •Recruitment of individuals •Delivery to clusters •Delivery to individuals •Response of clusters (understand nurses' experience with and perceptions of an intervention) •Response of individuals (explored patients' perceptions of and response to the intervention) •Maintenance •Effectiveness •Unintended consequences •Context	Participants: Between 4 and 8 formal sessions delivered to clusters (participants ranging n = 38–66 per site). n = 18 nursing staff (interviews), n = 19 patients (interviews). Data collection: •Descriptive analysis of recruited clusters •Screening log data •Descriptive analysis of intervention delivery •Semi-structured interviews •Observational data/structured observations •Literature/policy document
b Morello et al., 2017 [42] c Barker et al., 2016 [28]	Setting: 24 acute medical and surgical wards, 6 Australian hospitals Participants: n = 31, 411 patients. (46,245 admissions to 16 medical and eight surgical wards). Note: some patients admitted more than once	Design: c-RT Intervention: The 6-PACK programme included a fall risk tool and individualised use of one or more of six interventions: “falls alert” sign, supervision of patients in the bathroom, ensuring patients' walking aids are within reach, a toileting regimen, use of a low-low bed and use of a bed/chair alarm. Acronym: 6-PACk Trial	To examine the implementation fidelity - program adherence and organisational support - of the 6-PACK falls prevention program during a c-RT to assist with interpretation of trial results.	Framework(s): Carroll's framework for implementation fidelity [46] Components: •Implementation fidelity, including: •(i) Program adherence - content, frequency and duration (i.e. dose) and coverage; •(ii) organisational support - hospital and ward resources, implementation activities, staff perceptions	Participants: n = 17, 698 patients from the 12 intervention wards (n = 22, 670 admissions). n = 103, 398 daily observations and medical record audits. n = 208 ward nurses (survey) Data collection: •Structured observations at patient's bedside •Audit of patient medical records •Resource utilisation diaries •Resource implementation activity compliance log •Attendance at training sessions and network meetings •Nurse survey
a Walsh et al., 2016 [39]	Setting: 8 ICUs across 8 hospitals in Scotland Participants: Baseline period: n = 881 (data available for 847) patients (9187 care periods). Intervention period: n = 591 (data available for 577) patients (6947 care periods)	Design: c-RT Intervention: Education intervention - delivered a nine-module education package through the National Health Service provider of web-based educational materials (LearnPro NHS) to nurses working in the ICU. Covered topics relating to sedation, analgesia, agitation, sleep, and delirium management in the ICU and included inbuilt assessments. Acronym: DESIST Trial	To understand whether the interventions were implemented as planned, the barriers to implementation, and factors that worked well/less well.	Framework(s): Cited MRC's framework for process evaluations, but no explicit statement in methods to say “guided by” the framework. Components: •Changes in knowledge •Responsiveness monitoring •Fidelity of intervention (i.e. whether intervention was implemented as planned) •Barriers to implementation •Reach of intervention •Staff perceptions	Participants: n = 538 nurses completed the training. n = 394 nurses completed the re-test. Data collection: •Records •Survey/questionnaire •Focus groups
b Ranasinghe et al., 2014 [16] c Du et al., 2014 [31]	Setting: 75 rural hospitals (level 2 and 3 centres) were eligible, 5 were piloted and 70 were recruited, China Participants: n = 15,141 patients with ACS. n = 556 HCPs (surveys) (from 71/75 hospitals)	Design: c-RT Intervention: Implementation of three major generic clinical pathways (risk stratification, management of STEMI, and management of non–ST-segment–elevation myocardial infarction/unstable angina pectoris). Acronym: CPACS-2 Trial	To examine the system-level barriers to implementing clinical pathways in the dynamic healthcare environment of China.	Framework(s): Not reported/unclear. Component(s): •Barriers to implementation	Participants: n = 40 HCPs involved in their implementation of the pathway from 10 hospitals. •In-depth semi-structured interviews
b Drury et al., 2014 [23] c Middleton et al., 2011 [37]	Setting: 19 stroke units, Australia Participants: n = 1696 eligible patients with stroke.	Design: c-RT. Intervention: Intervention ASUs received treatment protocols to manage fever, hyperglycaemia, and swallowing dysfunction with multidisciplinary team building workshops to address implementation barriers. Acronym: QASC	To examine protocol adherence by measuring the proportion of patients managed according to the protocols.	Framework(s): Grant's framework for c-RTs [45]. Component(s): •Protocol adherence (proportion of patients for whom all relevant management and treatment protocols were delivered).	Participants: n = 1804 patients. Data collection: •Medical records audit
a Rycroft-Malone et al., 2012 [34] b Rycroft-Malone et al., 2013 [24]	Setting: 19 acute care hospitals offering elective surgery, UK Participants: n = 1575 pre-intervention n = 1930 post intervention. Patients undergoing elective and routine general, orthopaedic or gynaecological surgery.	Design: pragmatic c-RT Intervention: focused on reducing peri-operative fasting times. 3 arms: (i) standard dissemination of a guideline package, (ii) standard dissemination plus a web-based education package championed by opinion leaders, and (iii) standard dissemination plus a PDSA approach. Acronym: Nil reported	To determine how the implementation interventions were received within sites, whether any impacts were observed locally, and how implementation processes played out. An extension to enhance PARIHS as a conceptual framework that represents implementation is considered.	Framework(s): PARIHS framework [47] Component(s): •Evaluation framework: •Evidence •Context •Facilitation	Participants: n = 70 interviews n = 2284 patient questionnaires. Research and ward staff: n = 28 key contact interviews n = 24? (12 + 12) change agent interviews, n = 24 facilitator/opinion leader interviews 5 focus groups (n = 32 participants)) N = 1076 Learning Organisational Survey Data collection: •Interviews •Focus groups •Questionnaire •Learning Organisation Survey
b Tamminga et al., 2012 [25] c Tamminga et al., 2013 [35]	Setting: 8 departments from 6 hospitals, Netherlands Participants: n = 133 adult patients (aged 18–60 years) with cancer.	Design: multi-centre RCT Intervention: Hospital-based work support intervention: 1) ‘Meeting at the hospital’ delivering patient education and support and usual psycho-oncology care; 2) ‘Meeting with the participant's occupational physician and supervisor’; and 3) ‘Enhancing communication between treating physical and occupational physician’. Acronym: Nil reported	To perform a PE of a hospital-based work support intervention … to understand how well the intervention was delivered and received to help interpret findings related to effectiveness.	Framework(s): Linnan and Steckler's framework [48] Components: •Recruitment •Context •Reach •Intensity of the intervention delivered •Intensity of the intervention received (Exposure) •Intensity of the intervention received (Satisfaction) •Fidelity	Participants: n = 65 patients in IG n = 24 patient questionnaires (exposure). n = 45 patient questionnaires (satisfaction) n = 4 nurses •Questionnaires •Nurses' reports •Checklists
b de Vos et al., 2013 [18] c van der Veer et al., 2013 [36]	Setting: 30 ICUs, Netherlands Participants: n = 25,552 patient admissions	Design: c-RT Intervention: InFoQI program is a multifaceted intervention for the ICU setting aimed to promote the use of performance indicator data for systematic QI at ICUs. Main components of the InFoQI program included A) provision of comprehensive monthly and quarterly feedback reports, B) establishment of a local multidisciplinary QI team and C) two educational outreach visits Acronym: InFoQI	To investigate the exposure to and experiences with the intervention and explore potential explanations for why the intervention was effective or not.	Framework(s): PE framework for QI studies [49] Component(s): •Actual exposure to the InFoQI program (team members were asked to record the estimated time they invested in the various study activities.) •Experience with the InFoQI program (barriers perceived by those exposed, as well as their satisfaction with the program)	Participants: n = 43 (from 56) questionnaires completed. Number of participants in focus group not clear but invited delegates from 15 ICUs. •Questionnaires •Time investment reports •Focus group •Education outreach visits

Abbreviations: ACS QUIK, Acute Coronary Syndrome Quality Improvement in Kerala; A&F, Audit & Feedback; APM, action plan meeting; CPACS-2, Clinical Pathways for Acute Coronary Syndromes—Phase 2; CG, control group; CP-FIT, Clinical Performance Feedback Intervention Theory; c-RT, Cluster Randomised Trial; EPOCH, Enhanced Peri-Operative Care for High-risk patients; HCPs, healthcare professionals; ICU, intensive care unit; IG, intervention group; IHD, ischemic heart disease; InFoQI, Information Feedback on Quality Indicators; MIME, Multimedia Intervention for Managing patient Experience; MRC, Medical Research Council; NELA, National Emergency Laparotomy Audit; PAM, Patient Activation Measure; PDSA, Plan-Do-Study-Act; PE, process evaluation; PIP, pressure injury prevention; PRASE, Patient Reporting and Action for a Safe Environment; QASC, Quality of Acute Stroke Care; QI, Quality Improvement; TKR, total knee replacement; USA, United States of America; VAP, Ventilator-acquired pneumonia; VAPrapid-2, Ventilator-acquired pneumonia to reduce antibiotic use.

^aProcess evaluation and main paper reported in the one article.

^bProcess evaluation.

^cMain outcomes/trial.

9. Using process data to explain trial outcomes

Inductive content analysis revealed the following three themes describing how authors used process evaluation findings to interpret, explain or better understand trial outcomes: i) examining responses to the intervention from the perspective of key stakeholders such as patients, clinicians and researchers; ii) understanding the influence of context on implementation of the intervention; and iii) examining the implementation process, including adherence to study protocols and procedures. Table 2 describes these themes.

Table 2.

Qualitative themes.

Theme and categories	Description	Exemplar meaning units
Examining responses to the interventiona. Acceptance of the intervention; b. Level of engagement with the intervention; c. Perceptions of, and experiences with, the intervention	Focused on participants and staff responses to the intervention. Levels of acceptance for the intervention varied within and across studies [14,15,17,19,24,42]. Some perceived intervention components as useful and important [14,15,17,24,42], while others described them as generic, inapplicable [14], difficult to remember and unhelpful [19]. Varying levels of engagement with the intervention was also described [19,22,41].	“Patients thought the brochure was informative, simple, and concise and reported they would use it again in the future … a few patients found the brochure difficult to remember … others found it burdensome …” [21] “We were able to distinguish the intervention wards into five main ‘engagement typologies’ …” [22]
Understanding the implementation context a. Presence of leadership and support at both the individual and organisational level b.Existing organisational constraints c.Impact of organisational culture	Process data used to examine the influence of context on implementation, which was perceived as having an important bearing on intervention implementation [14,16,17,22,41,42]. Leadership and support for the trial emerged as an important facilitator [14,16,17,22,41] but was not always apparent [17,42]. Ability to implement an intervention in its entirety was influenced by its complexity or difficulty (14, 22) and organisational constraints (10, 12, 14, 20, 22, 31).	“Both units had trial champions (a PI or other senior clinician) who continuously ensured study awareness was high, and encouraged enthusiasm amongst the teams” [41] “Limited resources allocated to QI were consistently raised as a major barrier for implementation. Adequate knowledge, staff, time, funding, and administrative support were required for implementation of the CPACS-2 intervention but were lacking in many hospitals.” [16]
Examining the implementation process a. Fidelity and adaptation b.Level of intervention delivery (i.e. amount, frequency, duration) c.Number, proportion and representativeness of participants, reason for non-participation (i.e. reach)	Described trial operationalisation and implementation fidelity were describe broadly, in terms of staff training, participant recruitment and data collection procedures [19,24,42]. They also examined whether the intervention was implemented in accordance with study protocols, and to the full extent in terms of frequency, consistency and number of components (i.e. implementation fidelity) [14,15,17,19,22,24,[41], [42], [43]]. Studies described differences in implementation fidelity across study sites or clusters, between intervention components (i.e. whether some aspects were implemented more than others), and over time [14,17,22,41,42]. Some studies also found that intervention components were modified to enhance its suitability to the local context [15,22,24].	“Substantial fluctuations in program adherence over the study period and between hospitals were noted” [42] “… a number of activities had taken place to embed recommendations into local practice including changing patient information letters and leaflets, and information for staff.” [24]

10. Interrelationships between themes

We identified interrelationships between themes (Fig. 2). Synthesis of process evaluation findings revealed that participants’ and staff responses to the intervention (acceptance, level of engagement) (theme 1) were influenced by context (theme 2). Both responses to the intervention (theme 1) and the context (theme 2) inevitably influenced the implementation process and fidelity (theme 3). For example, Sheard and colleagues [22] described how varying levels of engagement with the intervention (theme 1) added to the complexity of the intervention, which may have resulted in non-standardisation of the intervention group (theme 3). Overall, findings suggest that by undertaking parallel process evaluations, authors were able to identify a range of interconnected factors relating to responsiveness, context and implementation process/fidelity, that could aid further explanation of trial outcomes.

Fig. 2.

Interrelationship between themes.

11. Discussion

To our knowledge, this review is the first to examine process evaluations of interventions tested in RCTs and c-RTs in the acute hospital setting. Previous reviews have synthesised studies reporting process evaluations but have focused on other settings and populations [[3], [4], [5], [6], [7]]. This scoping review expands on findings from those reviews and provides helpful information for researchers, clinicians, and organisations to plan process evaluations alongside RCTs or c-RTs (or implementation studies more broadly) in acute hospital settings. We have identified key research trends that may be useful, including areas requiring improvement such as the need for clarity regarding how referenced frameworks underpin the design, methods and outcomes of process evaluations, and greater consistency in design and reporting.

Synthesis of findings from included studies demonstrates the complex nature of intervention implementation in the hospital setting. We identified a range of interconnected factors relating to responses to the intervention, context and the implementation process, which authors were only able to illuminate by undertaking parallel process evaluations. Clearly, there are many factors that impact how an intervention is implemented and received, which can ultimately influence trial outcomes [1,49]. These findings share similarities with those of previous reviews undertaken in other settings [3,51] and reinforce the importance of evaluating implementation processes in the acute hospital setting.

It was surprising that we only identified 29 articles (14 trials) considering the large and increasing quantity of RCTs published every year [52]. This finding suggests greater efforts are needed to ensure researchers understand the importance of undertaking process evaluations in the acute hospital setting especially the barriers and facilitators to intervention uptake. Interestingly, the number of process evaluations that we identified is proportionally small compared to other healthcare settings [3,7,51]. Systematic reviews of process evaluations undertaken alongside RCTs testing chronic disease interventions in the primary care setting [3], implementation [7], and neurological rehabilitation [51] interventions included 69, 123 and 124 process evaluations, respectively. All included papers were published from 2011 onwards suggesting a growing awareness and acknowledgement of the importance of understanding mechanism of action to explain results, and the need for sustainability in implementation.

The small number of process evaluations published alongside hospital based RCTs may partly be a consequence of the difficulty in adapting existing frameworks to the acute hospital setting. To date, frameworks have mostly been developed for evaluations of complex interventions in the primary care and public health settings [2,15,18]. For example, the MRC guidance for evaluating complex interventions [1]; the ‘How-To Guide’ by Saunders, Evans and Joshi designed for assessing health promotion program implementation [15,18]; and the RE-AIM framework (Reach, Effectiveness, Adoption, Implementation and Maintenance), have been applied to public health and health behaviour change research [19,20]. While this does not negate these framework's applicability to a variety of health settings, this review identified limited evidence of their use in the acute hospital setting. As such, this review addresses an important research gap by providing examples of how authors have used these frameworks, as well as different methods and tools, to operationalise process evaluations in hospital settings.

Another important finding arising from this review is the considerable variability in how process evaluations were designed, conducted and reported. Firstly, the aims of included process evaluations differed. For example, the MIME trial simply aimed to compare patient activation between intervention and control group patients [40]. In contrast, the ACS QUIK trial process evaluation had a comprehensive aim to present development, implementation, acceptability, sustainability, facilitators, barriers and context to understand key findings from the trial from the perspective of physicians [15]. Consequently, the process evaluation components authors chose to evaluate also varied. Some studies examined many components [15,17,25], while others evaluated only one or two, such as the 6-PACK trial (fidelity) [42], the CPACS-2 trial (barriers to implementation) [16], the QASC trial (protocol adherence) [23] and the MIME trial (conduct of the trial and difference in patient activation) [40]. This variability was also observed in process evaluation reports, which ranged from a single paragraph describing a quantitative measure of patient activation [40] to a comprehensive mixed-methods process evaluation published in three separate articles [19].

The observed variability among studies predictably made data extraction and synthesis challenging. This trend has also been identified in previous reviews [3,51]. Clearly, there is need to establish a global definition for process evaluation, to clarify the most appropriate frameworks for researchers to use and to develop clear reporting guidelines for process evaluations (perhaps something for EQUATOR network to consider) [1,2,51]. This would facilitate evidence synthesis and transferability of interventions to other settings [1,2].

Utilising process evaluation frameworks encourages consistency [1,2]. However, authors need to clarify how each evaluation domain is mapped to a corresponding research question, and what research methods and data collection tools should be used to answer these questions. Few studies achieved this in the present scoping study [17,19,25]. In the EPOCH Trial, the evaluation results were structured in accordance with Grant's framework [17], while the INTACT Trial employed Grant's framework in its entirety, covering all evaluation domains and clearly describe how each evaluation domain was linked with specific research questions, appropriate methods, and data collection tools [19]. Using studies where frameworks are explicitly and transparently presented as exemplars can provide direction for researchers undertaking process evaluations in future. We advocate the use of an explicit statement from researchers, describing how selected framework(s) guided their process evaluation.

As evidenced by the results of qualitative content analysis, included studies hypothesised about the potential impact of processes on trial outcomes (i.e. mechanism of impact) [1]. Yet, authors rarely tested hypothesised causal pathways; a tendency that has been previously described [53]. The latter is challenging, relying on a sufficient sample size, availability of psychometrically sound measurement instruments, and appropriate researcher expertise [53]. Main trial analyses are not always designed to provide this level of detail. For example, the authors of the INTACT trial hypothesised that the ‘dose’ of the training may have been insufficient, but acknowledged that the study was insufficiently powered for exploratory analyses to test associations between dose/delivery and main trial outcomes [19]. Greater use of statistical methods to integrate process and outcome data are needed, for example, by combining quantitative process data (e.g. descriptive statistics relating fidelity, dose and reach) with outcomes datasets [1].

11.1. Strengths and limitations

We adopted the scoping review method which enabled a rigorous and transparent approach to synthesising a diverse range of research [8,9]. First, we assembled a study team with relevant clinical and methodological expertise to undertake this review [8,9], comprised of skilled healthcare researchers with experience in undertaking process evaluations of clinical trials in the hospital setting. Second, we developed clear eligibility criteria that enabled us to address a broad research question with focus and direction [9]. Finally, the data extraction form ensured a standardised approach to extracting key information; this was particularly important given the lengthy and detailed nature of included process evaluation publications, which made it challenging to review and identify key information. The iterative development process also ensured that valuable information was captured with increasing understanding of the topic.

We also acknowledge some limitations to this review. While we used a variety of terms to capture a broad range of studies (‘process evaluation’, ‘intervention implementation’, ‘process monitoring’ and ‘process assessment’), the search strategy may not be an exhaustive list of synonyms for ‘process evaluation’. However, the search strategy was carefully considered; we referred to key texts, including previous reviews on process evaluations [3,7], browsed database medical subject headings and had regular discussions among the research team. It would be beneficial to develop a list of synonyms for process evaluation studies that could be used to guide future research, such as by synthesising reviews on this research topic.

Studies were excluded at the title and abstract screening phase if they did not explicitly identify as a process evaluation, or the study did not clearly align with the MRC's definition of a process evaluation. Therefore, it is possible that articles omitting this information in the title and abstract may have been unintentionally missed. However, having two authors independently screen all articles (with a third to adjudicate) reduced the risk of potentially eligible articles being excluded, an approach that aligns with scoping review guidance [9]. While the identification of an additional 14 texts from ‘other’ sources suggests there may have been limitations to the search strategy, most studies identified through other sources were main trial papers reporting on the RCT outcomes and were found by reviewing the reference list of included process evaluations. Since we were interested in papers reporting RCTs and process evaluations, where the main paper did not describe the process evaluation, it is reasonable that this may not have been picked up in the search.

12. Conclusion

Findings from this review demonstrate the complex nature of intervention implementation in the acute hospital setting and emphasise the importance of undertaking process evaluations. Process evaluation findings enabled authors to interpret, understand and explain trial outcomes, to evaluate responses to the intervention, and to consider the implementation context. Yet, several shortcomings of included process evaluations were identified. Variation in their design and reporting made data extraction and synthesis difficult; some authors failed to clarify how referenced frameworks underpinned their process evaluation; and the relationship between process, intervention and trial outcomes was rarely tested. Finally, the small number of included trials suggests greater efforts are needed to ensure researchers understand the importance of undertaking process evaluations in acute hospital settings. Overall, these findings may be helpful to researchers planning process evaluations alongside hospital based RCTs.

CRediT author statement

Ishtar Sladdin: Conceptualization, Methodology, Formal analysis, Writing – Original Draft, Writing – Review & Editing, Visualisation. Rachel Walker: Conceptualization, Methodology, Formal analysis, Writing – Review & Editing. Sharon Latimer: Conceptualization, Methodology, Writing – Review & Editing. Wendy Chaboyer: Conceptualization, Methodology, Writing – Review & Editing. Marie Cooke: Conceptualization, Methodology, Writing – Review & Editing. Brigid Gillespie: Conceptualization, Methodology, Formal analysis, Writing – Review & Editing.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Declaration of competing interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following are the Supplementary data to this article: