Process evaluation and the development of behavioural interventions to improve psychological distress among survivors of critical illness

Depressive, anxiety and post-traumatic stress disorder (PTSD) symptoms are common, long-lasting and associated with worse quality of life among intensive care unit (ICU) survivors.^1–4 There are few clinical trials evaluating interventions to improve these symptoms. Cox and colleagues evaluated the feasibility of a mobile mindfulness training application (App) to address psychological stress after critical illness.⁵ Mindfulness promotes non-judgmental awareness of the present and acceptance of emotions, thoughts (eg, ‘I’m broken’, ‘I’m dependent’) and physiological states (eg, pain, shortness of breath), thus improving engagement with thoughts and behaviours related to illness along with coping and medical management of disease.⁶

This multisite pilot randomised controlled trial (RCT) randomised 80 participants to one of the three groups: mobile App, telephone mindfulness training or education control. Eligibility criteria included adults in the ICU ≥24 hours with cardiorespiratory failure. Participants randomised to mobile mindfulness intervention interacted with the App over 4 weekly mindfulness training sessions. Participants in the telephone group completed 4 weekly telephone mindfulness sessions (~30 min each) with a trained psychologist. The education control group received two telephone calls and a self-directed web-based programme focused on the nature and treatment of critical illness. Primary outcomes included feasibility, acceptability and usability. Secondary outcomes included depressive, anxiety and PTSD symptoms.

Approximately 83% (n=66) of patients completed the study. Loss to follow-up was 29%, 10% and 11% in the mobile mindfulness, telephone mindfulness and education groups, respectively. The percentage of participants who were not lost to follow-up completing all intervention sessions was ≥92% in each group. The results demonstrated ‘strong’ acceptability and usability with qualitative feedback considered overall ‘positive’ regarding the App. At 3-month follow-up, preliminary data demonstrated no significant improvement from baseline (before hospital discharge) in anxiety and depressive symptoms in the education control group while there was a trend toward improvements in these symptoms in the mobile and telephone mindfulness groups. There was no such trend for PTSD symptoms. In the mobile mindfulness group, the frequency and duration of App inter action correlated with improvement in depressive symptoms.

The success of this study is attributable, in part, to the authors’ extensive prior work in the field of postdischarge interventions to improve psychological stress symptoms in ICU survivors (table 1).^7–9 While not explicitly described, they incorporated aspects of a process evaluation when refining the complex behavioural intervention (ie, mobile mindfulness training). Process evaluation allows for better understanding of an intervention to allow for adaptation, more effective implementation or application to a different population and is necessary for each phase of intervention assessment (ie, pilot feasibility, effectiveness and postevaluation implementation).¹⁰ In 2015, the UK Medical Research Council published guidelines for the process evaluation of complex interventions, with a focus on four key components: (1) theoretical framework and causal hypothesis, (2) implementation, (3) mechanisms of impact and 4) context.¹¹ Process evaluation is often overlooked but is especially important in studying complex interventions in ICU survivors given the population’s heterogeneity in illness severity, comorbidities, exposure to ICU therapies and experiences of recovery. In the following paragraphs, we will discuss these four key components of process evaluation in the context of this study.

Table 1.

Summary of key components of process evaluation highlighted by each study conducted by the authors

	Qualitative study of critical illness survivors 200912	Telephone CST 20129	Telephone mindfulness training RCT 20148	Telephone and web-based CST vs education control RCT 20187	Mobile mindfulness vs telephone mindfulness vs education control RCT 20185
Implementation
’What’	N/A	CST	Mindfulness training	CST vs education control	Mindfulness training vs education control
’How’		Telephone delivery: acceptable and feasible	Telephone delivery: acceptable and feasible	Telephone and web-based	Telephone vs mobile App
Dose		6 weekly sessions	6 weekly sessions	6 weekly sessions	4 weekly sessions
Interventionist		Critical care nurse	Clinical psychologist and MD	Clinical psychologist	Clinical psychologist
Fidelity Monitoring		Not described	Not described	Not described	Audio recording (telephone) and technology-based tracking (App)
Adherence/ Uptake:		7/10 enrolled patients completed all study procedures 7/7 starting intervention completed it 92% (77/84) possible sessions completed	6/11 patients enrolled completed all study procedures Proportion of sessions completed not reported	62/86 (72%) and 69/89 (78%) patients randomised to CST and education control, respectively, completed all study procedures 54/86 (63%) and 58/89 (65%) of CST and education control patients, respectively, completed >1 session	22/31 (71%), 28/31 (90%) & 16/18 (89%) randomised to mobile mindfulness, telephone mindfulness and education control completed all study procedures, respectively 93/96 (97%) possible weekly app sessions were completed
Other		Able to incorporate caregivers	Opportunity to improve access to intervention (perhaps using technology)	Most common reason for low adherence was ‘medical illness’. Loss to follow-up associated with low education and family support; greater financial stress & baseline psychological distress; discharge to inpatient rehabilitation or skilled nursing facility	Self-paced might have improved adherence to App Patients requested more interactive features and enhanced visualisation of progress over time
Mechanisms of impact	N/A	Unclear if there is a differential response based on baseline symptoms and coping skills	Increases in mindfulness qualities were associated with improvements in psychological distress	Greater improvement in self-efficacy among those with higher baseline HADS scores	Increased frequency, duration and quality of App use associated with greater decrease in depression symptoms
Context	ICU survivors commonly experience psychological distress Few evidence-based interventions to improve psychological distress in ICU survivors	Ineffective coping is associated with increased psychological stress	Mindfulness training, a behavioural intervention, is beneficial in improving psychological distress in other populations	Web-based content might enhance the effects of telephone-delivered CST Patients with high baseline distress might be the most likely to benefit	There are potential barriers to delivering an intervention by phone. A mobile App might overcome such barriers

App, application; CST, coping skills training; HADS, Hospital Anxiety and Depression Scale; ICU, intensive care unit; MD, doctor of medicine; N/A, not applicable; RCT, randomised controlled trial.

THEORETICAL FRAMEWORK

Process evaluation is founded in a theoretical framework, often based on hypothesis-generating qualitative research in the population of interest.¹¹ In this case, the authors developed the theoretical framework for a mindfulness training intervention in ICU survivors (see figure 1 for our interpretation) using an initial qualitative study (n=23 ICU survivors and caregivers) demonstrating that symptoms of anxiety and depression are common among ICU survivors, who also reported difficulty coping with their illness.¹² In a subsequent pilot study aimed at developing and evaluating a coping skills training programme in ICU survivors, the authors identified several mindfulness-related skills that were highly valued but used infrequently.⁹

Figure 1.

Conceptual framework for mindfulness training in survivors of critical illness. ICU, intensive care unit; PTSD, post-traumatic stress disorder.

IMPLEMENTATION

Both quantitative and qualitative measures are used to examine the next aspect of process evaluation: implementation, or the ‘what’ and ‘how’ of intervention delivery.^11,13 The authors conducted a prior pilot study (n=8 ICU survivors and caregivers) demonstrating that a telephone mindfulness programme was acceptable and feasible. This study identified several unresolved implementation issues: dose (number and duration of sessions); access to the intervention (perhaps via more advanced technology) and patient preference for treatment modality (eg, self-management vs therapist-based).⁸ While each of these issues was considered when designing the current RCT (table 1), several issues remain unsolved, including the optimal dose of mindfulness sessions (frequency and duration). Studies of mindfulness training (eg, Mindfulness Based Stress Reduction) in other populations suggest that 6–12 sessions are sufficient to understand and gain mastery of mindfulness skills.¹⁴

Next, the investigators conducted an RCT (n=175) evaluating the effects of a telephone-based and web-based coping skills training intervention versus an education control on depressive symptoms in ICU survivors.⁷ When this RCT did not demonstrate significant reductions in depressive symptoms, the authors noted that adherence to the protocols was low in both groups, with ‘medical illness’ being the most common explanation.⁷ Hence, by considering this aspect of implementation, that is, adherence or uptake, the authors refined the eligibility criteria for the current RCT.⁷ While adherence was improved in the current RCT, twice as many participants randomised to the mobile versus telephone mindfulness group or education control were lost to follow-up. The majority were lost to follow-up before initiating the App. Such differential attrition might lead to bias if participants lost to follow-up versus those completing the study were more likely to rate the App as having low acceptability or usability or were more likely to have higher, more persistent psychological stress symptoms. Including use of a rating scale, such as the Mobile Application Rating Scale (MARS), in the process evaluation would provide a greater understanding of App quality, a factor that may have impacted participant engagement.¹⁵

Two additional aspects of implementation are intervention training and fidelity monitoring. Understanding ‘what’ is delivered requires a detailed description of the credentials, background and training provided for the interventionist, as done via publication of the protocol for the current RCT.⁵ The authors conducted fidelity monitoring using audio recording (telephone group) and electronic reporting (mobile group) of treatment sessions to ensure the intervention was being delivered as intended.¹¹ Intervention training and fidelity monitoring are especially important aspects of implementation in multisite studies, given increased potential for variability.¹⁶

MECHANISMS OF IMPACT

Another important aspect of process evaluation is a mechanism of impact, that is, a description of how the participants interact with the intervention, mediators and potential unexpected pathways. Mindfulness training is an effective intervention to reduce psychological distress in populations with chronic medical problems,⁶ though its mechanisms are poorly understood. Importantly, feasibility studies, such as the current study,⁵ are not powered to perform an analysis of mediators. Nonetheless, a crucial aspect of process evaluation is a comprehensive understanding of intervention theory that allows for targeted exploration of causal mechanisms using quantitative, when possible, and qualitative techniques.¹⁰ By failing to understand why an intervention is successful, investigators run the risk of drawing incorrect conclusions regarding causal pathways and overlooking key intervention components or mediators.¹⁰ Importantly, ICU survivors experience impairments in physical, cognitive and mental health, with prior studies demonstrating the close interplay between physical and mental health impairments.^17,18 Hence, it is important to understand how a complex intervention affecting one of these outcomes (eg, depressive symptoms) might impact others (eg, physical function), as well as the role of mediators (eg, coping skills) that may affect the likelihood of treatment uptake and efficacy. A qualitative analysis of potential mediators could strengthen insights gained from the current study.

CONTEXT

Process evaluation promotes an understanding of the context in which participants interact with complex interventions. Context includes demographics, socioeconomic factors, cultural systems and beliefs, and characteristics inherent to a given medical condition.¹⁰ A thorough understanding of the intervention’s acceptance, reach and effectiveness among subgroups of ICU survivors is key to informing future RCTs and broader application of the intervention. The coping skills training RCT⁷ demonstrated that patients with higher baseline distress were more likely to benefit from an intervention aimed at improving coping skills and self-efficacy, thus highlighting the need for studies of future interventions to target subgroups of ICU patients most likely to benefit.⁷

ONGOING WORK AND FUTURE STEPS

The extended body of work leading to the current RCT is unique within the field of postdischarge interventions to improve psychological stress among ICU survivors. The ICU diary, which is associated with reduced psychological stress in this population, is similarly a complex intervention.^19,20 Such complex behavioural interventions show promise in improving the multifaceted long-term sequelae of critical illness. By involving behavioural health experts in the design of such interventions and incorporating formal process evaluations, future studies can describe key components of behavioural interventions and their generalisability.

In summary, the current RCT of a mobile mindfulness training intervention provides important insights into the implementation of a complex behavioural intervention aimed at improving psychological stress among ICU survivors. Future research in this area should include further evaluation, using a larger sample size, of the optimal dose of mindfulness training; potential mediators of the association between mindfulness training and improved psychological stress symptoms; and the quality of future mobile apps.