Skip to main content
NCBI home page
Elsevier - PMC COVID-19 Collection logoLink to Elsevier - PMC COVID-19 Collection
. 2022 Aug 8;25:136–144. doi: 10.1016/j.jcbs.2022.07.005

Scalability of an ACT-Based strategy for improving well-being in health care providers: A mix-method and preliminary evaluation of efficacy

Amanda M Muñoz-Martínez a,∗, Clara S Otto-Scheiber b, Shadia Zuluaga-Jassir a, Angie Medina-Díaz a, Jessica Pulido-Rozo a, Margarita Venegas-Ramírez a, Alexander Igua-Jojoa a
PMCID: PMC9359768  PMID: 35966007

Abstract

The burden of the COVID-19 pandemic has been mainly carried by health care providers. Technology-Mediated Interventions (TMI) seem to be a feasible alternative to increase access to behavioral health resources in this population. However, scaling-up treatments into TMI requires developing user-friendly, accepted, and accessible formats. A two-stage study was conducted to assess scalability of an Acceptance and Commitment Therapy (ACT) based strategy (named FACE COVID) delivered using technology. First, a mix-method design connected qualitative and quantitative data from health providers and ACT experts by which changes were performed to enhance scalability. Second, a pretest-posttest study was conducted to preliminary evaluate the efficacy of FACE COVID intervention on well-being, psychological distress, and psychological flexibility. Results showed a positive impact on well-being, but not distress and psychological flexibility. While this intervention has promising results, changes in dose intensity, social support, and mental health literacy could improve retention as well as increase opportunities to target distress and psychological flexibility in future studies.

Keywords: Acceptance and commitment therapy, Well-being, Health care providers, Scalability, Technology-mediated interventions, Mix-methods

Before the COVID-19 (SARS-CoV-2) pandemic, the prevalence of burnout among health care providers ranged from 21% to 67% (Morse et al., 2012). A study found that approximately 60% of physicians experience at least one episode of burnout in their lives (Burton et al., 2016). This problem is associated with severe psychological problems such as depression, self-injury, and suicide attempts in health care providers (Hakanen et al., 2008).

Health crises, such as pandemics, presented several challenges for health care providers such as facing the death of colleagues, experiencing loss of control, feeling vulnerable, working excessively, fearing for their lives, and so forth (Maunder et al., 2006). In addition, the pandemic has exacerbated behavioral health problems such as high levels of distress, anger, fear, insomnia (Kang et al., 2020), and suicide, which has steeply increased (Jung & Jun, 2020).

Although over the years (not only during the COVID-19 pandemic) a considerable number of programs have been designed to aid health care providers in coping with burnout and other behavioral problems, they face barriers such as stigma and time constraints that hinder accessibility (Chen et al., 2020; Johnson et al., 2018; Sastre-Rus., 2019). Mental health stigma, the belief that engaging in psychological services undermines their professional roles, careers, and prestige is one of the main barriers that health care providers face. This is exacerbated by fear of being seen by a colleague from their workplace or inner circle (Johnson et al., 2018). Even during the pandemic of COVID-19 when they have been highly vulnerable, stigma has limited their acceptance of psychological help. In a study conducted in China, healthcare professionals refused to participate in individual, phone, or group interventions for improving their well-being for reasons just mentioned (Chen et al., 2020). Time constraints have also hindered accessibility to behavioral health services, considering they have long shifts that limit their availability to complete other activities (Burton et al., 2016).

Implementation science has brought attention to the importance of improving the ecological validity of empirically based interventions. Studies on scalability are fundamental to enhancing usability and defeating barriers usually observed in non-controlled contexts and low motivated populations (Bauer et al., 2015). Scaling-up is defined as “the ability of a health intervention […] to be expanded under real-world conditions to reach a greater proportion of the eligible population while retaining effectiveness” (Milat et al., 2013). Although some practical tools for measuring scalability have been developed, these guidelines still need to be evaluated and adapted to different types of contexts, populations, interventions, and disciplines (Milat et al., 2020). To our knowledge, none of these scales has been standardized or psychometrically validated. According to Milat et al. (2020), scalability comprises the following five domains: (a) fidelity and adaptation which involves interventions' susceptibility to being adapted while keeping consistent with their core components, (b) acceptability and reach which refer to adapted treatment ability to connect and target the intended population, (c) delivery setting and workforce that imply context capacity to deliver an intervention, (d) implementation infrastructure which includes potential setup needed to deliver interventions (e.g., organizational structure), and (e) sustainability that envisions long-term outcomes of the scale-up.

A special emphasis has been made on acceptability as a key aspect of implementation science. Acceptability is comprised of seven components (a) affective attitude, individuals' feelings towards an intervention (b) burden, perceptions on the effort to participate in a treatment (c) perceived effectiveness, odds of achieving treatment purpose (d) ethics, interventions fitting with people's value system (e) intervention coherence, capacity to effectively communicate how an intervention works (f) costs, balance on benefits profits or values within a program, and (g) self-efficacy, participants' confidence on their ability to complete intervention as intended (Sekhon et al., 2017). Acceptability might be crucial to reducing stigma, ensuring that users perceive interventions as responsive to their emotional, time, cognitive, and structural needs.

As the present study seeks to assess ACT strategies scaled into Technology-Mediated Interventions (TMI) without looking for a systemic or organizational change in a long-term track, we developed a survey to assess four elements of scalability that are crucial for TMIs, fidelity, reach, delivery, and acceptability.

Scaled interventions include various ways of delivery adaptations, for instance, non-specialists’ involvement, self-help tools, or mobile applications. TMIs are one of the most promising tools for scaling intervention to the general public as they require fewer human resources and mainly rely on active and passive technology (Linn et al., 2011; Mistry et al., 2015). A systematic review of nine studies that implemented TMIs (e.g., videos, website access) to enhance mental health literacy found that this modality produces a reduction in stigma towards behavioral problems (Tay et al., 2018). Whealin et al. (2021) delivered cognitive-behavioral tools such as mindfulness and problem-solving via text messages to veterans presenting COVID-19-related distress that evaluated it as helpful and educative. Safieh et al. (2021) reviewed studies that optimized mental health strategies during the COVID-19 pandemic. In their analysis of TMIs, they found that three out of four studies that provided mental health assistance showed improvements in participants’ quality of life and distress. Within this context, scaling empirical-based intervention from usual methods of delivery (e.g., one-on-one therapeutic sessions) to TMI could be useful to overcome psychological, social, and structural barriers faced by healthcare care providers.

1. Scaling up Acceptance and Commitment Therapy for health care providers

Acceptance and Commitment Therapy (ACT) is a contextual behavioral intervention that has demonstrated strong effects on a diverse array of behavioral problems as listed by the Society of Clinical Psychology (2021). ACT focuses on promoting psychological flexibility “as the process of contacting the present moment fully as a conscious human being and persisting or changing behavior in the service of chosen values” (Hayes et al., 2006, p. 9). This intervention seeks to disrupt the influence of rigid verbal rules (verbal contexts) with the ultimate goal of aiding people to reach a self-chosen valuable life (Hayes et al., 2013). For this endeavor, ACT targets the following six components of psychological flexibility: acceptance, defusion, present moment, self-as-context, values, and commitment actions.

In recent years, ACT studies have shown positive results on non-syndromic targets such as well-being (i.e., life satisfaction; Wahyun et al., 2019) and distress (Hayes et al., 2006; Räsänen et al., 2016). Stenhoff et al. (2020) meta-analysis found positive effects of ACT on various transdiagnostic outcomes, a set of different problem presentations that share the same causal pathway and produce similar outcomes (Cicchetti & Rogosch, 1996), such as subjective well-being, emotional regulation, and interpersonal functioning. Another meta-analysis evaluating ACT effects on burnout and psychological distress did not find effects on burnout; however, people who reported higher levels of total distress reported a significant reduction in these after ACT implementation (Reeve et al., 2018). A recent meta-analysis by Prudenzi et al. (2021) analyzed the pooled size effect of ACT on health professionals’ distress, finding a small effect on discomfort after the intervention. These promising results have encouraged the dissemination of ACT procedures and principles to public health scenarios.

Some research on ACT and burnout has explored the explanatory processes underlying the impact of this intervention on distress and burnout. Lloyd et al. (2013) found that increases in psychological flexibility mediated the relationship between ACT and a reduction in emotional exhaustion. Other research has shown that values-driven behaviors play a relevant role in helping professionals to cope with stress and burnout (Berkout, 2022; Vilardaga et al., 2011). In fact, research shows that ACT promotion of engagement in value-driven behaviors is a useful alternative to reduce experiential avoidance as they improve health professionals’ well-being and general functioning (Wersebe et al., 2018). Theoretically, it is likely that the transformation of functions, defined as the “untrained acquisition of stimulus functions among members of stimulus equivalence classes or relational frames” (Dougher et al., 2002, p. 63), alters the relational context between emotional and cognitive unpleasant experiences and engagement in values. Thus, health care provides learn that unpleasant private experiences are not opposite to meaning and values, but they are part of engaging in meaningful actions. It is possible that this process facilitates psychological flexibility and reduces burnout and distress.

OSF recent example of ACT dissemination is the FACE COVID acrostic designed by Russ Harris (2020). This acrostic intended to offer ACT-based strategies to the general public, aiming to help them to face distress related to COVID-19. FACE COVID stands for F=Focus on what's in your control, A = Acknowledge your thoughts & feelings, C=Come back into your body, E = Engage in what you're doing, C=Committed action, OSF=Opening up, V= Values, I=Identify resources, and D = Disinfect & distance. Each letter on FACE COVID alluded to recommendations based on the six components of the psychological flexibility model plus particular health-promoting behaviors associated with reducing contagion. A few months after its publication, the World Health Organization (WHO, 2020) rapidly recommended its use in Australia by posting an illustrated guide, translated into more than 15 languages, that explained concepts underlying FACE COVID and included practical tools such as exercises (e.g., audios) and cartoons. Despite FACE COVID is grounded on an empirical-based intervention, its rapid adoption left open questions related to its scalability and usability. In addition, it is not clear to what extent this strategy reaches users' needs and what could be their potential in the long run.

In order to gather preliminary information on the implementation of a scaled ACT-based strategy for health care providers by using the FACE COVID acrostic, a two-stage study. First, mix-method research was conducted to assess the scalability and acceptability of FACE COVID as a TMI. Second, a preliminary evaluation of study efficacy on distress and life satisfaction was evaluated.

2. Method

In the first stage of this study, the scalability of FACE COVID as a technology-mediated strategy was assessed. The mix-method approach utilized in this stage allowed us to understand in-deep whether the FACE COVID TMI fitted health care professional language, needs, and perspective regarding distress and well-being, as well as its fidelity to the ACT approach. It aimed to produce a context- and user-sensitive intervention. In the second stage, a quasi-experimental pretest-posttest design was conducted to assess its preliminary efficacy on psychological distress and well-being.

2.1. Procedure and participants

Upon receiving approval from the local Institutional Review Board (IRB) at the University of Los Andes and the Fundación Santa Fe de Bogotá University Hospital, an explanatory mixed-method study with a sequential approach from quantitative to qualitative data was conducted to assess the scalability and acceptability of FACE COVID strategy in the first stage. First, an adaptation of ACT strategies to FACE COVID acrostic for health care providers was completed (Appendix A). Second, scalability and acceptability were assessed within six-online focus groups with health care providers and one meeting with two ACT experts.

Adaptation of ACT Strategies within FACE COVID Acrostic. The first and second authors, who have received direct supervision and training in ACT, adapted several ACT metaphors, analogies, and exercises to a technology-based format (e.g., videos). Ten video screenplays based on ACT key components for health care professionals facing the COVID-19 pandemic were designed. Some of the tools included in this adaption were the lighthouse metaphor, dropping anchor exercises, the sailing boat metaphor (Stoddard & Afari, 2014, the sky and the weather analogy (Harris, 2009, p. 175), the body Scan exercise (Walser & Westrup, 2007), among others (see Appendix A).

In addition, four audio scripts and five infographics/cartoons were developed as complementary material that was sent between each video. FACE COVID acrostic was adapted for health care providers (see Appendix A for a detailed description). Powtoon application was utilized to produce FACE COVID videos and complementary activities (e.g., infographics) were designed in Canva®.

In the focus groups with health care professionals, they evaluated FACE COVID videos that contained the main active components of ACT while ACT experts reviewed FACE COVID's main and complementary activities.

Scalability and Acceptability Assessment. Twenty health care providers were recruited by public advertisement and snowballing methods at a private hospital in Bogotá, Colombia. Flyers had information on the study's objectives, compensation (US$13 groceries coupons) for participating, and a Qualtrics link for completing informed consent according to APA and IRB local standards. Health care professionals who saw patients (at least six months) during the COVID-19 pandemic, one-on-one, online, or in-site services, were included in the study, while the administrative staff were excluded. Participants reported different professions, physiotherapists (75%), occupational therapists (20%), and physical educators (5%). They reported an average professional experience of 12-years, and their time working during the COVID-19 pandemic ranged between six to eight months. Most participants were women (90%), and the mean age was 35.95.

Focus groups conducted with health care providers were 90 min long and were led by two members of the research team. The number of participants varied between two to five per group according to their time availability. Sessions started with a description of participants' tasks and session structure. Later, a FACE COVID video was played to participants (a maximum of three videos were presented by the end of each session). Upon the video conclusion, facilitators sent a link to participants in which they rated to what degree they believed scalability (reach and distribution) was achieved. Finally, facilitators used Delphi questions with the group to understand their quantitative reports and their perception of the videos’ scalability. Towards the end of the focus groups, participants quantitatively assessed video acceptability.

An additional evaluation of scalability, particularly, fidelity was conducted by two Colombian ACT experts. One of them was a doctor in behavior analysis that had more than 10-years of experience in conducting research and training in contextual behavioral therapies, particularly, ACT. The other expert was a master's in clinical psychology with more than 15-years of clinical and research experience applying third-wave therapies within healthcare settings and populations presenting health-related issues. They were both familiar with particular dynamics embedded within Colombian's culture and health system.

In order to evaluate scalability, they first reviewed and provided written quantitative and qualitative feedback on FACE COVID videos and their complementary activities (e.g., checklists, infographics, cartoons). Second, a 2-h meeting was conducted to obtain further feedback and a deeper understanding regarding ACT adaptation for TMI implementation of health care providers in Colombia.

Based on the analysis of the first stage data, The original FACE COVID adaptation was modified (Appendix A). After adjusting the main and complementary activities of the program, the second stage of this study was conducted.

Preliminary Efficacy Evaluation. Eighteen health care providers and medical students were recruited by public advertisement and snowball methods using flyers and presentations in classes and team meetings. Participants had provided online or in-site health services during the COVID-19 pandemic and were interested in modifying their levels of well-being and distress. All participants had a smartphone with daily internet access. Those who reported frequent or very frequent substance use and/or risk behaviors towards themselves (e.g., self-injury) or others (e.g., aggression) were excluded.

Only twelve participants were eligible and granted informed consent following APA and local IRB standards. Participants ranged between 52- and 22-years-old (MAge = 37) and most participants were heterosexual men. In addition, there was a wide variety of health care professionals in the study (Table 1 ). Social support varied across participants, 25% of the participants lived alone and only 8% of them reported not having a close friendship. None of the participants were receiving one-on-one or online mental health services.

Table 1.

Sociodemographic characteristics of participants at the baseline.

Baseline characteristic Full sample
n (12) %
Sex
Male 8 67
Female 4 33
Sexual orientation
Heterosexual 11 92
Bisexual 1 8
Nationality
Colombian 11 92
Venezuelan 1 8
City of residence
Bogotá 12 100
Education level
University studies, without graduating 6 50
Undergraduate 1 8
Postgraduate 5 42
Profession
Physical educator 1 8
Nurse 1 8
Physiotherapist 3 25
Internist 3 25
Occupational therapist 1 8
Medical students 3 25
Range of hours worked in the last week
>11 2 17
12–23 1 8
24–48 3 25
49–59 4 33
60 < 2 17
Monthly Wage
<1 MMW 6 50
2–3 MMW 3 25
3–4 MMW 3 25
Medical condition
Chronic physical condition 1 8
Any other disability 1 8
None of the above 10 83
MMW = Minimum Monthly Wage
Open in a new tab

Regarding perceptions of risk associated with COVID-19, 33% of the participants reported low-medium fear to have health complications if infected, 100% have had someone close infected, and 25% reported that they had COVID-19 at some point during the pandemic. About vaccination, 100% were fully vaccinated.

After completing pretest questionnaires, participants were contacted by a research assistant who provided them with a video tutorial and a manual that contained instructions for using the app (Expiwell ©), a mobile application through which intervention was delivered. Twelve participants used a code to sign up in Expiwell, three people enrolled but did not complete videos or activities during treatment (Appendix D). During two weeks participants used Expiwell to access FACE COVID strategies sent daily at 4:59 a.m. Interventions were available for 24hrs, to control dose confounding, and participants had access to them at any time of the day. Three questions on daily life satisfaction and values-driven behaviors were also sent at 8:00 p.m. One week after ending the intervention, nine participants completed the posttest questionaries. Access to the whole intervention was available for participants for one month after ending data collection in case they wanted to take or replay any of the modules in the program.

2.2. Instrument and materials

For assessing scalability, two surveys were designed to quantitatively measure to what degree FACE COVID videos met scalability and acceptability (Appendix B). Items were rated on a 4-point Likert scale from 1 (totally disagree) to 4 (totally agree).

A semi-structured interview was designed to assess health care providers' perceptions and attitudes towards FACE COVID videos, this method allowed getting into particular themes to achieve a better understanding of the phenomena. This interview contained 19-Delphi questions which are a set of questions designed to guide a discussion with a panel of experts. These are review and rethink based on other experts’ answers in the same meeting. Delphi scalability questions were embedded in two oriented categories (a) reach and (b) distribution (Appendix C).

Two instruments to measure treatment efficacy were administered in stage two. The Temporal Satisfaction with Life Scale (TSLS; Galiana et al., 2015) is a measure of general life satisfaction with past, present, and future life. The TSLS reflects people's well-being fluctuation over time. The TSLS is a 15-items on a 5-point Likert scale from 1 (not at all) to 5 (a lot). The validated Spain-version has adequate construct validity and good levels of reliability according to Cronbach's alpha for satisfaction with past life (α = 0.83), present life (α = 0.81), and future life (α = 0.86). In Colombia, a study performed with university employees (M = 17.56; SD = 4.20) evaluated TSLS-present subscale construct validity and reliability, finding a consistent factorial structure and good reliability according to Cronbach's alpha (α = 0.88; Naismith et al., under review). Only TSLS-present was used for this study data analysis.

The Depression Anxiety Stress Scale (DASS-21; Ruiz et al., 2017) is a measure of depression, anxiety, and stress that offers a global index of psychological distress. The DASS is a 21-items on a 4-point Likert scale from 0 (Did not apply to me at all) to 3 (Applied to me very much or most of the time). The Colombian version has adequate construct validity and good reliability according to Cronbach's alpha in depression (α = 0.88), anxiety (α = 0.83), stress (α = 0.83), and psychological distress (a = 0.93) scales. The mean of psychological distress was 18.03 (SD = 12.39) in the sample validated.

The Comprehensive Assessment of Acceptance and Commitment Therapy Processes (CompACT; Francis et al., 2016) was a process measure also administered at pretest and posttest. A back-and-forward translation of the original validation performed with population in the UK was performed for this study. The CompACT is a 23-items in 7-point Likert scale from 0 (strongly disagree) to 6 (strongly agree) that evaluates individuals’ psychological flexibility with higher scores indicating greater psychological flexibility. The instrument has shown adequate validity and reliability across factors (inter-item correlation 0.34). The UK sample mean was 86 (SD = 20.79; Trindade et al., 2021).

2.3. Data analysis

A connecting strategy that entailed quantitative and qualitative data was performed. First, a descriptive analysis of scalability and acceptability surveys was conducted using Microsoft Excel®.

A content analysis following the constant comparison analysis (CCA), a Word Count, and Keyword in Context (WC; KWiC) were conducted using in NVivo 12. The CCA allowed to gather participants’ narratives into central themes following these three steps: (a) open coding, narratives from the main text were analyzed and attached to a descriptor (code), (b) axial coding, codes were grouped within similar categories, and (c) integrative coding, axial codes were gathered into main themes or theory that needed to achieve a saturation (Leech & Onwuegbuzie, 2008; Strauss & Corbin, 1998).

The WC identified words often used by participants when describing intervention scalability. This method was entailed with a KWiC that provided an analysis of the culture of words, giving context to participants' narratives. The conjunction of these methods allowed analysts to understand participants' speech meaning (Leech & Onwuegbuzie, 2008). Finally, quantitative and qualitative analyses were connected in an integrative narrative. This provided an understanding of FACE COVID strategies' scalability as well as strengths and areas to be changed to improve the intervention.

Descriptive analyses were performed using Excel Microsoft®. Pretest-posttest data were collected using Qualtrics and RedCap platforms. Retention rate was calculated by dividing the number of people that stayed the entire treatment by the number of participants who started on day one. As this was a preliminary efficacy study for FACE COVID intervention, no larger or medium effects were expected. To explore potential changes in treatment outcomes, a comparison of each participant's mean to instrument standard deviation (1SD) was performed. This procedure allowed identifying whether the intervention moved participants' scores from their initial trend when no larger effects are expected.

3. Results

Results will be presented in two sections (a) scalability mix-methods analyses, and (b) preliminary efficacy pretest-posttest assessment.

Scalability Mix-Methods Analyses. Descriptive analysis of health care professional surveys on scalability indicated a general agreement of FACE COVID's ability to connect with their needs and characteristics (MReach = 3.26; SDReach = 0.87), as well as being easy to access and engage in the adapted format of the intervention (MDistribution = 3.31; SDDistribution = 0.79). When rating acceptability, health care professionals also agreed that videos were appropriate for their community (MAcceptability = 3.30; SDAcceptability = 0.80). ACT experts agreed FACE COVID main and complementary strategies achieved good reach (MReach = 3.55, SDReach = 0.57) and distribution (MDistribution = 3.93, SDDistribution = 0.27). Additionally, they agreed that FACE COVID program was consistent with core components of ACT (MFidelity = 3.68; SDFidelity = 0.47).

The constant comparison analysis (CCA) analysis is displayed in a tree diagram. Citations (references) and the number of code relations within a category (density) are in a bracket (i.e., {0,4}), the first number represents references and the second density. Scalability was the main family that gathered codes from the analysis. Reach, delivery, and fidelity build-up axial categories. Delivery was the densest category as seven codes were included within it while reach embedded six codes and fidelity only two. Participants provided recommendations to improve FACE COVID strategies in all scalability areas but also highlighted how videos were novel, attractive, engaging, and relating. On delivery, most references pointed to recommendations and format adjustments, as well as how striking and user-friendly the treatment format was (videos; Fig. 1 ). Participants' suggestions on reach were focused on video content, suggesting the inclusion of more examples related to their work and daily activities. In addition, they positively evaluated FACE COVID pragmatics. Finally, fidelity narratives recommended improving the comprehensiveness of ACT components, specifically, psychological flexibility. They suggested intertwining the ACT processes across the 10-intervention videos so that the target population would be able to flexibly address distress by implementing ACT strategies.

Fig. 1.

Fig. 1

Open in a new tab

Scalability family tree.

WC-KWiC was performed for each axial code from the ACC analysis to deeply understand participants’ perspectives and recommendations relative to FACE COVID (Fig. 2 ). When analyzing the delivery category, videos were the most frequent word due to ACT strategies being adapted to this format. Improving videos prosody (e.g., pauses, duration, pitch, etc.), avatars clothing (e.g., uniforms, gowns), activities (e.g., attending patients), and settings (e.g., procedures room, hospital restrooms, etc.) were suggestions found in the KWiC analysis. Participants highlighted how videos made them think about their experiences, feelings, and values through the exercises and examples presented.

Fig. 2.

Fig. 2

Open in a new tab

Word clouds illustrate word count analysis.

Stress and feeling had the highest rates of occurrence in the reach category. Word cloud also shows that COVID, patients, helping, and work were often mentioned by participants, which implies that videos targeted topics and contexts of interest for this study. However, they disagreed with “F = focus on what is under your control” content. For instance, one participant stated “when [the video] saying “focus" …, I thought “it is obvious that one already does it”, and it sounds annoying to me.” Another participant mentioned, “coming into the area [COVID rooms] is already stressful to be pointed to “be focused” … Well, it is not necessarily bad, but it feels a little aggressive.” Similarly, ACT experts noted this module could be misinterpreted by health providers and recommended shifting content towards an invitation to “be open” while exploring value directions.

Fidelity's most frequent words were importance and values. Participants reported that videos led them to understand the relevance of engaging in values to improve their well-being. However, they suggested including an explicit definition of values earlier in the program. They also asked to include more examples of value-oriented behaviors and ways to distinguish them from goals. WC denoted that acceptance, present moment, and commitment processes were seldom mentioned by participants. The KWiC analysis noted that the intervention overemphasized values and gave less importance to other processes. ACT experts suggested including metaphors and examples that expanded whole psychological flexibility components and problems related to inflexible patterns (e.g., control, avoidance) from the beginning of the program.

When connecting quantitative and qualitative data, it was observed that on average participants agreed FACE COVID was up to be scaled. However, they did not give a totally agreed score for all scalability components, qualitative data provided some insights into participants’ quantitative data.

On delivery, audio features such as prosody, latencies, and volume were considered barriers to understanding and engaging in FACE COVID strategies. The other factor that hindered delivery was the small number of exercises and examples in the videos.

Reach scores were downplayed by pragmatics in some examples. Participants perceived some content unfitted their needs. Health care providers stated that they already spend a lot of time at work and are fully concentrated on helping patients; therefore, they felt invalidated when video messages suggest them to “help others” or “be more focused”. ACT experts agreed videos would be benefitted by including examples, analogies, and metaphors related to health providers’ jobs and other various roles they have. In addition, they recommended improving pragmatics by employing phrases and words from medical settings but with no jargon.

Finally, fidelity would be improved by including more information in all ACT processes. Experts highlighted that the values process was present throughout the program but other components such as acceptance, present moment, experiential avoidance, and control as a problem were skipped. According to them, this may distort participants’ expectancies about ACT objectives and how it can help to manage suffering, stress, and pain.

Preliminary Efficacy Pretest-Posttest Assessment. Retention rate was 55.5%, and the last video watched for more non-completers (people who engaged with less than 70% of FACE COVID activities) was number five (“Experience the value of the world around you”).

Three out of twelve participants did not complete posttest measures, none of them reported high levels of distress (DASS) or inflexibility (CompACT) at pretest. In addition, three data points of CompACT were missed in the posttest.

On comparing pretest-posttest scores in life satisfaction in the present, five out of nine reported improvements. Two participants reported a reduction in life satisfaction from average to low, one moved from high to average, and one did not show changes (Table 2 ).

Table 2.

Pretreatment and posttreatment participants comparison.

Ps Pretreatment
 Posttreatment
DASS DASS clinic TSLS TSLS clinic CompACT CompACT clinic DASS DASS clinic TSLS TSLS clinic CompACT CompACT
Clinic
Ann 8 ∼ 25 + 66 ∼ 5 – 20 ∼ 73 ∼
Joan 46 + 10 – 87 ∼ 38 + 15 ∼ 67 ∼
Lili 35 + 20 ∼ 111 + 28 ∼ 12 – 102 ∼
Joe 0 – 12 – 89 ∼ 0 – 15 ∼ 75 ∼
Cloe 14 ∼ 16 ∼ 104 ∼ 58 + 10 – 90 ∼
Dian 5 – 18 ∼ 55 – 5 – 20 ∼ 79 ∼
Katy 14 ∼ 19 ∼ 86 ∼
Lexi 50 + 7 – 86 ∼
Paul 18 ∼ 13 – 81 ∼ 20 ∼ 53 + M
Max 22 ∼ 14 ∼ 91 ∼ 13 ∼ 47 + M
Tim 14 ∼ 16 ∼ 83 ∼
Mary 51 + 12 – 74 ∼ 5 – 59 + M
x̄ 84,42 15,17 84,42 86,50 27,89 81,00
Me 86,00 15,00 86,00 84,50 20,00 77,00
SD 15,05 4,90 15,05 10,24 19,34 12,82
Open in a new tab

Ps = Pseudonym; + = 1SD above mean from questionnaires' validation; - = 1SD below mean from questionnaires' validation; ∼ = between normal ranges on the mean from questionnaire validation. M = missing data.

Regarding distress, three participants reported a reduction in psychological distress when comparing pretest and posttest scores. Five participants did not evidence changes and one showed deterioration. Psychological flexibility scores remained stable for most participants from pretest to posttest. Only one participant reported improvements, and another had lower scores, moving from highly flexible to average flexibility (Table 2).

4. Discussion

This study provides useful information on the process to scale up an ACT-based TMI and its feasibility. The first stage evaluates FACE COVID scalability for health care providers, showing good rates by participants. Using a mix-method approach provided further information on the usability and acceptability of the intervention. Beyond the quantitative scores, experts and users were able to offer important insights to improve treatment implementation. For instance, presenting simultaneously at least two ACT processes through each phase of the study, designing engaging strategies, and utilizing context- and user-sensitive content.

While scalability has been frequently called an important step to enhancing adaptability and usability of treatments in new contexts (Bauer et al., 2015; Milat et al., 2013), most guidelines to assess it have focused on great scale interventions in public health but not on technology adaptations neither psychology treatments (Zamboni et al., 2019). This study offers a set of Delphi questions for evaluating scalability of TMI qualitatively and quantitatively for psychological intervention that can be validated in future studies.

On the scalability assessment, both health care professionals and ACT experts agreed that FACE COVID could facilitate engagement and access to strategies that target well-being and distress. Participants thought FACE COVID met acceptability as they felt positive about its content, adherence to ethical standards, efficiency, and completion of the program (Sekhon et al., 2017). On intervention coherence, participants reported a clear understanding of the values strategies, relating to metaphors linking values, healthy behaviors, and meaningful life (i.e., lighthouse metaphor as values; navigating towards lighthouses as value-oriented behaviors). Depicting stress as a normal experience for health care providers was perceived as validating; they reported feeling less resistant to stress after watching videos on this topic. FACE COVID recommendations supporting social connection despite epidemiological restrictions were appealing and well-received.

Some adjustments were made to improve this program based on qualitative evaluations in the first stage. For instance, to improve health providers' engagement, we included more value exemplars and erased those related to “helping others”. The negative response to this suggestion could be related to compassion fatigue (feelings of exhaustion, helplessness, and confusion associated with working in the line of duty; Fligey, 1995) that steeply increased among health providers during the COVID-19 pandemic (Franza et al., 2020; Missouridou et al., 2021). A comprehensive definition of values, and how they differ from goals and committed actions was also included, a key component of other ACT online programs (Levin et al., 2017).

In addition, content about other ACT processes such as acceptance, experiential avoidance, and mindfulness was integrated. Some studies have shown experiential avoidance is a prevalent issue that boosts distress and burnout in healthcare professionals (Iglesias et al., 2010). In order to reduce the impact of experiential avoidance we included various strategies within FACE COVID such as literacy on experiential avoidance and control as a problem, “forcing to experience an emotion” exercise, “body connection vs experiential avoidance” exercise, and so forth (Appendix A). Finally, the awareness module in FACE COVID labeled “Focus on what is under your control” was renamed to “Familiarize with your feelings and get disentangled from your hooks” to improve mindfulness-process reception. Improving pragmatics for this module was particularly important as one-on-one and TMI studies with health care providers have shown that meditative practices are associated with lower levels of burnout and compassion fatigue (Goldhagen et al., 2015; Goodman & Schorling, 2012; Heeter et al., 2017).

On delivery, research in web-based interventions has found voice features are relevant to making messages persuasive (Lehto & Oinas-Kukkonen, 2011). Based on this, video and audio were re-recorded to improve delivery. We also redesigned practice materials (e.g., audios, cartoons), a useful strategy found in previous ACT web-based interventions (Levin et al., 2017).

The preliminary efficacy data provided important information on the potential effects of FACE COVID and its implementation for health care providers. Most participants (67%) who completed posttest data reported improvements in present life satisfaction, but distress improvements were lower. Although retention was medium and changes on pretest-posttest were only observed in life satisfaction, this study showed that ACT-based strategies adapted to technology might be feasible to enhance well-being in health care providers. As this population is at risk due to general job demands and the pandemic (Chen et al., 2020; Johnson et al., 2018; Sastre-Rus., 2019), this study provides some insights into ways to defeat barriers they face toward mental health treatments.

In addition, this study's results add evidence to support ACT-based programs that have shown good outcomes (Levin et al., 2016; Kelson et al., 2019) and extend the scope of action by intervening over a non-clinical population. Indeed, participants suggested that even though FACE COVID was originally developed for healthcare professionals facing the COVID-19 crisis, this program content could be useful to enhance their well-being at any moment. This also can be useful for particular medical specializations such as emergency medicine in which health care providers are especially vulnerable to burnout and suicidal thoughts (Stehman et al., 2019).

Limitations. One of the major limitations of the scalability assessment was the absence of nurses and physicians (key health care professionals) in the sample. This is common in research with health care providers as they usually experience important time constraints (Flanigan et al., 2008), which has been even more challenging during the COVID-19 pandemic. To avoid scheduling barriers, future research may include more flexible one-on-one data collection techniques such as in-depth interviews. Other strategies such as short open-ended questions about TMIs features can be also handled via online. Post-pandemic studies also can invite participants to lunch or dinner at their workplaces to facilitate participation as these activities do not require investing part of their free time.

The nature of the preliminary assessment of efficacy, in which the sample size is small, hinders the capacity of generalizing results from this study. Implementation of this treatment in a larger sample with a comparison group is needed to accurately determine its effects on well-being and distress.

Participants’ retention is another limitation observed in stage two of this study. Lacking social contact with treatment facilitators may likely impact retention. Newman et al. (2011a, 2011b) who conducted two systematic reviews on technology-mediated interventions for diverse psychological problems found that offering support from a direct facilitator increases adherence to TMI. Particularly, low compliance occurred when TMIs were designed to be used at home without involving human contact. Rather, a review using TMI in mindfulness showed promising treatment outcomes even when a facilitator was not included (Fish et al., 2016). The present study only offered social contact by a research assistant during the pre-intervention, it is likely that the lack of social companionship by the research team made participants less likely to complete. Further studies should incorporate regular human contact aiming to increase engagement with TMI. Some strategies for offering regular contact include sending emails, messages, or calls through the intervention. Indeed, phone contacts have shown positive effects on treatment outcomes, retention, and acceptability of TMIs (Rooke et al., 2014). Pre-treatment sessions can also be helpful to improve adherence as researchers can offer information about treatment components, rationale, and steps, as well as solve questions about the program (Coyle et al., 2019).

Stigma and poor knowledge about mental health problems and services are factors that may hinder retention (Chen et al., 2020; Johnson et al., 2018; Sastre-Rus., 2019). Some research has demonstrated that participants’ poor understanding and knowledge of mental health can interfere with their ability to complete therapy. Even when they are willing to receive help, their commitment could be affected by beliefs about psychological interventions and whether they would be helpful (Lattie et al., 2019; Switsers et al., 2018). While this study implemented a TMI in order to defeat stigma and poor knowledge about mental health, it could have not been enough. Lacking specific interventions on stigma could have affected retention in this study as health care providers are a community often impacted by mental health stigma (Chen et al., 2020; Johnson et al., 2018; Sastre-Rus et al., 2019). Future studies might include measures of mental health stigma to identify to what degree it influences retention. In addition, to demote stigma they can include strategies to promote defusion of unhelpful cognitive content on mental health such as (a) definitions of well-being and psychological flexibility, (b) the happiness trap and cultural rules that hinder well-being, (c) ACT components and its approach to well-being, and (d) the role of values and committed actions in well-being.

The absence of pre-intervention value assessments might affect adherence. Although participants were asked to provide a subjective evaluation of their engagement in value-oriented behaviors daily, we relied on their understanding of what values meant based on the information provided in the first intervention video; however, there was not a follow-up to evaluate to what degree participants values were clear. A brief session introducing and clarifying values and committed actions would enhance people's participation. In addition, including idiographic, objective, and direct (e.g., occurrence, duration, or product) measures of value-oriented behaviors would improve progress evaluation, which has proven useful in other technology-mediated ACT studies (Kurumiya et al., under review).

Skillful technology usage might have influenced participants' engagement. Stiles-Shields et al. (2017) found that low levels of participation in TMIs can be related to lacking knowledge in technology management, privacy, cost, and lack of guidance and feedback. Some participants likely face problems managing technology features (i.e., notifications). Future studies may want to include the level of technology use as an exclusion criterion or set training sessions for assuring participants get technology savvy, as well as offering a contact channel for problem-solving.

Time commitment is another factor that could hinder adherence and retention. While FACE COVID strategies were delivered through an app, its dose intensity (daily) may have overwhelmed or interfered with participants' life. In addition, activities were only accessible during a 24hr period, a short time frame for health care professionals who had time constraints. Some studies with health care providers identify that lacking time to complete interventions is associated with dropping out (Burton et al., 2016; Johnson et al., 2018). Perceiving those interventions are too long and skills learned are hard to integrate into their daily routines (Yardley et al., 2016) might be related to participants’ difficulties in completing the intervention. Based on this, it is recommended to extend treatment length and allow ongoing access to interventions to increase retention, skills practice, and generalization.

Finding an appropriate space and timing to access intervention are other aspects that might influence participants' completion. Carolan and de Visser (2018) found people's rates of completion in online mental health programs are hindered by difficulties in finding a private space at home or work to use them.

In addition, most participants reported lower- or average levels of distress contrary to expectations as they were facing the burden of providing health services during the COVID-19 pandemic. Therefore, it is likely that the results were based on a negative sampling that offers insights and opens some questions. First, this data reinforces the notion of independence between well-being and distress, people can continue improving their life satisfaction while reporting no changes in distress. Second, even when there is some leverage to continue working on improving their well-being, health care professionals likely drop out of mental health programs when their levels of distress are low. Future studies would compare treatment outcomes based on well-being and retention with samples reporting different levels of distress.

Three recommendations are drawn for overcoming the methodological limitations of this study. First, to include measures that assess health care professional workplace factors. Some studies proposed that organizational factors played an important role in supporting or hindering healthcare professionals' well-being during the COVID-19 pandemic (Gavin et al., 2020). Second, administer instruments validated in the target context and population. In this study, scalability surveys and CompACT lacked psychometric information. However, as scalability assessment is in an early stage of development is a pervasive limitation among several studies in the area (Milat et al., 2020; Zamboni et al., 2019). Third, improve recruitment to include diverse participants from different contexts and professions. The current sample came from a private hospital which lessens conclusions’ generalizability. It would be interesting to evaluate scalability in other environments and populations within the health care realm to expand the scope of these results.

5. Conclusion

In sum, this research provides a systematic assessment of scalability for ACT-based strategies adapted to TMI, contributing to the growing body of adaptations of empirically based interventions in non-traditional formats. In addition, this preliminary results on the efficacy of FACE COVID show a positive effect on well-being in a non-clinical population as health care providers, offering some information on the utility of ACT out of the clinical scope. It is expected that this investigation encourages further studies that scale contextual-behavioral interventions oriented to improve general population well-being.

Data availability

The raw data required to reproduce the above findings are available at the Open Science Framework repository: DOI 10.17605/OSF·IO/XUGSJ.

Acknowledgments

The study was fund by the College of Social Science of the University of Los Andes.

Footnotes

Appendix A

Supplementary data to this article can be found online at https://doi.org/10.1016/j.jcbs.2022.07.005.

Appendix A. Supplementary data

The following is the Supplementary data to this article:

Multimedia component 1
mmc1.docx (105.2KB, docx)

References

  1. Bauer M.S., Damschroder L., Hagedorn H., Smith J., Kilbourne A.M. An introduction to implementation science for the non-specialist. BMC Psychology. 2015;3(1):1–12. doi: 10.1186/s40359-015-0089-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Berkout O.V. Working with values: An overview of approaches and considerations in implementation. Behavior Analysis in Practice. 2022;(15):104–114. doi: 10.1007/s40617-021-00589-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Burton A., Burgess C., Dean S., Koutsopoulou G., Hugh-Jones S. How effective are mindfulness-based interventions for reducing stress among healthcare professionals? A systematic review and meta-analysis. Stress and Health. 2016 doi: 10.1002/smi.2673. [DOI] [PubMed] [Google Scholar]
  4. Carolan S., de Visser R.O. Employees' perspectives on the facilitators and barriers to engaging with digital mental health interventions in the workplace: Qualitative study. Journal of Medical Internet Research Mental Health. 2018;5(1) doi: 10.2196/mental.9146. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Chen N., Zhou M., Dong X., Qu J., Gong F., Han H., Qiu Y., Wang J., Liu Y., Wei Y., Xia J., Yu T., Zhang X., Zhang L. Epidemiological and clinical characteristics of 99 cases of 2019 novel coronavirus pneumonia in wuhan, China: A descriptive study. The Lancet. 2020;395:507–513. doi: 10.1016/S0140-6736(20)30211-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Cicchetti D., Rogosch F.A. Equifinality and multifinality in developmental psychopathology. Development and Psychopathology. 1996;8:597–600. doi: 10.1017/S0954579400007318. [DOI] [Google Scholar]
  7. Coyle T.N., Brill C.D., Aunon F.M., Peterson A.P., Gasser M.L., Kuehn K.S., Harned M.S. Beginning to envision a life worth living: An introduction to pretreatment sessions in dialectical behavior therapy. Psychotherapy. 2019;56(1):21. doi: 10.1037/pst0000205. [DOI] [PubMed] [Google Scholar]
  8. Dougher M., Perkins D.R., Greenway D., Koons A., Chiasson C. Contextual control of equivalence‐based transformation of functions. Journal of the Experimental Analysis of Behavior. 2002;78(1):63–93. doi: 10.1901/jeab.2002.78-63. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Fish J., Brimson J., Lynch S. Mindfulness interventions delivered by technology without facilitator involvement: What research exists and what are the clinical outcomes? Mindfulness. 2016;7(5):1011–1023. doi: 10.1007/s12671-016-0548-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Flanigan T.S., McFarlane E., Cook S. Proceedings of the survey research methods section. American Statistical Association; 2008, May. Conducting survey research among physicians and other medical professionals: A review of current literature.https://www.asasrms.org/Proceedings/y2008/Files/flanigan.pdf Retrieved from: [Google Scholar]
  11. Fligey C.R. Brunner/Routledge; 1995. Compassion Fatigue as Secondary Traumatic Stress Disorder an overview. Compassion fatigue: Coping with secondary traumatic stress disorder in those who treat the traumatized. [Google Scholar]
  12. Francis A.W., Dawson D.L., Golijani-Moghaddam N. The development and validation of the comprehensive assessment of acceptance and commitment therapy processes (CompACT) Journal of Contextual Behavioral Science. 2016;5(3):134–145. [Google Scholar]
  13. Franza F., Basta R., Pellegrino F., Solomita B., Fasano V. The role of fatigue of compassion, burnout, and hopelessness in healthcare: Experience in the time of COVID-19 outbreak. Psychiatria Danubina. 2020;32(Suppl 1):10–14. [PubMed] [Google Scholar]
  14. Galiana L., Gutiérrez M., Sancho P., Oliver A., Tomás J.M. Psychometric properties of the Spanish version of the temporal satisfaction with life scale (TSLS): A study on elderly attending to university programs. European Journal of Investigation in Health, Psychology, and Education. 2015;5(3):335–344. [Google Scholar]
  15. Gavin B., Lyne J., McNicholas F. Mental health and the COVID-19 pandemic. Irish Journal Of Psychological Medicine. 2020;37(3):156–158. doi: 10.1017/ipm.2020.72. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Goldhagen B.E., Kingsolver K., Stinnett S.S., Rosdahl J.A. Stress and burnout in residents: Impact of mindfulness-based resilience training. Advances in Medical Education and Practice. 2015;6:525. doi: 10.2147/AMEP.S88580. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Goodman M.J., Schorling J.B. A mindfulness course decreases burnout and improves well-being among healthcare providers. The International Journal of Psychiatry in Medicine. 2012;43(2):119–128. doi: 10.2190/PM.43.2.b. [DOI] [PubMed] [Google Scholar]
  18. Hakanen J., Perhoniemi R., Toppinen-Tanner S. Positive gain spirals at work: From job resources to work engagement, personal initiative and work-unit innovativeness. Journal of Vocational Behavior. 2008;73:78–91. [Google Scholar]
  19. Harris R. New Harbinger; Oakland, CA: 2009. ACT made simple: A quick-start guide to ACT basics and beyond. [Google Scholar]
  20. Harris FACE COVID How to respond effectively to the Corona crisis. 2020. https://www.baps.org.uk/content/uploads/2020/03/FACE-COVID-by-Russ-Harris-pdf-pdf.pdf
  21. Hayes S.C., Levin M.E., Plumb-Vilardaga J., Villatte J.L., Pistorello J. Acceptance and commitment therapy and contextual behavioral science: Examining the progress of a distinctive model of behavioral and cognitive therapy. Behavior Therapy. 2013;44(2):180–198. doi: 10.1016/j.beth.2009.08.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Hayes S.C., Luoma J.B., Bond F.W., Masuda A., Lillis J. Acceptance and commitment therapy: Model, processes, and outcomes. Behaviour Research and Therapy. 2006;44(1):1–25. doi: 10.1016/j.brat.2005.06.006. [DOI] [PubMed] [Google Scholar]
  23. Heeter C., Lehto R., Allbritton M., Day T., Wiseman M. Effects of a technology-assisted meditation program on healthcare providers' interoceptive awareness, compassion fatigue, and burnout. Journal of Hospice and Palliative Nursing. 2017;19(4):314–322. [Google Scholar]
  24. Iglesias M.E.L., de Bengoa Vallejo R.B., Fuentes P.S. The relationship between experiential avoidance and burnout syndrome in critical care nurses: A cross-sectional questionnaire survey. International Journal of Nursing Studies. 2010;47(1):30–37. doi: 10.1016/j.ijnurstu.2009.06.014. [DOI] [PubMed] [Google Scholar]
  25. Johnson S., Lamb D., Marston L., Osborn D., Mason O., Henderson C., Ambler G., Milton A., Davidson M., Christoforou M., Sullivan S., Hunter R., Hindle D., Paterson B., Leverton M., Piotrowski J., Forsyth R., Mosse L., Goater N., Lloyd-Evans B. Peer-supported self-management for people discharged from a mental health crisis team: A randomised controlled trial. The Lancet. 2018;392:409–418. doi: 10.1016/S0140-6736(18)31470-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Jung S.J., Jun J.Y. Mental health and psychological intervention amid COVID-19 outbreak: Perspectives from South Korea. Yonsei Medical Journal. 2020;61(4):271–272. doi: 10.3349/ymj.2020.61.4.271. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Kang L., Li Y., Hu S., Chen M., Yang C., Yang B.…Liu Z. The mental health of medical workers in Wuhan, China dealing with the 2019 novel coronavirus. The Lancet. 2020;7(3):e14. doi: 10.1016/S2215-0366(20)30047-X. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Kelson J., Rollin A., Ridout B., Campbell A. Internet-delivered acceptance and commitment therapy for anxiety treatment: Systematic review. Journal of Medical Internet Research. 2019;21(1) doi: 10.2196/12530. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Kurumiya, Y., Garcia, Y., Szabo, T., & Griffith, A. (Under review). Using online acceptance and commitment training matrix for Japanese-speaking parents with distress. Journal of Child and Family Studies.. [DOI] [PMC free article] [PubMed]
  30. Lattie E.G., Adkins E.C., Winquist N., Stiles-Shields C., Wafford Q.E., Graham A.K. Digital mental health interventions for depression, anxiety, and enhancement of psychological well-being among college students: Systematic review. Journal of Medical Internet Research. 2019;21(7) doi: 10.2196/12869. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Leech N.L., Onwuegbuzie A.J. Qualitative data analysis: A compendium of techniques and a framework for selection for school psychology research and beyond. School Psychology Quarterly. 2008;23(4):587–604. doi: 10.1037/1045-3830.23.4.587. [DOI] [Google Scholar]
  32. Lehto T., Oinas-Kukkonen H. Persuasive features in web-based alcohol and smoking interventions: A systematic review of the literature. Journal of Medical Internet Research. 2011;13(3) doi: 10.2196/jmir.1559. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Levin M., Haeger J., Pierce1 B., Twohig M. Web-based acceptance and commitment therapy for mental health problems in college students: A randomized controlled trial. Behavior Modification. 2017;41:141–162. doi: 10.1177/0145445516659645. [DOI] [PubMed] [Google Scholar]
  34. Levin M.E., Hayes S.C., Pistorello J., Seeley J.R. Web‐based self‐help for preventing mental health problems in universities: Comparing acceptance and commitment training to mental health education. Journal of clinical psychology. 2016;72(3):207–225. doi: 10.1002/jclp.22254. [DOI] [PubMed] [Google Scholar]
  35. Linn A.J., Vervloet M., van Dijk L., Smit E.G., Van Weert J.C. Effects of health interventions on medication adherence: A systematic review of the literature. Journal of Medical Internet Research. 2011;13(4) doi: 10.2196/jmir.1738. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Lloyd J., Bond F.W., Flaxman P.E. The value of psychological flexibility: Examining psychological mechanisms underpinning a cognitive behavioural therapy intervention for burnout. Work & Stress. 2013;27(2):181–199. doi: 10.1080/02678373.2013.782157. [DOI] [Google Scholar]
  37. Maunder R., Lancee W., Balderson K., Bennett J., Borgundvaag B., Evans S., Fernandes C., Goldbloom D., Gupta M., Hunter J., McGillis L., Nagle L., Pain C., Peczeniuk S., Raymond G., Read N., Rourke S., Steinberg R., Stewart T., Wasylenki D. Long-term psychological and occupational effects of providing hospital healthcare during SARS outbreak. Emerging Infectious Diseases. 2006;12(2):1924–1932. doi: 10.3201/eid1212.060584. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Milat A.J., King L., Bauman A.E., Redman S. The concept of scalability: Increasing the scale and potential adoption of health promotion interventions into policy and practice. Health Promotion International. 2013;28(3):285–298. doi: 10.1093/heapro/dar097. [DOI] [PubMed] [Google Scholar]
  39. Milat A., Lee K., Conte K., Grunseit A., Wolfenden L., Van Nassau F., Bauman A. Intervention scalability assessment tool: A decision support tool for health policymakers and implementers. Health Research Policy and Systems. 2020;18(1):1–17. doi: 10.1186/s12961-019-0494-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Missouridou E., Mangoulia P., Pavlou V., Kritsotakis E., Stefanou E., Bibou P., Fradelos E.C. Wounded healers during the COVID‐19 syndemic: Compassion fatigue and compassion satisfaction among nursing care providers in Greece. Perspectives in Psychiatric Care. 2021 doi: 10.1111/ppc.12946. Advance online publication. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Mistry N., Keepanasseril A., Wilczynski N.L., Nieuwlaat R., Ravall M., Haynes R.B., amp; Patient Adherence Review Team Technology-mediated interventions for enhancing medication adherence. Journal of the American Medical Informatics Association. 2015;22(e1):e177–e193. doi: 10.1093/jamia/ocu047. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Morse S., Mazet J., Woolhouse M., Parrish C., Carroll D., Karesh W., Zambrana-Torrelio C., Lipkin W., Daszak P. Prediction and prevention of the next pandemic zoonosis. The Lancet. 2012;380:1956–1965. doi: 10.1016/S0140-6736(12)61684-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Naismith, I., Carrillo, S., Muñoz-Martínez, A., Taborda, R., & Wills, E. (Under review). Socioemotional and workplace predictors of job satisfaction, life satisfaction, and positive mental health in university employees.
  44. Newman M.G., Szkodny L.E., Llera S.J., Przeworski A. A review of technology-assisted self-help and minimal contact therapies for anxiety and depression: Is human contact necessary for therapeutic efficacy? Clinical Psychology Review. 2011;31(1):89–103. doi: 10.1016/j.cpr.2010.09.008. [DOI] [PubMed] [Google Scholar]
  45. Newman M.G., Szkodny L.E., Llera S.J., Przeworski A. A review of technology-assisted self-help and minimal contact therapies for drug and alcohol abuse and smoking addiction: Is human contact necessary for therapeutic efficacy? Clinical Psychology Review. 2011;31(1):178–186. doi: 10.1016/j.cpr.2010.10.002. [DOI] [PubMed] [Google Scholar]
  46. OSF: 10.17605/OSF.IO/XUGSJ.
  47. Prudenzi A., Graham C.D., Clancy F., Hill D., O'Driscoll R., Day F., O'Connor D.B. Group-based acceptance and commitment therapy interventions for improving general distress and work-related distress in healthcare professionals: A systematic review and meta-analysis. Journal of Affective Disorders. 2021;295:192–202. doi: 10.1016/j.jad.2021.07.084. [DOI] [PubMed] [Google Scholar]
  48. Räsänen P., Lappalainen P., Muotka J., Tolvanen A., Lappalainen R. An online guided ACT intervention for enhancing the psychological well-being of university students: A randomized controlled clinical trial. Behaviour Research and Therapy. 2016;78 doi: 10.1016/j.brat.2016.01.001. [DOI] [PubMed] [Google Scholar]
  49. Reeve A., Tickle A., Moghaddam N. Are acceptance and commitment therapy-based interventions effective for reducing burnout in direct-care staff? A systematic review and meta-analysis. Mental Health Review Journal. 2018 doi: 10.1108/MHRJ-11-2017-0052. [DOI] [Google Scholar]
  50. Rooke S.E., Gates P.J., Norberg M.M., Copeland J. Applying technology to the treatment of cannabis use disorder: Comparing telephone versus internet delivery using data from two completed trials. Journal of Substance Abuse Treatment. 2014;46(1):78–84. doi: 10.1016/j.jsat.2013.08.007. [DOI] [PubMed] [Google Scholar]
  51. Ruiz F.J., Martín M.B.G., Falcón J.C.S., González P.O. The hierarchical factor structure of the Spanish version of Depression Anxiety and Stress Scale-21. International Journal of Psychology and Psychological Therapy. 2017;17(1):97–105. [Google Scholar]
  52. Safieh J., Broughan J., McCombe G., McCarthy N., Frawley T., Guerandel A.…Cullen W. Interventions to optimize mental health outcomes during the COVID-19 pandemic: A scoping review. International Journal of Mental Health and Addiction. 2021 doi: 10.1007/s11469-021-00558-3. Advance online publication. [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Sastre-Rus M., García-Lorenzo A., Lluch-Canut M., Tomás-Sábado J., Zabaleta-Del-Olmo E. Instruments to assess mental health-related stigma among health professionals and students in health sciences: A systematic psychometric review. Journal of Advanced Nursing. 2019;75(9):1838–1853. doi: 10.1111/jan.13960. [DOI] [PubMed] [Google Scholar]
  54. Sekhon M., Cartwright M., Francis J.J. Acceptability of healthcare interventions: An overview of reviews and development of a theoretical framework. BMC Health Services Research. 2017;17(1):1–13. doi: 10.1186/s12913-017-2031-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  55. Society of Clinical Psychology Psychological treatments. 2021. https://div12.org/treatments/ Retrieved from.
  56. Stehman C., Testo Z., Gershaw R., Kellogg A. Burnout, drop out, suicide: Physician loss in emergency medicine, Part I. Western Journal of Emergency Medicine. 2019;20(3):485–494. doi: 10.5811/westjem.2019.4.40970. [DOI] [PMC free article] [PubMed] [Google Scholar]
  57. Stenhoff A., Steadman L., Nevitt S., Benson L., White R.G. Acceptance and commitment therapy and subjective well-being: A systematic review and meta-analyses of randomised controlled trials in adults. J. Contex. Behav.Sci. 2020;18:256–272. doi: 10.1016/j.jcbs.2020.08.008. [DOI] [Google Scholar]
  58. Stiles-Shields C., Montague E., Lattie E.G., Kwasny M.J., Mohr D.C. What might get in the way: Barriers to the use of apps for depression. Digital Health. 2017;3:1–11. doi: 10.1177/2055207617713827. [DOI] [PMC free article] [PubMed] [Google Scholar]
  59. Stoddard J.A., Afari N. New Harbinger Publications; 2014. The big Book of ACT metaphors: A practitioner's guide to experiential exercises and metaphors in Acceptance and commitment therapy. [Google Scholar]
  60. Strauss A., Corbin J. 2nd ed. Sage Publications, Inc; 1998. Basics of qualitative research: Techniques and procedures for developing grounded theory. [Google Scholar]
  61. Switsers L., Dauwe A., Vanhoudt A., Van Dyck H., Lombaerts K., Oldenburg J.F.E. Users' perspectives on mHealth self-management of bipolar disorder: Qualitative focus group study. J. Med. Int. Res. mHealth and uHealth. 2018;6(5) doi: 10.2196/mhealth.9529. [DOI] [PMC free article] [PubMed] [Google Scholar]
  62. Tay J.L., Tay Y.F., Klainin‐Yobas P. Effectiveness of information and communication technologies interventions to increase mental health literacy: A systematic review. Early Intervention. Psy. 2018;12(6):1024–1037. doi: 10.1111/eip.12695. [DOI] [PubMed] [Google Scholar]
  63. Trindade I.A., Ferreira N.B., Mendes A.L., Ferreira C., Dawson D., Golijani-Moghaddam N. Comprehensive assessment of Acceptance and Commitment Therapy processes (CompACT): Measure refinement and study of measurement invariance across Portuguese and UK samples. J. Contex. Behav.Sci. 2021;21:30–36. [Google Scholar]
  64. Vilardaga R., Luoma J.B., Hayes S.C., Pistorello J., Levin M.E., Hildebrandt M.J., Kohlenberg B., Roget N., Bond F. Burnout among the addiction counseling workforce: The differential roles of mindfulness and values-based processes and work-site factors. Journal of Substance Abuse Treatment. 2011;40(4):323–335. doi: 10.1016/j.jsat.2010.11.015. [DOI] [PMC free article] [PubMed] [Google Scholar]
  65. Wahyun E., Nurihsan J., Yusuf S. Effectiveness of acceptance and commitment therapy to enhance students' wellness. J. Evidence-Based Psycho. 2019;19(1):91–113. [Google Scholar]
  66. Walser R.D., Westrup D. New Harbinger Publications; 2007. Acceptance and commitment therapy for the treatment of post-traumatic stress disorder and trauma-related problems: A practitioner's guide to using mindfulness and acceptance strategies. [Google Scholar]
  67. Wersebe H., Lieb R., Meyer A.H., Hofer P., Gloster A.T. The link between stress, wellbeing, and psychological flexibility during an Acceptance and Commitment Therapy self-help intervention. International Journal of Clinical and Health Psychology. 2018;18(1):60–68. doi: 10.1016/j.ijchp.2017.09.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  68. Whealin J.M., Saleem J.J., Vetter B., Roth J., Herout J. Psychological Services; 2021. Development and cross-sectional evaluation of a text message protocol to support mental health well-being. Advance online publication. [DOI] [PubMed] [Google Scholar]
  69. World Health Organization Doing what matters in times of stress: An illustrated guide. 2020. https://www.who.int/publications/i/item/9789240003927
  70. Yardley L., Spring B.J., Riper H., Morrison L.G., Crane D.H., Curtis K.…Blandford A. Understanding and promoting effective engagement with digital behavior change interventions. American Journal of Preventive Medicine. 2016;51(5):833–842. doi: 10.1016/j.amepre.2016.06.015. [DOI] [PubMed] [Google Scholar]
  71. Zamboni K., Schellenberg J., Hanson C., Betran A., Dumont A. Assessing scalability of an intervention: Why, how and who? Health Policy and Planning. 2019;34:544–552. doi: 10.1093/heapol/czz068. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Multimedia component 1
mmc1.docx (105.2KB, docx)

Data Availability Statement

The raw data required to reproduce the above findings are available at the Open Science Framework repository: DOI 10.17605/OSF·IO/XUGSJ.

Articles from Journal of Contextual Behavioral Science are provided here courtesy of Elsevier

RESOURCES