Skip to main content
BMJ Mental Health logoLink to BMJ Mental Health
. 2023 Jun 1;26(1):e300697. doi: 10.1136/bmjment-2023-300697

Effectiveness of a mental health stepped-care programme for healthcare workers with psychological distress in crisis settings: a multicentre randomised controlled trial

Roberto Mediavilla 1,2,3,4, Mireia Felez-Nobrega 2,5, Kerry R McGreevy 1,2, Anna Monistrol-Mula 6, María-Fe Bravo-Ortiz 1,2,7, Carmen Bayón 1,2,7, Iago Giné-Vázquez 2,6, Rut Villaescusa 6, Ainoa Muñoz-Sanjosé 7, Salvatore Aguilar-Ortiz 6, Natasha Figueiredo 8, Pablo Nicaise 9, A-La Park 10, Papoula Petri-Romão 4, Marianna Purgato 11, Anke B Witteveen 12, James Underhill 12, Corrado Barbui 11, Richard Bryant 13, Raffael Kalisch 4,14, Vincent Lorant 9, David McDaid 10, Maria Melchior 8, Marit Sijbrandij 12, Josep Maria Haro 2,6, Jose Luis Ayuso-Mateos 1,2,3,; the RESPOND Consortium
PMCID: PMC10254812  PMID: 37263708

Abstract

Background

Evidence-based mental health interventions to support healthcare workers (HCWs) in crisis settings are scarce.

Objective

To evaluate the capacity of a mental health intervention in reducing anxiety and depression symptoms in HCWs, relative to enhanced care as usual (eCAU), amidst the COVID-19 pandemic.

Methods

We conducted an analyst-blind, parallel, multicentre, randomised controlled trial. We recruited HCWs with psychological distress from Madrid and Catalonia (Spain). The intervention arm received a stepped-care programme consisting of two WHO-developed interventions adapted for HCWs: Doing What Matters in Times of Stress (DWM) and Problem Management Plus (PM+). Each intervention lasted 5 weeks and was delivered remotely by non-specialist mental health providers. HCWs reporting psychological distress after DWM completion were invited to continue to PM+. The primary endpoint was self-reported anxiety/depression symptoms (Patient Health Questionnaire-Anxiety and Depression Scale) at week 21.

Findings

Between 3 November 2021 and 31 March 2022, 115 participants were randomised to stepped care and 117 to eCAU (86% women, mean age 37.5). The intervention showed a greater decrease in anxiety/depression symptoms compared with eCAU at the primary endpoint (baseline-adjusted difference 4.4, 95% CI 2.1 to 6.7; standardised effect size 0.8, 95% CI 0.4 to 1.2). No serious adverse events occurred.

Conclusions

Brief stepped-care psychological interventions reduce anxiety and depression during a period of stress among HCWs.

Clinical implications

Our results can inform policies and actions to protect the mental health of HCWs during major health crises and are potentially rapidly replicable in other settings where workers are affected by global emergencies.

Trial registration number

NCT04980326.

Keywords: anxiety disorders, depression & mood disorders, COVID-19, adult psychiatry


WHAT IS ALREADY KNOWN ON THIS TOPIC.

  • Evidence-based psychological interventions for healthcare workers in crises settings are scarce.

WHAT THIS STUDY ADDS

  • This is the first randomised controlled trial to provide evidence on the short-term and midterm effectiveness of a remotely delivered, scalable, stepped-care programme to reduce anxiety, depression and post-traumatic stress symptoms among healthcare workers.

  • This is also the first trial to explore the effectiveness of the guided self-help intervention called Doing What Matters in Times of Stress.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

  • This stepped care is free, safe, effective and potentially scalable, and can already be implemented in health services at a larger scale.

Background

The COVID-19 pandemic has put many health systems under much pressure. Pooled prevalence estimates of mental health problems show that at least one in three healthcare workers (HCWs) report symptoms compatible with depression, anxiety and/or post-traumatic stress disorder (PTSD).1–4 In Spain, one of the earliest pandemic hotspots worldwide, one in four HCWs had a probable major depressive disorder and one in five had a probable anxiety disorder or PTSD by late 2020.5 Only a few randomised controlled trials (RCTs) have focused on HCWs’ anxiety and depression symptoms during the COVID-19 pandemic.6–13 However, most have used short psychological interventions with no previous evidence, included small sample sizes (mostly limited to nurses) and used short follow-up periods. Two studies reported positive results on anxiety symptoms, but only immediately after a single mindfulness training session,6 7 and one reported a decrease in general psychiatric comorbidity at 8 weeks from baseline using a smartphone app but did not find any effects on anxiety or depression symptoms.13

This general lack of evidence is in keeping with findings from previous epidemics14 and other public health emergencies.15 The WHO developed two free-access mental health intervention programmes intended to be highly scalable within crisis settings: a stress management course named Self Help Plus (SH+)16 and a brief intervention based on cognitive–behavioural and problem-solving strategies called Problem Management Plus (PM+).17 Both interventions have proved feasible and effective in previous crisis settings,18–20 but offered mixed results during the COVID-19 pandemic among adults with COVID-19-related distress and nursing home workers.21 22 To overcome these limitations and potential implementation barriers, we worked with the WHO to adapt these interventions to COVID-19 (context) and HCWs (population). The contextual adaptation aimed at increasing scalability by supporting different levels of psychological distress even during strict lockdowns. To this end, we combined these interventions into a stepped-care programme compatible with fully remote training, delivery and supervision. The population adaptation aimed at tailoring the intervention to HCWs’ specific problems and needs. To do this, we revised intervention protocols to include both HCWs-related stressors (eg, shortages of protective equipment) and implementation barriers (eg, changing working shifts).23

Objective

In this study, we examined the efficacy of a stepped-care programme in reducing short-term and midterm anxiety and depression symptoms in a sample of psychologically distressed HCWs from two initial pandemic hotspots in Spain.

Methods

Study design and participants

We conducted a multicentre, parallel-group, analyst-blinded RCT to explore the effectiveness of a stepped-care programme versus enhanced care as usual (eCAU) among HCWs with psychological distress. The trial protocol was prospectively registered on 28 July 2021 (identifier NCT04980326) and described in detail elsewhere 24 (online supplemental file 1). After recruitment started, the Project Executive Board agreed to modify the criterion for completing the interventions (see the Statistical analysis section), which modified one of the prespecified secondary analyses (per-protocol analysis). These changes did not affect our primary intention-to-treat (ITT) analysis.

Supplementary data

bmjment-2023-300697supp001.pdf (3.6MB, pdf)

Our target population consisted of HCWs employed by the Department of Health. All participants provided written informed consent, and none was paid for taking part in the study. We set the following inclusion and exclusion criteria (there were no withdrawal criteria): the inclusion criteria were being employed by either the Madrilenian or the Catalan Department of Health (doctors, psychologists, nurses, nursing technicians, orderlies and administrative staff), being psychologically distressed (as measured by a score of ≥16 on the Kessler Psychological Distress Scale (K10))25 and being able to read Spanish or Catalan; the exclusion criteria were having an acute medical condition that required immediate hospitalisation, imminent risk of suicide or self-harm or risk of harming others, severe mental disorder (eg, psychotic disorder, delirium), severe cognitive impairment (eg, intellectual disability, dementia), and having initiated, stopped or significantly modified pharmacotherapy or standardised psychological treatment in the previous 8 weeks. We recruited 20 intervention providers and 6 trainers/supervisors. The intervention providers were mental health providers (psychiatry, clinical psychology and mental health nursing trainees) who received specific preparation (~50 hours) and attended weekly supervision sessions while the trial was ongoing (~30 hours). The trainers/supervisors were psychiatrists and clinical psychologists instructed by the intervention developers. Adherence to the intervention protocols was overall good (80%–100% for Doing What Matters in Times of Stress (DWM) and 72%–100% for PM+).

The Hospital Universitario La Paz (Madrid, Spain) coordinated the study, which took place at Hospital Universitario La Paz and Parc Sanitari Sant Joan de Déu (Barcelona, Spain). The results are reported following the Consolidated Standards of Reporting Trials.

Participant and public involvement

As part of the adaptation process, we interviewed a hundred HCWs in Madrid and Barcelona, who provided input and guidance to design this stepped care.23 After the trial, we interviewed roughly 10 participants at each site to better understand their experience during the trial. Once the trial has been published, we will disseminate the results on the WHO Collaborating Centre at Universidad Autónoma de Madrid (www.ccomsuam.org), and we will inform the Departments of Health of the Community of Madrid and Catalonia to increase scalability.

Randomisation and masking

We used random blocks of unequal sizes (4 and 6) to randomise participants to each study arm (1:1 allocation ratio). A research assistant generated an allocation sequence per centre using the electronic data capture (EDC) software Castor (www.castoredc.com). Local project managers who did not provide any intervention nor were aware of the randomisation sequence enrolled participants and assigned them to each study arm. We restricted access of the data analyst to this information by assigning random values to the variable ‘randomisation group’ in the statistical code. Since no observer-reported outcomes obtained by outcome assessors were included, masking did not apply. Neither the participants nor the intervention providers were blind to allocation.

Procedures

All participants initially received a 15 min phone call where the intervention providers informed them about which group they were allocated to and offered information about specific resources following the principles of psychological first aid (PFA).26 Participants allocated to the control arm received care as usual enhanced with PFA (eCAU). Participants allocated to the intervention arm received eCAU and were then offered the stepped-care programme, which comprised two scalable psychological interventions: initially, a guided stress management course based on the SH+ booklet called Doing What Matters in Times of Stress, and potentially an individual intervention based on cognitive–behavioural therapy called Problem Management Plus (PM+). The modules of DWM include information on the acceptance and commitment therapy techniques (eg, acting on values, being kind, ‘unhooking’ from negative experiences), along with audio recordings to support practice. The PM+ protocol provides different strategies that range from problem-solving techniques to strategies for promoting social support or preventing relapses. Both interventions had an online format, and each took 5–6 weeks to be delivered. As a result of the local adaptation process, we transformed the DWM into a mobile-friendly website, rerecorded audio in Spanish or Catalan, and adapted some content to reflect barriers or stress triggers that might affect HCWs in Spain. Based on another WHO intervention named Step-by-Step,27 we provided it as guided self-help and included weekly phone-based or message-based contacts lasting roughly 15 min to provide support and guidance. In the adaptation process, we also adapted PM+. We shortened sessions from 90 to 60 min to fit better within a work setting and tailored case examples to HCWs. The criterion for stepping up to PM+ was reporting significant levels of psychological distress 5–7 days after the DWM (t2), as measured by the K10 scale (score of ≥16). The original intervention manual for PM+ is available on the WHO website, and a guided self-help manual is forthcoming from the WHO. The adapted versions used in this study are available from the corresponding author and described in more detail in the study protocol.24

Outcomes

Assessments were conducted at baseline (t1) and at the three endpoint assessments (weeks 7, 13 and 21, or t2, t3 and t4) using two EDC software (Castor and Qualtrics). Our primary endpoint was self-reported anxiety and depression symptoms 2 months after the full stepped-care programme (t4). Secondary outcomes included anxiety, depression and PTSD symptoms, at all endpoints (t2, t3 and t4).

Primary outcome

The primary outcome was measured using the Patient Health Questionnaire-Anxiety and Depression Scale (PHQ-ADS),28 a 16-item instrument that combines the nine-item Patient Health Questionnaire (PHQ-9)29 and the seven-item Generalised Anxiety Disorder (GAD-7) questionnaire30 into a composite measure of depression and anxiety. The scale ranges from 0 to 48, with higher scores indicating higher levels of depression and anxiety symptoms. We used the validated Spanish versions of PHQ-931 and GAD-7.32 At baseline, the Cronbach’s alpha was 0.90 (95% CI 0.88 to 0.92).

Secondary outcomes

Depression and anxiety symptoms were measured using PHQ-9 and GAD-7, also subscales of the PHQ-ADS. The Spanish version of both instruments has a cut-off score of ≥10 to detect people with probable depression and anxiety. PTSD symptoms were measured using the eight-item version of the PTSD Checklist (PCL-5).33 The scale ranges from 0 to 32, with higher scores indicating higher levels of PTSD symptoms. The instrument is based on the PCL-C, a DSM-IV-based (Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition) checklist validated in Spanish.34

Assessment of safety and adverse events

Local project managers monitored suicidal thoughts across assessment points by including the following items in the outcomes list: ‘Since the last assessment (about two months ago), have you seriously considered or have plans to end your life?’ Similarly, local project managers and intervention providers registered any serious adverse event reported by the participants and informed the independent RESPOND (Preparedness of health systems to reduce mental health and psychosocial concerns resulting from the COVID-19 pandemic) Ethics and Data Advisory Board, chaired by Dr Sonja Rutten.

Sample size and power calculation

We used Stata 17.0 to estimate the sample size required to detect a small to moderate effect size (defined as the square root of the ratio of the variance of the tested effect to the comparison error variance, Cohen’s d=0.3) on the PHQ-ADS summary score at t4. A power calculation for an analysis of variance repeated measurement design with two time periods to identify the effect of treatment at the last endpoint (t4), with a two-sided 5% significance level, a power of 95% and an estimated attrition of 30%, provides a required sample size of 106 participants per group (n=212 total sample size).

Statistical analysis

Overview

Our broad aim was to explore the effect of the intervention on mental health outcomes. We used linear and logistic mixed models with robust SEs to estimate the treatment effect at t2, t3 and t4. All models included both the randomisation group and the outcome score measured at baseline as fixed effects and allowed for an individual random intercept for each participant. We did not impute missing data. We analysed the stepped-care programme as a single intervention (regardless of whether the participant steps up to PM+ or finishes the intervention after the DWM). All statistical analyses were performed in Madrid and independently replicated in Barcelona using R (main packages: lme4, emmeans, clubSandwich and genodds). The statistical analysis protocol24 and the full analytical code (https://github.com/CCOMS-UAM/respond-spa) are available elsewhere. No interim analyses were planned or performed.

Main analyses (prespecified)

We answered our main research question based on the ITT analysis of the primary endpoint (PHQ-ADS summary score at t4) across all randomised participants for whom data were available. We estimated the treatment effect at all time points, but our main comparison used the estimated marginal means (EMMs) and their 95% CI at t4. To ease interpretability, we also reported the standardised effect sizes (SES), which were computed as pairwise differences of EMMs divided by the residual SD of the model.

We answered secondary research questions with a series of prespecified analyses. First, we replicated the ITT analysis on all secondary outcomes (PHQ-9, GAD-7 and PCL-5 summary scores). Second, we undertook three subgroup analyses to explore the moderation effect of relevant variables, namely gender, involvement in the treatment of patients with COVID-19 and symptom severity. Last, we ran fully adjusted models, including age, gender, level of education, use of mental health services prior to enrolment and site, as random effects, to test whether our models were robust against misspecifications.

Sensitivity analyses (exploratory)

We ran three exploratory analyses on our primary and secondary outcomes that were not specified in the study protocol. First, we conducted a complete-case analysis to explore the robustness of our findings in the presence of missing data, for which we analysed all participants who filled in all the assessments in the study. Second, we conducted a modified per-protocol analysis because we changed the completion criteria after the first participant was enrolled, and consequently we do not report it as a prespecified analysis. It included all participants in the control arm and all participants who completed the intervention programme; completers were participants who stepped up and finished both the DWM (ie, clicked through at least three modules) and PM+ (ie, attended at least four sessions) and participants who did not step up and finished the DWM. Last, we carried out ITT binary logistic mixed models to estimate the effect of the intervention on the odds of scoring below the cut-off on PHQ-9 and GAD-7. We reported the adjusted estimates and the number needed to treat along with 95% CIs to supplement and potentially expand the reach of our main findings.

Findings

We randomised 232 participants between 3 November 2021 and 31 March 2022, all of whom were included in the ITT analysis (see figure 1). Participants included 200 women (86%) with a mean age of 37.5 years (SD 10.3). Most (n=215, 93%) were or had been involved in the direct care of patients with COVID-19, and more than half (n=137, 59%) had been infected with COVID-19. Most participants reported symptoms compatible with a probable diagnosis of major depressive disorder (n=124, 53%) or a probable diagnosis of an anxiety disorder (n=134, 58%) (see table 1). Three in four participants stepped up to PM+ (n=86, 75%). Recruitment stopped when the target sample size was achieved.

Figure 1.

Figure 1

Trial overview. DWM, Doing What Matters in Times of Stress; eCAU, enhanced care as usual; HCWs, healthcare workers; ICF, informed consent form; PM+, Problem Management Plus.

Table 1.

Characteristics of the participants at baseline (t1)

Overall, N=232 Group
Control, n=117 Intervention, n=115
Age, M (SD) 37.5 (10.3) 37.1 (10.4) 37.9 (10.1)
Gender, n (%)
 Female 200 (86) 99 (85) 101 (88)
 Male 32 (14) 18 (15) 14 (12)
Educational level, n (%)
 Secondary 1 (0.4) 0 (0) 1 (0.9)
 Technical–professional 41 (18) 18 (15) 23 (20)
 University 190 (82) 99 (85) 91 (79)
Type of job, n (%)
 Physician 50 (22) 28 (24) 22 (19)
 Nurse 130 (56) 66 (57) 64 (56)
 Nursing technician 29 (13) 12 (10) 17 (15)
 Administration 6 (2.6) 1 (0.9) 5 (4.3)
 Other 16 (6.9) 9 (7.8) 7 (6.1)
Job facility, n (%)
 Hospital facilities 147 (63) 72 (62) 75 (65)
 Primary care facilities 68 (29) 35 (30) 33 (29)
 Specialised care facilities 5 (2.2) 3 (2.6) 2 (1.7)
 Emergencies 10 (4.3) 6 (5.1) 4 (3.5)
 Other 2 (0.9) 1 (0.9) 1 (0.9)
Front-line worker (ever), n (%) 215 (93) 108 (92) 107 (93)
COVID-19 infection (ever), n (%) 137 (59) 69 (59) 68 (60)
Site, n (%)
 Madrid 110 (47) 55 (47) 55 (48)
 Catalonia 122 (53) 62 (53) 60 (52)
Anxiety/depression symptoms (PHQ-ADS score, 0–48), M (SD) 20.5 (8.5) 20.2 (8.8) 20.8 (8.1)
Depression symptoms (PHQ-9 score, 0–27), M (SD) 10.3 (4.8) 10.0 (4.9) 10.6 (4.6)
Anxiety symptoms (GAD-7 score, 0–21), M (SD) 10.2 (4.4) 10.2 (4.5) 10.2 (4.2)
Post-traumatic stress symptoms (PCL-5 score, 0–32), M (SD) 12.9 (6.2) 12.7 (6.2) 13.1 (6.3)
Probable major depressive disorder (PHQ-9 >9), n (%) 124 (53) 56 (48) 68 (59)
Probable anxiety disorder (GAD-7 >9), n (%) 134 (58) 70 (60) 64 (56)

GAD-7, seven-item Generalised Anxiety Disorder; M, mean; PCL-5, Post-Traumatic Stress Disorder Checklist for DSM-5 (Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition); PHQ-9, nine-item Patient Health Questionnaire; PHQ-ADS, Patient Health Questionnaire-Anxiety and Depression Scale.

At the end of our follow-up period (31 August 2022), data were available for 184 participants (79.3%), and the overall decrease in anxiety and depression symptoms at the primary endpoint (t4) was larger in the intervention arm compared with the eCAU (baseline-adjusted mean difference 4.4, 95% CI 2.1 to 6.7), with a large to medium effect size (baseline-adjusted SES 0.8, 95% CI 0.4 to 1.2). The decrease was also larger in the intervention arm relative to eCAU at all remaining time points, that is, after DWM or t2 (baseline-adjusted mean difference 3.4, 95% CI 1.7 to 5.2) and after PM+ or t3 (baseline-adjusted mean difference 5.9, 95% CI 3.6 to 8.1) and across all secondary outcomes, except for post-traumatic stress symptoms after the DWM (baseline-adjusted mean difference 1.1, 95% CI -0.4 to 2.6) (see table 2). Overall, effect sizes ranged from small to large and were stronger after PM+ (see figure 2). We did not detect any serious adverse events. Two participants in the intervention arm reported suicidal thoughts during follow-up assessments. They were screened by local facilitators and supervisors and referred to mental health services.

Table 2.

Estimated marginal means of primary and secondary outcomes

Baseline-adjusted models* Fully adjusted models†
Control Intervention Difference Control Intervention Difference
PHQ-ADS‡ Week 7 (post-DWM) 17 (15.9, 18) 13.5 (12.1, 14.9) 3.4 (1.7, 5.2) 18.5 (17.1, 20) 15 (13.3, 16.6) 3.5 (1.8, 5.3)
Week 13 (post-PM+) 15.5 (13.9, 17.2) 9.7 (8.1, 11.3) 5.9 (3.6, 8.1) 17.2 (15.3, 19.1) 11.2 (9.2, 13.3) 6 (3.7, 8.3)
Week 21 (follow-up) 14.8 (13.1, 16.4) 10.4 (8.8, 12) 4.4 (2.1, 6.7) 16.4 (14.6, 18.3) 11.9 (9.7, 14) 4.5 (2.2, 6.9)
PHQ-9§ Week 7 (post-DWM) 8.3 (7.7, 8.9) 6.7 (5.9, 7.5) 1.6 (0.6, 2.6) 9 (8.3, 9.8) 7.4 (6.5, 8.3) 1.6 (0.6, 2.6)
Week 13 (post-PM+) 7.8 (6.9, 8.7) 4.6 (3.8, 5.4) 3.2 (2, 4.4) 8.6 (7.6, 9.6) 5.4 (4.4, 6.3) 3.3 (2.1, 4.5)
Week 21 (follow-up) 7.5 (6.6, 8.4) 5.1 (4.2, 6) 2.5 (1.2, 3.7) 8.3 (7.4, 9.3) 5.8 (4.7, 6.9) 2.6 (1.3, 3.8)
GAD-7¶ Week 7 (post-DWM) 8.7 (8.1, 9.2) 6.8 (6, 7.5) 1.9 (0.9, 2.8) 9.6 (8.7, 10.4) 7.6 (6.6, 8.6) 2 (1, 2.9)
Week 13 (post-PM+) 7.7 (6.8, 8.6) 5.1 (4.3, 5.9) 2.6 (1.4, 3.8) 8.6 (7.5, 9.7) 5.9 (4.8, 7.1) 2.7 (1.5, 3.9)
Week 21 (follow-up) 7.2 (6.4, 8.1) 5.3 (4.5, 6.2) 1.9 (0.7, 3.1) 8.1 (7.1, 9.2) 6.2 (4.9, 7.4) 2 (0.8, 3.2)
PCL-5** Week 7 (post-DWM) 11 (9.9, 12.1) 9.9 (8.8, 10.9) 1.1 (−0.4, 2.6) 11.4 (9.8, 12.9) 10.3 (8.8, 11.8) 1.1 (−0.4, 2.6)
Week 13 (post-PM+) 9.7 (8.5, 10.9) 7.4 (6.3, 8.5) 2.3 (0.7, 3.9) 10.2 (8.6, 11.8) 7.7 (6.1, 9.4) 2.5 (0.8, 4.1)
Week 21 (follow-up) 9.5 (8.2, 10.9) 7.4 (6.2, 8.6) 2.2 (0.3, 4) 10 (8.2, 11.8) 7.7 (5.9, 9.5) 2.3 (0.4, 4.1)

All models include participants as a random effect. All CIs were calculated using robust SEs.

*Adjusted for outcome score measured at baseline.

†Adjusted for age, gender, level of education, use of mental health services prior to enrolment, outcome score measured at baseline and site.

‡The numbers of participants valid at weeks 21, 13 and 7 are 94, 88 and 102 in the control arm, and 90, 86 and 99 in the intervention arm.

§The numbers of participants valid at weeks 21, 13 and 7 are 94, 89 and 102 in the control arm, and 90, 86 and 99 in the intervention arm.

¶The numbers of participants valid at weeks 21, 13 and 7 are 94, 88 and 102 in the control arm, and 90, 86 and 99 in the intervention arm.

**The numbers of participants valid at weeks 21, 13 and 7 are 90, 82 and 99 in the control arm, and 88, 85 and 96 in the intervention arm.

DWM, Doing What Matters in Times of Stress; GAD-7, seven-item Generalised Anxiety Disorder; PCL-5, Post-Traumatic Stress Disorder Checklist for DSM-5; PHQ-9, nine-item Patient Health Questionnaire; PHQ-ADS, Patient Health Questionnaire-Anxiety and Depression Scale; PM+, Problem Management Plus.

Figure 2.

Figure 2

Standardised effect sizes across outcomes and endpoints. All models include participants as random effect and outcomes measured at baseline as fixed effect. The residual SDs of the models were 5.49 for the PHQ-ADS, 3.06 for the PHQ-9, 2.98 for the GAD-7 and 4.24 for the PCL-5. All CIs were calculated using robust SEs. *Primary endpoint. **Primary outcome. DWM, Doing What Matters in Times of Stress; GAD-7, seven-item Generalised Anxiety Disorder; PCL-5, Post-Traumatic Stress Disorder Checklist for the DSM-5 (Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition); PHQ-9, Patient Health Questionnaire; PHQ-ADS, Patient Health Questionnaire-Anxiety and Depression Scale; PM+, Problem Management Plus; SES, standardised effect size.

The results of the prespecified secondary analyses and the sensitivity analyses confirmed our main models. The ITT analyses using fully adjusted models and secondary outcomes are reported in table 2 and figure 2, along with the main analysis. The remaining models are reported in the online supplemental appendix.

Supplementary data

bmjment-2023-300697supp002.pdf (822.6KB, pdf)

Discussion

We explored the effectiveness of a stepped-care programme combining DWM and PM+ on anxiety, depression and PTSD symptoms among HCWs with psychological distress working in COVID-19 pandemic hotspots. Participants in the intervention arm reported fewer anxiety and depression symptoms than the eCAU participants 2 months after the intervention (t4), with overall moderate to large effect sizes and no serious adverse events detected. Overall, our results fill an important research and practice gap, namely the lack of evidence-based mental health support for HCWs in crisis settings, and offer a free-access, stepped-care programme that can rapidly be implemented and delivered by mental health providers (psychiatry, clinical psychology and mental health nursing trainees).

During the COVID-19 pandemic, some RCTs used online and phone-based emotion regulation interventions similar to the first step of our programme, namely DWM. Their effects on anxiety and depression symptoms were either non-existent8 11 13 or limited to the completion of a unique mindfulness training session.6 7 This contrasts with our positive findings with the DWM at 6 weeks from baseline, probably because it was more intense both in terms of duration (5 weeks vs 2–4 weeks) and level of support (weekly follow-up calls vs unguided self-help).35 36 Nevertheless, comparisons are somewhat hindered because we did not include an active comparator and because the stepped-care design did not allow for longer follow-ups for the DWM independently. We found two additional RCTs carried out during the COVID-19 pandemic that used individual counselling12 and group-based cognitive–behavioural therapy9—two interventions slightly similar to the second step of our programme, namely PM+. None reported any positive effects, probably due to their reduced sample sizes. However, our study does not provide effect estimates for stand-alone PM+, which also hinders comparability. We are not aware of any stepped-care programme that supports HCWs with psychological distress before or during the COVID-19 pandemic. Altogether, the combination of DWM and PM+ into a stepped-care programme differs from previously evaluated interventions in many ways, for example, the employment of peers as intervention providers, the programme’s length, especially for HCWs who stepped up to PM+, or the combination of action mechanisms coming from acceptance and commitment therapy, mindfulness-based programmes, or problem-solving techniques. The hypothesis of whether these factors serve as moderators or mediators was beyond our scope and could inform the design of further studies.

During global crises, such as the COVID-19 pandemic, and more local emergencies, such as wars or disasters, mental health interventions must reach as many people in need as possible. Previous evidence has shown that stepped-care models and low-intensity interventions can play a role in supporting wide reach in the context of both country-wide mental health policies (eg, Improving Access to Psychological Therapies programme in the UK37) and specific responses to crises (eg, e-health psychological interventions during COVID-19,38 PM+ in humanitarian contexts18). Similarly, our findings suggest that this stepped-care programme is not only effective but also potentially scalable. Based on the large number of participants assessed for eligibility over a relatively short period of time, the programme could probably reach more people if offered at a larger scale and without the constraints of an RCT. Further, the interventions showed larger effects among participants with probable depression and/or anxiety disorders, suggesting that they could be feasibly delivered under real-world conditions—one in four HCWs are at risk of mental health disorders. Last, our estimates are robust against model misspecification, indicating that the positive effects of the interventions may persist after scaling it up.

We identified the following limitations to our study. First, we did not include an alternative intervention as an active comparator, so we cannot rule out the possibility of non-specific (eg, intervention provider) effects. We enhanced the usual care with one session of PFA during which the intervention providers offered guidance to access free mental health support resources; thus, some participants in the controlled arm could have received psychological treatment while the trial was ongoing. Second, we covered a relatively short follow-up period of 21 weeks. Although the effects of cognitive–behavioural therapies often persist over time, this still poses a major challenge for psychological treatments research,39 and we do not know whether our positive effects would decrease with longer follow-up periods. Third, we collected all outcomes using self-reports, which increases the probability of social desirability effects. In light of the barriers to conducting inperson assessments and/or collecting objective data during the COVID-19 pandemic, self-reports may have still been the most feasible method to answer our main research question.40 Fourth, although we measured PTSD symptoms, we are not sure about the actual exposure of our sample to traumatic events, which hinders the interpretation of our findings. However, the finding that the stepped-care programme improves highly distressing and invalidating symptoms, such as those of PTSD, may have important implications for both the design of trauma-focused interventions and the allocation of treatment resources. Fifth, most participants were women, which reflects the gender distribution among HCWs but limits generalisation to other genders. Last, as we reviewed and coded only 10% of the DWM and PM+ calls as part of the quality assessment of the trial, we may neglect some significant deviations from the intervention protocols—of note, most of these calls were randomly chosen, which reduces the chance of intervention providers selecting calls with good performances only. Recommendations for further studies include using active comparators, such as the DWM not followed by PM+, and exploring the effect of the interventions across settings, such as healthcare systems outside Western Europe, and over more extended periods.

Clinical implications

Our findings provide initial evidence on how to support HCWs’ mental health during the COVID-19 pandemic and indirect evidence for other public health emergencies and crisis settings. Our treatment estimates are larger than those of similarly designed studies using stepped-care programmes41 42 and PM+,18 22 and our harm assessment suggests that the intervention is safe. These robust results and the early indicators of potential scalability can already inform occupational and public health actions and policies. Importantly, such actions must always be part of overarching policies ensuring appropriate working conditions for HCWs.

Acknowledgments

The authors wish to thank Mark van Ommeren and Kenneth Carswell for their valuable contributions throughout the study.

Footnotes

Collaborators: The RESPOND Consortium includes the following collaborators: Beatriz Rodríguez-Vega, Eduardo Fernández-Jiménez, Javier Curto-Ramos, Iker Louzao, Fernando Arias-Vicente, Laura Castilla-Rodríguez, Álvaro de-Vicente-Blanco, Andrea Fernández-López, Blanca García-Vázquez, Luis Heredia-Castro, Paula Ibáñez-Mendoza, Cristina Martín-Madrigal, Beatriz Orgaz-Álvarez, Irene Pérez-de-Ciriza-Galarza, Miguel Velasco-Santos, Santiago Palomo Conti, Paula Cristóbal, Paula Arin Gonzalez, Laura Sanchez Rodriguez, Alba Jimenez Lafuente, Aida Fernandez Sanz, Elisabet Salomon Mallat, Maria Roura Adserias, Maria Porcel, Henrik Walter, Ellenor Mittendorfer-Rutz, Antonio Lora, Giovanni Corrao and Brenda Penninx.

Contributors: RM: conceptualisation, methodology, software, validation, formal analysis, data curation, writing—original draft, visualisation, project administration. MF-N: conceptualisation, methodology, software, formal analysis, investigation, data curation, writing—review and editing, visualisation, project administration. KRM and AM-M: investigation, software, writing—review and editing. M-FB-O and CB: methodology, writing—review and editing, visualisation. IG-V: validation, formal analysis, writing—review and editing. AM-S, SA-O and RV: validation, investigation, writing—review and editing. CB, MP, MM, RK, RB, VL and DM: conceptualisation, writing—review and editing, funding acquisition. MS: conceptualisation, writing—review and editing, supervision, project administration, funding acquisition. PP-R, PN, A-LP, NF, and ABW: conceptualisation, writing—review and editing. JU: conceptualisation, resources, writing—review and editing. JLA-M and JMH: conceptualisation, methodology, resources, writing—review and editing, visualisation, supervision, project administration, funding acquisition. JLA-M: manuscript guarantor. RESPOND consortium: conceptualisation, investigation; BR-V, EF-J, JC-R, IL, FA-V, LC-R, Ád-V-B, AF-L, BG-V, LH-C, PI-M, CM-M, BO-Á, IP-d-C-G, MV-S, SPC, PC, PAG, LSR, AJL, AFS, ESM, MRA, MP: investigation; HW, EM-R, AL, GC and BP: funding acquisition.

Funding: The RESPOND project was funded by the European Commission under Horizon 2020 - the Framework Programme for Research and Innovation (2014–2020) (grant number: 101016127). The work of RM was funded by the State Plan for Scientific and Technical Research and Innovation 2021–2023 of the Spanish Agencia Estatal de Investigación (FJC2021-047610-I), the Acción Estratégica en Salud programme of the Instituto de Salud Carlos III (CD22/00061), and the Centro de Investigación Biomédica en Red (CB/07/09/0013). The work of MF-N was funded by the Acción Estratégica en Salud programme of the Instituto de Salud Carlos III (CD20/00036).

Competing interests: A-LP, AM-M, CB, DM, JLA-M and MS received support from the European Commission for the submitted work (grant number: 101016127). MS is president of the International Society for Traumatic Stress Studies. RK has received personal advisory honoraria from Joy Ventures (Herzlia, Israel) and is secretary of the International Resilience Alliance (Mainz, Germany).

Provenance and peer review: Not commissioned; externally peer reviewed.

Supplemental material: This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Contributor Information

the RESPOND Consortium:

Beatriz Rodríguez-Vega, Eduardo Fernández-Jiménez, Javier Curto-Ramos, Iker Louzao, Fernando Arias-Vicente, Laura Castilla-Rodríguez, Álvaro de-Vicente-Blanco, Andrea Fernández-López, Blanca García-Vázquez, Luis Heredia-Castro, Paula Ibáñez-Mendoza, Cristina Martín-Madrigal, Beatriz Orgaz-Álvarez, Irene Pérez-de-Ciriza-Galarza, Miguel Velasco-Santos, Santiago Palomo Conti, Paula Cristóbal, Paula Arin Gonzalez, Laura Sanchez Rodriguez, Alba Jimenez Lafuente, Aida Fernandez Sanz, Elisabet Salomon Mallat, Maria Roura Adserias, Maria Porcel, Henrik Walter, Ellenor Mittendorfer-Rutz, Antonio Lora, Giovanni Corrao, and Brenda Penninx

Ethics statements

Patient consent for publication

Consent obtained directly from patient(s).

Ethics approval

This study involves human participants. The protocol obtained ethics approval from the ethics committees at Hospital Universitario La Paz in Madrid (identifier PI-4857) and Parc Sanitari Sant Joan de Déu in Barcelona (identifier PIC-129-21). The study was conducted in accordance with the Declaration of Helsinki. Participants gave informed consent to participate in the study before taking part.

References

  • 1. Aymerich C, Pedruzo B, Pérez JL, et al. COVID-19 pandemic effects on health worker’s mental health: systematic review and meta-analysis. Eur Psychiatry 2022;65:e10. 10.1192/j.eurpsy.2022.1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2. Chen Y, Wang J, Geng Y, et al. Meta-analysis of the prevalence of anxiety and depression among frontline Healthcare workers during the COVID-19 pandemic. Front Public Health 2022;10:984630. 10.3389/fpubh.2022.984630 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3. Ghahramani S, Kasraei H, Hayati R, et al. Health care workers’ mental health in the face of COVID-19: a systematic review and meta-analysis. Int J Psychiatry Clin Pract 2022:1–10. 10.1080/13651501.2022.2101927 [DOI] [PubMed] [Google Scholar]
  • 4. Andhavarapu S, Yardi I, Bzhilyanskaya V, et al. Post-traumatic stress in Healthcare workers during the COVID-19 pandemic: a systematic review and meta-analysis. Psychiatry Res 2022;317:114890. 10.1016/j.psychres.2022.114890 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5. Alonso J, Vilagut G, Alayo I, et al. Mental impact of COVID-19 among Spanish healthcare workers. A large longitudinal survey. Epidemiol Psychiatr Sci 2022;31:e28. 10.1017/S2045796022000130 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6. Dincer B, Inangil D. The effect of emotional freedom techniques on nurses' stress, anxiety, and burnout levels during the COVID-19 pandemic: a randomized controlled trial. Explore (NY) 2021;17:109–14. 10.1016/j.explore.2020.11.012 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7. Yıldırım D, Çiriş Yıldız C. The effect of mindfulness-based breathing and music therapy practice on nurses’ stress, work-related strain, and psychological well-being during the COVID-19 pandemic: a randomized controlled trial. Holist Nurs Pract 2022;36:156–65. 10.1097/HNP.0000000000000511 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8. Fiol-DeRoque MA, Serrano-Ripoll MJ, Jiménez R, et al. A mobile phone–based intervention to reduce mental health problems in health care workers during the COVID-19 pandemic (psycovidapp): randomized controlled trial. JMIR Mhealth Uhealth 2021;9:e27039. 10.2196/27039 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9. Ferreres-Galán V, Navarro-Haro MV, Peris-Baquero Ó, et al. Assessment of acceptability and initial effectiveness of a unified protocol prevention program to train emotional regulation skills in female nursing professionals during the COVID-19 pandemic. Int J Environ Res Public Health 2022;19:5715. 10.3390/ijerph19095715 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10. Fong JSY, Hui ANN, Ho KM, et al. Brief mindful coloring for stress reduction in nurses working in a Hong Kong hospital during COVID-19 pandemic: a randomized controlled trial. Medicine (Baltimore) 2022;101:e31253. 10.1097/MD.0000000000031253 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11. Keng SL, Chin JWE, Mammadova M, et al. Effects of mobile app-based Mindfulness practice on healthcare workers: a randomized active controlled trial. Mindfulness (N Y) 2022;13:2691–704. 10.1007/s12671-022-01975-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12. Gupta S, Kumar M, Rozatkar AR, et al. Feasibility and effectiveness of telecounseling on the psychological problems of frontline healthcare workers amidst COVID-19: a randomized controlled trial from central India. Indian Journal of Psychological Medicine 2021;43:343–50. 10.1177/02537176211024537 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13. Gnanapragasam SN, Tinch-Taylor R, Scott HR, et al. Multicentre, England-wide randomised controlled trial of the ‘foundations’ smartphone application in improving mental health and well-being in a healthcare worker population. Br J Psychiatry 2023;222:58–66. 10.1192/bjp.2022.103 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14. Pollock A, Campbell P, Cheyne J, et al. Interventions to support the resilience and mental health of frontline health and social care professionals during and after a disease outbreak, epidemic or pandemic: a mixed methods systematic review. Cochrane Database Syst Rev 2020;11:CD013779. 10.1002/14651858.CD013779 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15. Palmer J, Ku M, Wang H, et al. Public health emergency and psychological distress among healthcare workers: a scoping review. BMC Public Health 2022;22:1396. 10.1186/s12889-022-13761-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16. Epping-Jordan JE, Harris R, Brown FL, et al. Self-help plus (SH+): a new WHO stress management package. World Psychiatry 2016;15:295–6. 10.1002/wps.20355 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17. Dawson KS, Bryant RA, Harper M, et al. Problem management plus (PM+): a WHO transdiagnostic psychological intervention for common mental health problems. World Psychiatry 2015;14:354–7. 10.1002/wps.20255 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18. Rahman A, Khan MN, Hamdani SU, et al. Effectiveness of a brief group psychological intervention for women in a post-conflict setting in Pakistan: a single-blind, cluster, randomised controlled trial. Lancet 2019;393:1733–44. 10.1016/S0140-6736(18)32343-2 [DOI] [PubMed] [Google Scholar]
  • 19. Acarturk C, Uygun E, Ilkkursun Z, et al. Effectiveness of a WHO Self‐Help psychological intervention for preventing mental disorders among Syrian refugees in Turkey: a randomized controlled trial. World Psychiatry 2022;21:88–95. 10.1002/wps.20939 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20. de Graaff AM, Cuijpers P, Twisk JWR, et al. Peer-provided psychological intervention for Syrian refugees: results of a randomised controlled trial on the effectiveness of problem management plus. BMJ Ment Health 2023;26:e300637. 10.1136/bmjment-2022-300637 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21. Riello M, Purgato M, Bove C, et al. Effectiveness of self-help plus (SH+) in reducing anxiety and post-traumatic symptomatology among care home workers during the COVID-19 pandemic: a randomized controlled trial. R Soc Open Sci 2021;8:210219. 10.1098/rsos.210219 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22. Bryant RA, Dawson KS, Keyan D, et al. Effectiveness of a videoconferencing-delivered psychological intervention for mental health problems during COVID-19: a proof-of-concept randomized clinical trial. Psychother Psychosom 2022;91:63–72. 10.1159/000520283 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23. Mediavilla R, Monistrol-Mula A, McGreevy KR, et al. Mental health problems and needs of frontline healthcare workers during the COVID-19 pandemic in Spain: a qualitative analysis. Front Public Health 2022;10:956403. 10.3389/fpubh.2022.956403 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24. Mediavilla R, McGreevy KR, Felez-Nobrega M, et al. Effectiveness of a stepped-care programme of Internet-based psychological interventions for Healthcare workers with psychological distress: study protocol for the RESPOND healthcare workers randomised controlled trial. DIGITAL HEALTH 2022;8:205520762211290. 10.1177/20552076221129084 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25. Kessler R, Mroczek D. Final versions of our non-specific psychological distress scale. Ann Arbor (MI): University of Michigan, 1994. [Google Scholar]
  • 26. World Health Organization, War Trauma Foundation, World Vision International . Internet. Psychological first aid: facilitator’s manual for orienting field workers. Geneva: World Health Organization, 2013. Available: https://apps.who.int/iris/handle/10665/102380 [Google Scholar]
  • 27. Cuijpers P, Heim E, Ramia JA, et al. Guided digital health intervention for depression in Lebanon: randomised trial. Evid Based Ment Health 2022;25:e34–40. 10.1136/ebmental-2021-300416 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28. Kroenke K, Wu J, Yu Z, et al. The patient health questionnaire anxiety and depression scale (PHQ-ADS): initial validation in three clinical trials. Psychosom Med 2016;78:716–27. 10.1097/PSY.0000000000000322 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29. Kroenke K, Spitzer RL, Williams JBW. The PHQ-9. J Gen Intern Med 2001;16:606–13. 10.1046/j.1525-1497.2001.016009606.x [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30. Spitzer RL, Kroenke K, Williams JBW, et al. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med 2006;166:1092–7. 10.1001/archinte.166.10.1092 [DOI] [PubMed] [Google Scholar]
  • 31. Muñoz-Navarro R, Cano-Vindel A, Medrano LA, et al. Utility of the PHQ-9 to identify major depressive disorder in adult patients in Spanish primary care centres. BMC Psychiatry 2017;17:291. 10.1186/s12888-017-1450-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32. García-Campayo J, Zamorano E, Ruiz MA, et al. Cultural adaptation into Spanish of the generalized anxiety disorder-7 (GAD-7) scale as a screening tool. Health Qual Life Outcomes 2010;8:8. 10.1186/1477-7525-8-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33. Price M, Szafranski DD, van Stolk-Cooke K, et al. Investigation of abbreviated 4 and 8 item versions of the PTSD checklist 5. Psychiatry Res 2016;239:124–30. 10.1016/j.psychres.2016.03.014 [DOI] [PubMed] [Google Scholar]
  • 34. Miles JNV, Marshall GN, Schell TL. Spanish and English versions of the PTSD checklist–civilian version (PCL-C): testing for differential item functioning. J Trauma Stress 2008;21:369–76. 10.1002/jts.20349 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35. Gellatly J, Bower P, Hennessy S, et al. What makes self-help interventions effective in the management of depressive symptoms? Meta-analysis and meta-regression. Psychol Med 2007;37:1217–28. 10.1017/S0033291707000062 [DOI] [PubMed] [Google Scholar]
  • 36. Spek V, Cuijpers P, Nyklícek I, et al. Internet-based cognitive behaviour therapy for symptoms of depression and anxiety: a meta-analysis. Psychol Med 2007;37:319–28. 10.1017/S0033291706008944 [DOI] [PubMed] [Google Scholar]
  • 37. Wakefield S, Kellett S, Simmonds-Buckley M, et al. Improving access to psychological therapies (IAPT) in the United Kingdom: a systematic review and meta-analysis of 10-years of practice-based evidence. Br J Clin Psychol 2021;60:1–37. 10.1111/bjc.12259 [DOI] [PubMed] [Google Scholar]
  • 38. Witteveen AB, Young S, Cuijpers P, et al. Remote mental health care interventions during the COVID-19 pandemic: an umbrella review. Behav Res Ther 2022;159:104226. 10.1016/j.brat.2022.104226 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39. Holmes EA, Ghaderi A, Harmer CJ, et al. The Lancet psychiatry Commission on psychological treatments research in tomorrow’s science. Lancet Psychiatry 2018;5:237–86. 10.1016/S2215-0366(17)30513-8 [DOI] [PubMed] [Google Scholar]
  • 40. Mercieca-Bebber R, King MT, Calvert MJ, et al. The importance of patient-reported outcomes in clinical trials and strategies for future optimization. Patient Relat Outcome Meas 2018;9:353–67. 10.2147/PROM.S156279 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41. van Straten A, Hill J, Richards DA, et al. Stepped care treatment delivery for depression: a systematic review and meta-analysis. Psychol Med 2015;45:231–46. 10.1017/S0033291714000701 [DOI] [PubMed] [Google Scholar]
  • 42. Ho FYY, Yeung WF, Ng THY, et al. The efficacy and cost-effectiveness of stepped care prevention and treatment for depressive and/or anxiety disorders: a systematic review and meta-analysis. Sci Rep 2016;6:29281. 10.1038/srep29281 [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary data

bmjment-2023-300697supp001.pdf (3.6MB, pdf)

Supplementary data

bmjment-2023-300697supp002.pdf (822.6KB, pdf)


Articles from BMJ Mental Health are provided here courtesy of BMJ Publishing Group

RESOURCES