Skip to main content
NCBI home page
Cambridge Open Access logoLink to Cambridge Open Access
. 2022 Nov 25;53(13):6232–6241. doi: 10.1017/S0033291722003531

Automated mobile virtual reality cognitive behavior therapy for aviophobia in a natural setting: a randomized controlled trial

T Donker 1,2,✉, JR Fehribach 1, C van Klaveren 3,4, I Cornelisz 3,4, M B J Toffolo 1, A van Straten 1, J-L van Gelder 5,6
PMCID: PMC10520596  PMID: 36426618

Abstract

Background

Access to evidence-based psychological treatment is a challenge worldwide. We assessed the effectiveness of a fully automated aviophobia smartphone app treatment delivered in combination with a $5 virtual reality (VR) viewer.

Methods

In total, 153 participants from the Dutch general population with aviophobia symptoms and smartphone access were randomized in a single-blind randomized controlled trial to either an automated VR cognitive behavior therapy (VR-CBT) app treatment condition (n = 77) or a wait-list control condition (n = 76). The VR-CBT app was delivered over a 6-week period in the participants' natural environment. Online self-report assessments were completed at baseline, post-treatment, at 3-month and at 12-month follow-up. The primary outcome measure was the Flight Anxiety Situations Questionnaire (FAS). Analyses were based on intent-to-treat.

Results

A significant reduction of aviophobia symptoms at post-test for the VR-CBT app compared with the control condition [p < 0.001; d = 0. 98 (95% CI 0.65–1.32)] was demonstrated. The dropout rate was 21%. Results were maintained at 3-month follow-up [within-group d = 1.14 (95% CI 0.46–1.81)] and at 12-month follow-up [within-group d = 1.12 (95% CI 0.46–1.79)]. Six participants reported adverse effects of cyber sickness symptoms.

Conclusions

This study is the first to show that fully automated mobile VR-CBT therapy delivered in a natural setting can maintain long-term effectiveness in reducing aviophobia symptoms. In doing so, it offers an accessible and scalable evidence-based treatment solution that can be applied globally at a fraction of the cost of current treatment alternatives.

Key words: Automated, aviophobia, smartphone application, specific phobia, virtual reality

Introduction

It is estimated that one in 10 people worldwide require mental health care at any one point in time, yet only nine mental health workers are available for every 10 000 people globally (World Health Organization, 2018). If anything, the coronavirus disease 2019 (COVID-19) pandemic has further exacerbated this already challenging state of affairs as the prevalence of mental health problems has increased dramatically (Winkler et al., 2020), while access to evidence-based treatment has declined (World Health Organization, 2020).

With the post-pandemic reopening of societies, novel challenges relating to ‘re-entry anxiety’ (American Psychological Association, 2021) have appeared. Re-entry anxiety, or spontaneous recovery, refers to the re-emergence of conditioned responding to an extinguished conditioned stimulus (Craske, Treanor, Conway, Zbozinek, & Vervliet, 2014). Specifically, the lack of exposure to feared objects or situations (e.g. social interactions, flying) has resulted in increased anxiety levels in phobics and an overall increase in the prevalence of phobias (American Psychological Association, 2021; Winkler et al., 2020). In addition, phobias may increase the risk of developing other anxiety disorders (Trumpf, Margraf, Vriends, Meyer, & Becker, 2010) and depression (Choy, Fyer, & Goodwin, 2007). Hence, the need for scalable solutions to meet these challenges is acute. In this study, we test the effectiveness of a low-cost, scalable, and standalone virtual reality exposure therapy (VRET) for one of the most prevalent phobias, namely aviophobia (Curtis, Magee, Eaton, Wittchen, & Kessler, 1998).

VRET has shown effectiveness in reducing aviophobia symptoms (Cardoş, David, & David, 2017; Fodor et al., 2018; Krijn et al., 2007; Rothbaum et al., 2006). For example, Rothbaum et al. (2006) were one of the first to compare VRET to standard in vivo treatment and a waitlist control condition in a randomized controlled trial (RCT) among 75 participants and concluded that VR exposure was superior to waitlist, and essentially equivalent to exposure in vivo. Yet, despite promising effects, clinical virtual reality (VR) has faced implementation challenges due to high costs, technical obstacles, and lack of availability (Geraets, Van der Stouwe, Pot-Kolder, & Veling, 2021). The development of consumer VR platforms has led to the emergence of automated VR treatment, with encouraging results (Bentz et al., 2021; Freeman et al., 2018; Miloff et al., 2019). Generally, specialized equipment remains required, however, and prior efforts have all involved contact with a research team or therapist, thus limiting their scalability. One recent study, however, demonstrated that automated VR-CBT for treating acrophobia in a natural setting, without human contact is feasible and effective, also at 3-month follow-up (Donker et al., 2019). In addition, the reduction in acrophobia symptoms was larger when the feeling of being present in the VR and user-friendliness were higher (Donker et al., 2019).

To demonstrate the potential of fully automated mobile virtual reality cognitive behavior therapy (VR-CBT) for more complex disorders, and also its long-term potential, we examined the effectiveness of a VR-CBT app for aviophobia. Aviophobia is a heterogeneous condition with large variation in onset and acquisition characteristics and with high rates of comorbidity with other phobias and anxiety disorders. This makes aviophobia more difficult to treat than other phobias (Van Almen & Van Gerwen, 2013). To ensure scalability, our treatment relies on low-cost and widely available equipment – a participant's own smartphone and a $5 cardboard VR viewer – and was delivered in a natural setting without human contact.

We hypothesized that the VR-CBT app would demonstrate a greater reduction in flight anxiety, general anxiety, and depressive symptoms from pre- to post-treatment compared to the wait-list control condition, and that these effects would be maintained at 3-month and 12-month follow-up. Furthermore, we hypothesized that the app would be rated as user-friendly and satisfactory, and that a higher sense of presence in VR, higher perceived realism of VR, and higher user-friendliness would all be associated with a larger reduction in fear-of-flying symptoms at post-test.

Method

Participants and design

Participants for this two-arm, single-blind RCT were recruited from the Dutch general population through the placement of advertisements on websites and local media. Data were collected pre-treatment, post-treatment, and at 3-month and 12-month follow-up. Ethics approval was granted by the Medical Ethical Committee of the Vrije Universiteit Medical Center (registration number: 2019-321). The study was prespecified in the trial protocol (Fehribach et al., 2021). No changes to methods or trial outcomes were made after trial commencement and no data were analyzed before study completion. Results are published in accordance with CONSORT guidelines for RCTs (Schulz, Altman, & Moher, 2010). Eligible for inclusion were individuals who scored ⩾ 56 (indicating mild flight anxiety) on the Flight Anxiety Situations Questionnaire (FAS) (Van Gerwen, Spinhoven, Van Dyck, & Diekstra, 1999), had smartphone access (⩾ iPhone 5, or Android with ⩾ Lollipop 5.1 OS, gyroscope, and 4.7–5.5 inches screen size), were between the ages of 18 and 64, and provided written informed consent. Individuals who were currently being treated for any phobia, were taking psychotropic medication (unless dosage had been stable for ⩾ 3 months and no changes were planned during study period), and/or had insufficient Dutch language proficiency were excluded from participation. Participants provided written informed consent after receiving a complete description of the study. This study is registered in the Netherlands Trial Registry (NL 70238.029.19., https://www.trialregister.nl/trial/8257). All procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008.

Randomization and masking

A randomization list using Sealed Envelope™ Random Allocation Software (https://www.sealedenvelope.com/simple-randomiser/v1/lists) was created and kept by an independent researcher who revealed the next randomization outcome after every inclusion, thus ensuring that the research team was blind to treatment allocation. Block randomization (using blocks of 6, 8, 10, and 12 participants) with 1:1 allocation was used. The allocation sequence was concealed from the researchers. Randomization was performed after pre-treatment assessment. Participants were aware of which arm they had been allocated to. All self-report measures were completed online.

Intervention: VR-CBT app

The treatment was delivered through a VR-CBT app that participants could download on their own smartphone in combination with a $5 VR viewer. The VR-CBT app was launched as a beta test in both the Google Play store (Android) and the App store (IOS), meaning that the app could only be accessed through an URL and a unique pin code for each participant that was provided by the research team. Technical questions from participants were answered by e-mail by a research team member.

The VR-CBT app consisted of six CBT-based animated modules that included a virtual therapist (modeled on the first author). The modules took between 5 and 40 min each to complete. The app also included severalgamified immersive VR environments and interactions such as checking in at an airport, the boarding process, finding one's seat, taking off, flying with turbulence, and landing (see online Supplementary Fig. SF1 and a video SV1 in the online supplement). The scenarios covered the entire exposure spectrum of flight anxiety and was developed based on input from a user-test among 15 participants suffering from flight anxiety. Examples of (gamified) interactions included locating objects that passengers brought onto the airplane (scavenger hunt) and comforting another nervous passenger. In addition, the app included sounds typical of airplane travel (e.g. engine roar, announcements from the cabin crew, chattering voices). Participants were able to progress to subsequent exposure levels if they rated their anxiety as ⩽3 on a ten-point scale after finishing a level. Participants who rated the VR level with a score of >3 received a message explaining that they should practice the same level again before moving on to the next level. Virtual environment navigation occurred via gaze control. See Fehribach et al. (2021) for intervention details. The scenarios were developed by the first and last author (TD, JLvG) in collaboration with app developers and designers. The VR environment was developed with the Unity game engine (version 2018.4.1f1; Unity Technologies). Participants followed the therapy in their own time and natural environment and were encouraged to complete it within 6 weeks.

The VR-CBT app is available in Google Play (for Android devices) and the App Store (for Apple smartphones) for about $10. The app runs on nearly all smartphones from 2016 or newer (as long as they are equipped with a gyroscope). In terms of hardware, only a rudimentary VR viewer is required (cost approx. $5–10).

Outcomes

Primary outcome measure

Flight Anxiety Situations Questionnaire

The 32-item Flight Anxiety Situations Questionnaire (FAS) (Van Gerwen et al., 1999) was used to assess flight anxiety (item score 1–5; total score range 32–160) at base-line, post-test and follow-up. The FAS is reliable and has excellent validity with a cut-off score of 56 (Nousi, Van Gerwen, & Spinhoven, 2008). The Cronbach Alpha in this study was 0.97.

Secondary outcome measures

Flight Anxiety Modality Questionnaire

The 18-item Flight Anxiety Modality Questionnaire (FAM; Van Gerwen et al., 1999) targets flight anxiety as well (item score 1–5; total score range 18–90) and was included to test for the robustness of the results. The reliability is good (Nousi et al., 2008). The FAM was assessed at all time-points.

Beck Anxiety Inventory

The Beck Anxiety Inventory (BAI; Beck, Epstein, Brown, and Steer, 1988), Dutch translation (Beck & Steer, 2015) consists of 21 items (item score 0–4, total score range 0 to 63) measuring general anxiety and has good reliability and validity (Brown, Beck, Newman, Beck, & Tran, 1997). The BAI was assessed at all time-points.

Patient health questionnaire

The 9-item Patient Health Questionnaire (PHQ-9) (Kroenke, Spitzer, & Williams, 2001), Dutch version was used to assess depressive symptoms at all time-points. Validity is good (Wittkampf, Naeije, Schene, Huyser, & van Weert, 2007).

Web screening questionnaire

Three items from the baseline Web Screening Questionnaire (Donker, Van Straten, Marks, & Cuijpers, 2009) [items pertaining to panic disorder, agoraphobia, and obsessive-compulsive disorder (OCD)], were assessed to examine whether they influenced post-test flight anxiety outcomes. Reliability and validity of these items was found to be good (Donker et al., 2009).

Credibility/Expectancy questionnaire

The 6-item Credibility/Expectancy Questionnaire (CEQ; Devilly and Borkovec, 2000), Dutch version (Mertens, Moser, Verbunt, Smeets, & Goossens, 2017) was used to asses self-reported treatment expectations at baseline (intervention group only). Reliability and validity of CEQ was found to be good (Devilly & Borkovec, 2000; Mertens et al., 2017).

Client satisfaction questionnaire

The Dutch version of the 8-item Client Satisfaction Questionnaire (CSQ-8; Attkisson & Greenfield, 2004; de Brey, 1983) assessed global client treatment satisfaction at post-test (intervention group only) using scale response options from 1–4 (total score range: 8–32) and has found to be reliable and valid (de Brey, 1983).

System usability scale

The 10-item System Usability Scale (SUS; Bangor, Kortum, and Miller, 2008) measured user-friendliness of the VR-CBT app at post-test [item score range 0–5, total score was calculated and converted to range 0–100 (44)] with scores of at least 70 are considered as passable and better products ranging into the 80s. Reliability and validity are good (Bangor et al., 2008). This measure was administered to the intervention group only.

Igroup presence questionnaire

The 14-item Dutch version of the Igroup Presence Questionnaire (IPQ; (Schubert, Friedmann, & Regenbrecht, 2001) assessed realism and the feeling of ‘presence’ or immersion in the VR environments (item score range −3 to 3; total score range −42–42). The IPQ demonstrated reliability ((Schubert et al., 2001) and was administered at post-test to the intervention group only.

Additional questions

Professional aviophobia treatment usage, assessed at each time point, and vision and hearing impairment were included as control variables. Adverse events were inquired about at post-test and follow-up.

All assessments were self-report measures completed online in participants' natural environment and outside researchers' presence and were programmed with Survalyzer software (http://www.survalyzer.com).

Statistical analyses

A full statistical analysis plan was designed before the trial (Fehribach et al., 2021). Power calculations were based on the FAS, the primary outcome measure. A previous meta-analysis that examined the effect of VRET on fear-of-flying symptoms (Fodor et al., 2018) yielded an effect size of Cohen's d = 0.82. However, because the VR-CBT app is self-guided and uses rudimentary equipment, a more conservative estimate of Cohen's d = 0.70 was used. We determined that 34 participants per group would provide 80% power at a 5% significance level (two-tailed). With an anticipated dropout rate of 40% (see Donker et al., 2019), 114 participants were required (n = 57 per condition).

Baseline characteristics were tabulated for each condition. Differences across the two groups were assessed with chi-square tests or analyses of variance, as appropriate. To assess whether dropout cases (participants who discontinued treatment after commencing) and complete cases (participants who completed post-test) were non-random, we constructed a balancing table comparing the background characteristics, pre-scores, and other covariates of participants with and without missing outcome observations. Missing outcome values for the dropout sample were imputed using a multiple imputation (MI) procedure that exploited pre-scores and a set of pre-specified background characteristics (gender, age, severity of symptoms, and a dummy variable for whether a participant had missing values on one or more of the background characteristics). Ordinary least squares regression models with pre-scores and background characteristics included were used on an intention-to-treat (ITT) basis to estimate the treatment effect. In addition to standardized mean differences (Cohen's d), we conducted two robustness analyses. First, because non-random sample attrition may bias the estimated treatment effects (Cornelisz, Cuijpers, Donker, & Van Klaveren, 2020; Donker et al., 2019), nonparametric treatment effect bounds were estimated using the Random Forest Lee Bounds (RFLB) procedure (Lee, 2009; Schonlau & Zou, 2020). Second, MI using an iterative Markov chain Monte Carlo (MCMC) method (Hamra, MacLehose, & Richardson, 2013) based on initial treatment assignment was conducted to deduce the statistical significance of the reported differences between conditions. Furthermore, potential heterogeneity and mechanisms of treatment were analyzed by estimating the variation in treatment effects with pre-scores, general anxiety, user-friendliness, treatment expectation/credibility and satisfaction, and presence. To investigate whether fear-of-flying symptom severity at baseline predicted outcome, we analyzed the predicted reductions in post-test score for 10-point bins of pre-scores using regression analysis. A sensitivity analysis was conducted by examining whether the results among complete cases (participants who completed post-test and/or follow-ups) differed from the results of the ITT analysis. Clinically meaningful changes on the FAS were analyzed for the ITT sample using the ‘method (C)’ clinically significant change (CSC) formula described by Evans (Evans, 1998) with an FAS score of M = 41.55 (s.d.: 11.64) (Skolnick, Schare, Wyatt, & Tillman, 2012). The reliable change criterion was computed based on the standard deviation of the FAS pre-test scores and the pre-test Cronbach's alpha of the FAS (α = 0.97) (ITT sample). This yielded a standard error of change of 4.67 and, in turn, a reliable change criterion of 9.15 (4.67 × 1.96) points on the FAS (Christensen & Mendoza, 1986; Jacobson & Truax, 1991). Finally, using the formula by Furukawa and Leucht (2011), the number-needed-to-treat (NNT) was assessed for both the ITT sample and the complete-case sample based on the size of the effects observed on the FAS. For all analyses, two-sided p values < 0.05 were taken to indicate statistical significance. Stata (version 16; StataCorp) was used for the analyses.

Results

Participants

Enrollment commenced on 9 November 2019 and ceased on 7 May 2020. Data analysis began in December 2020.

Between 9 November 2019 and 7 May 2020, 251 individuals self-referred online for the study, of whom 204 proceeded to screening. Of these, 50 were ineligible based on the exclusion criteria. Accordingly, between 15 November 2019 and 7 May 2020, 154 participants were randomized to the VR-CBT app treatment condition (n = 77) or the wait-list control condition (n = 77). However, due to a technical error, one participant in the control condition was excluded after randomization, but before treatment commencement, because the participant scored below the cut-off on the baseline FAS. The last participant completed 12-month follow-up data on 20 July 2021. See Fig. 1 for participant flowchart.

Fig. 1.

Fig. 1.

Open in a new tab

CONSORT diagram of participant flow through the trial. Abbreviations: FAS, Flight Anxiety Situations Questionnaire; VR, virtual reality.

Participants in the two conditions were balanced with regard to the baseline characteristics and demographic variables (Table 1). Most participants had experienced fear-of-flying symptoms for longer than 5 years (121 of 153, 79%). Sixteen participants (10%) had no flight experience whatsoever.

Table 1.

Demographic and baseline characteristicsa

VR treatment group (n = 77) Wait-list control group (n = 76)
Measure Mean s.d. Mean SD p-value diff
Age (years) 41 12.50 43 11.79 0.314
n % n %
Female 64 83.11 63 82.89 0.696
Education
Primary 1 1.30 1 1.32 0.993
Secondary 9 11.69 12 15.79 0.464
Postsecondary 67 87.01 63 15.79 0.479
Employment status
Full-time employed 36 46.75 32 42.11 0.566
Part-time employed 35 45.45 37 48.68 0.691
Unemployed 6 7.79 7 9.21 0.755
Marital status
Single 10 12.99 8 10.53 0.639
Married, living together 56 72.73 49 64.47 0.274
Other 11 14.29 19 25.00 0.096
Psychotropic medication 7 9.09 4 5.26 0.363
Duration of fear of flying
< 6 months 0 0.00 0.00 0.00 n/a
6–12 months 2 2.60 2 2.63 0.990
1–5 years 13 16.88 15 19.74 0.651
> 5 years 62 80.52 59 77.63 0.663
Never flown in an airplane 10 12.99 6 7.89 0.307
Smartphone type
iPhone 37 48.05 40 52.63 0.574
Android 40 51.95 36 47.37 0.574
Open in a new tab
a

Abbreviations: VR, virtual reality.

Treatment adherence and attrition

Of the 77 participants in the intervention group, 54 (70%) completed the post-test, 37 (49%) completed the 3-month follow-up, and 40 (52%) completed the 12-month follow-up; in the wait-list group, 91% (69 of 76) completed the post-test. Dropout (32 of 153, 21%) was not related to background characteristics, pre-scores, or other covariates except for educational level. Specifically, a higher proportion of participants who completed only secondary education did not drop out (z = 2.44, p = 0.015). Also, a higher proportion of participants who completed postsecondary education did drop out (z = −2.00, p = 0.046) (online Supplementary Table ST1 in the online supplement). Therefore, outcome values were not missing completely at random. This was addressed by using RFLB and MI.

Effectiveness: Intention-to-treat analysis

Compared to the control condition, the VR-CBT app treatment condition exhibited a significant reduction in fear-of-flying symptoms as measured by the primary outcome measure (FAS) at post-test [b = −21.90 (95% CI −28.65 to −15.14), t147 = −6.44, p < 0.0001]. The effect size was d = 0.98 (95% CI 0.65–1.32). The NNT was 3.3. In addition, a significant difference at post-test between the intervention and the control group was also observed on the secondary fear-of-flying outcome measure (FAM), [b = −12.68 (95% CI −18.13 to −7.23), t147 = −4.62, p < 0.0001; d = 0.62 (95% CI 0.29–0.94)). No differences were observed for general anxiety (BAI) [b = −2.02 (95% CI −6.48–2.45), t147 = −0.90, p = 0.372; d = 0.09 (95% CI −0.23 to 0.40)] or depressive symptoms (PHQ-9) [b = −0.68 (95% CI −1.82 to 0.45), t147 = −1.20, p = 0.234; d = 0.25 (95% CI −0.07 to 0.57)]. Full results of the ITT analysis are shown in Tables 2 and 3.

Table 2.

Outcome measure scores, intention to treat (n = 153)a

VR treatment group (n = 77) Wait-list control group (n = 76)
Outcomes Mean s.d. Mean s.d.
Primary outcome
FAS
Baseline 106.66 18.32 104.32 19.83
Post-test 83.09 19.29 103.71 22.58
Secondary outcome
FAM
Baseline 69.25 15.80 68.04 17.92
Post-test 69.25 17.51 69.63 18.58
BAI
Baseline 12.96 13.51 11.59 12.05
Post-test 13.56 13.11 14.74 13.93
PHQ
Baseline 3.82 4.60 4.45 4.44
Post-test 3.39 3.34 4.30 3.99
Open in a new tab

Abbreviations: VR, virtual reality; FAS, Flight Anxiety Situations Questionnaire; FAM, Flight Anxiety Modality Questionnaire; BAI, Beck Anxiety Inventory; PHQ, Patient Health Questionnaire.

a

Inferential statistics of treatment outcome measures, single-regression-based imputation.

Table 3.

Estimated effects of VR-CBT app on fear of flying, general anxiety, and depressiona

Post-testb
Model 1 (n = 153) Model 2 (n = 121) Model 3 (n = 153)
Measure b ITT (s.e.) Adj. R p valuec Effect size (95% CI) b Lower bound (s.e.) b Upper bound (s.e.) b MI (s.e.) p valuec
Primary outcome
FAS −22.55 (2.41) 0.61 <0.0001 0.98 (0.65–1.32) −31.59 (4.46) −14.25 (4.76) −21.90 (3.40) <0.0001
Secondary outcomes
FAM −12.38 (2.04) 0.57 <0.0001 0.62 (0.29–0.94) −19.90 (3.97) −4.93 (4.05) −12.68 (2.75) <0.0001
BAI −2.16 (1.63) 0.46 0.190 0.09 (−0.23 to 0.40) −5.67 (3.24) 5.22 (2.64) −2.02 (2.25) 0.372
PHQ −0.54 (0.41) 0.55 0.194 0.25 (−0.07 to 0.57) −2.21 (0.86) 0.88 (0.66) −0.68 (0.57) 0.234
Open in a new tab
a

Abbreviations: VR, virtual reality; ITT, intent-to-treat; FAS, Flight Anxiety Situations Questionnaire; FAM, Flight Anxiety Modality Questionnaire; BAI, Beck Anxiety Inventory; PHQ, Patient Health Questionnaire; MI, multiple imputation, s.e., standard error.

b

For ethical reasons, we do not have follow-up data for the wait-list control group participants: they were given access to the VR treatment after the post-test. Hence, we could not analyze the follow-up results for the intent-to-treat sample.

c

Two-sided. Treatment effects reported in the preferred, conservative model (Model 1) are ITT effects based on single-regression-based imputation using initial assignment status, pre-score, gender, age, and a dummy variable for whether a participant had missing values on one or more background characteristics. Treatment effects reported in Model 2 are Lee Bounds estimates. Treatment effects in Model 3 are derived through multiple (50) imputations, assuming missing at random, applying multivariate normal regression with imputation by experimental group, and accommodating arbitrary missing outcome value patterns using an iterative MCMC method. Pre-score and background controls are included for improved precision of the regression point estimates reported in Models 1 and 3. Background control variables are gender, age, and a dummy variable for whether a participant had missing values on one or more background characteristics. Effect sizes reported in Models 1 and 3 are all Cohen's d for unadjusted group means based on the pre-measure–post-measure difference.

Robustness and sensitivity analyses

Because there was a differential dropout rate between the two conditions, RFLB were estimated (see Lee, 2009; Schonlau and Zou, 2020). The resulting upper and lower bounds (see Table 3, Model 2) confirmed the statistically significant treatment effect on the primary outcome measure (FAS). The regression coefficient estimated in Model 1 (b = −22.55, s.e. = 2.41) lies in the middle of the Lee Bounds.

Because one participant reported at post-test that s/he had received professional aviophobia treatment during the time between baseline and post-test, a robustness analysis was conducted by repeating the ITT analysis with this participant excluded. This did not influence the results significantly [FAS: b = −21.96 (95% CI −28.78 to −15.14), t146 = −6.39, p < 0.0001]. Nine participants indicated that they had received professional aviophobia treatment during the time between 3-month and 12-month follow-up. A robustness analysis excluding these participants indicated that this did not influence the primary outcome results significantly (p = 0.70); the within-group effect size of the 31 participants that did not receive professional aviophobia treatment between follow-ups was d = 1.06 (95% CI 0.53–1.60). As indicated by the robustness analysis, the VR-CBT app impacted the anxiety specific to aviophobia, and general anxiety did not drive the results (BAI: b = −2.02, s.e. = 2.25, p = 0.37).

Heterogeneous treatment effects and mechanisms

Baseline aviophobia symptoms influenced treatment effectiveness: participants with more severe fear-of-flying symptoms at baseline benefitted more from the VR-CBT app (b = −0.0579, s.e. = 0.173, p < 0.0001). Participants who thought that the app would be less credible before the start of the treatment (CEQ–credibility) derived less benefit than those who rated the app as more credible (Fig. 2). However, no significant differences in post-treatment FAS scores were found between those scoring high and those scoring low on treatment expectancy (CEQ–expectancy), usability of the VR-CBT app (SUS), satisfaction with the VR-CBT app (CSQ), or presence (IPQ) (Fig. 2). Likewise, having a visual impairment was not related to post-treatment FAS scores (b = −0.94, s.e. = 6.05, p = 0.89). We also explored whether underlying symptoms of panic disorder (n = 84), OCD (n = 88), and claustrophobia (n = 81) – that is, symptoms likely to influence participants' flight anxiety – affected the results. With little statistical power, no statistically significant differences in treatment effectiveness were observed for these subgroups of participants.

Fig. 2.

Fig. 2.

Open in a new tab

Heterogeneous treatment effects and mechanisms. The graphs show the change in FAS difference scores by the VR-CBT app CEQ expectancy (n = 54), CEQ credibility (n = 54), CSQ satisfaction (n = 54), FAS pre-score (n = 54), IPQ (n = 54), and SUS usability score (n = 53), from baseline to 6 weeks' post-test. Values reflect least-square means; error bars indicate standard error of the mean. Abbreviations: CEQ, Credibility Expectancy Scale; CSQ, Client Satisfaction Scale; FAS, Flight Anxiety Situations Questionnaire; IPQ, Igroup Presence Questionnaire; SUS, System Usability Scale.

Complete cases

The between-group FAS effect size for those who completed post-test was d = 0.98 (95% CI 0.60–1.36). The NNT was 3.3. In total, 41 of 54 participants (76%) demonstrated reliable change. In addition, among participants who completed post-test, 14 of 54 (26%) experienced CSC (a FAS change of ⩾ 65.80). At 3-month and 12-month follow-up, the within-group FAS effect size increased to d = 1.14 (95% CI 0.46–1.81) and d = 1.12 (95% CI 0.46–1.79) respectively. A medium effect size at 3-month and 12-month follow-up was obtained for the FAM (d = 0.78 and d = 0.67 respectively), but no significant within-group change between baseline and follow-ups was observed for general anxiety and depression (online Supplementary Table ST2 in the online supplement). The total mean IPQ score at post-test (intervention group only) was −1.30 (s.d. = 17.22, observed range: −42–42). The results on flight usage were almost certainly impacted by the COVID-19 pandemic, during which recruitment took place and during which air travel was restricted and therefore not shown.

User-friendliness, treatment satisfaction, and adverse effects

The VR-CBT app was rated as user-friendly (SUS: M = 79.38, s.d. = 13.78). Treatment expectation and credibility (CEQ) before commencing VR-CBT treatment was M = 36.94 (s.d. = 7.19; range: 14.6–53.2). Total mean treatment satisfaction (CSQ) was 22.17 (s.d. = 4.62; range: 10–32). No deterioration or negative effects as defined by Rozental et al. (2018) were identified except for six participants who reported one or more symptoms of transient cyber sickness and two participants who had VR viewer problems (i.e. missing strap).

Discussion

The results of this single-blind RCT demonstrated that a VR-CBT app using a rudimentary cardboard VR viewer and delivered in a natural setting without human contact was effective in reducing fear-of-flying symptoms at post-test and at both 3-month and 12-month follow-up. The effect sizes are similar to therapist-guided in vivo exposure for specific phobias (Wolitzky-Taylor, Horowitz, Powers, & Telch, 2008). Importantly, our results are also within the range of what is reported in meta-analyses of therapist-guided, high-end VR studies targeting fear of flying (Cardoş et al., 2017; Fodor et al., 2018). Furthermore, the app was rated as both user-friendly and satisfactory by participants, and dropout rates were similar to those reported in therapist-guided VR studies (Benbow & Anderson, 2019). No significant changes emerged for VR-CBT participants on general anxiety and depression. This is likely due to the low baseline levels of general anxiety and depression. Surprisingly, even though participants reported relatively low levels of presence, this was unrelated to treatment effectiveness. Although this finding was contrary to expectations, previous research has demonstrated mixed findings regarding the influence of presence on effectiveness (e.g. Donker et al., 2019; Fodor et al., 2018; Miloff et al., 2019; Price & Anderson, 2007). The influence of presence on treatment outcomes requires more research in order to better understand how it interacts with anxiety and treatment outcome, and to clarify the causal relationship between presence and fear (Botella, Fernández-Álvarez, Guillén, García-Palacios, & Baños, 2017). Interestingly, whereas those who rated the VR-CBT app as less credible pre-treatment, also derived less benefit from the VR-CBT app. Post-treatment satisfaction rates were unrelated to effectiveness. Moreover, treatment expectancy, user-friendliness, and underlying causes of aviophobia (e.g. panic disorder) did not influence responses on the primary outcome measure. However, power was low for these exploratory analyses, which could explain the absence of significant findings. Notably, only 26% of participants completing the app reached CSC, meaning that the majority of the participants still suffered from some degree of flight anxiety. This could be related to the complex and heterogeneous nature of the condition compared to other phobias, which makes it more difficult to treat (Van Almen & Van Gerwen, 2013). However, as the large effect size of the primary outcome measure demonstrates, participants who used the VR-CBT app derived significant benefit from it.

It has been argued that VR exposure lowers the threshold for exposure in vivo (Lindner et al., 2021). This study was carried out during the COVID-19 Pandemic. It is therefore likely that the effects of the current intervention would have been even larger if participants were able to take actual flights. In addition, because of its portability, the VR-CBT app can easily be used to practice with exposure between flights.

One of the main strengths of this trial was that the intervention took place in the participants' natural environment, thus increasing ecological validity. Furthermore, any influence of human contact on the findings was ruled out because participants did not receive any therapist guidance during treatment or when responding to assessments. Additional strengths were that the treatment had cross-platform compatibility (i.e. could be used on both iPhone and Android smartphones) and that the treatment was developed for common smartphones (up to approximately 5 years old), thereby increasing generalizability. Finally, this is the first study to demonstrate that the effects of fully automated VR-CBT in a real-life setting can be maintained long term.

This study also had several limitations. First, the Simulator Sickness Questionnaire, which is commonly used to measure cyber sickness, could not be included because it overlaps with symptoms of anxiety and might therefore not be a valid measure of cyber sickness when used in studies of phobia treatment (Donker et al., 2019). Consequently, we were unable to assess participants' cyber sickness levels during the trial. We could therefore not establish whether low presence levels reported in this study might have been caused by cyber sickness. However, only six participants reported mild symptoms of cyber sickness at post-test. Second, our data relied entirely on self-report measures. However, previous research comparing self-report and behavioral measures revealed no differences between them (Morina, Ijntema, Meyerbröker, & Emmelkamp, 2015). Finally, the research design allowed for assessing the effectiveness of the therapy as a whole but not for assessing specific components, as our sample size precluded the possibility of examining change mechanisms. Future research is needed to examine which treatment components influence the effectiveness of the treatment, and to examine possible predictors or moderators of treatment effectiveness (e.g. presence, cyber sickness, education level) and drop out in automated VR research.

Conclusions

Mental healthcare access has traditionally been a global challenge. If anything, the COVID-19 pandemic has accelerated the need for accessible, scalable, and affordable evidence-based mental health care, especially now that, as society reopens, ‘re-entry anxiety’ is on the rise. The results of this study confirm that a low-cost, highly scalable, and fully automated VR-CBT app can be an accessible and effective health care solution for reducing phobia symptoms. In doing so, it offers an accessible and scalable evidence-based treatment solution that can be applied globally at a fraction of the cost of current treatment alternatives.

Acknowledgements

We would like to thank Doruk Eker, MSc, and Gianluca Di Vincenzo, MSc, (Orb Amsterdam), for programming ZeroPhobia, Bruno de Vos, BA, (Studio Barbaar), for designing ZeroPhobia, and Rufus van Baardwijk, MSc, for sound design. Studio Barbaar, Orb Amsterdam, and Rufus van Baardwijk received financial compensation. Additionally, we would like to thank Rhiannon Sandfort for assistance with recruitment and participant management and Teije Koopmans (VALK Foundation) for assistance with recruitment. Finally, we sincerely thank all of the participants who participated in the study.

Supplementary material

For supplementary material accompanying this paper visit https://doi.org/10.1017/S0033291722003531.

S0033291722003531sup.zip (1.2MB, zip)

click here to view supplementary material

Financial support

The study was funded by the Dutch Research Council (NWO) Aspasia grant (ref 015.014.072). The funders had no role in the study design, data collection, data analysis, data interpretation, writing of the report, or the decision to submit the paper for publication.

Conflict of interest

TD and JLvG (InSession) developed the VR-CBT application that was used in the present study. The VR-CBT app is intended for commercial release. Hence, TD and JLvG were not involved in data analysis or with any decisions related to the publication of findings. The other authors declare that they have no competing interests.

References

  1. American Psychological Association. (2021). Stress in America. One year later, a new wave of pandemic health concerns. Washington, DC: American Psychological Association. [cited 2021 Nov 10]. Retrieved from https://www.apa.org/news/press/releases/stress/2021/one-year-pandemic-stress. [Google Scholar]
  2. Attkisson, C. C., & Greenfield, T. K. (2004). The UCSF client satisfaction scales: I. The client satisfaction questionnaire-8. In Maruish M. E. (Ed.), The use of psychological testing for treatment planning and outcomes assessment: Instruments for adults (pp. 799–811). Mahwah: Lawrence Erlbaum Associates Publishers. [Google Scholar]
  3. Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the system usability scale. International Journal of Human- Computer Interaction, 24(6), 574–594. doi: 10.1080/10447310802205776 [DOI] [Google Scholar]
  4. Beck, A., & Steer, R. (2015). BAI-NL Beck Anxiety Inventory – Nederlandse versie. [cited 2022 May 23]. Retrieved from https://www.pearsonclinical.nl/catalog/product/view/id/5467/s/beck-anxiety-inventory/category/876/#more-information-description.
  5. Beck, A. T., Epstein, N., Brown, G., & Steer, R. A. (1988). An inventory for measuring clinical anxiety: Ppsychometric properties. Journal of Consulting and Clinical Psychology, 56(6), 893–897. doi: 10.1037/0022-006X.56.6.893 [DOI] [PubMed] [Google Scholar]
  6. Benbow, A. A., & Anderson, P. L. (2019). A meta-analytic examination of attrition in virtual reality exposure therapy for anxiety disorders. Journal of Anxiety Disorders, 61, 18–26. doi: 10.1016/j.janxdis.2018.06.006 [DOI] [PubMed] [Google Scholar]
  7. Bentz, D., Wang, N., Ibach, M. K., Schicktanz, N. S., Zimmer, A., Papassotiropoulos, A., & De Quervain, D. J. F. (2021). Effectiveness of a stand-alone, smartphone-based virtual reality exposure app to reduce fear of heights in real-life: A randomized trial. NPJ Digital Medicine, 4, 16. doi: 10.1038/s41746-021-00387-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Botella, C., Fernández-Álvarez, J., Guillén, V., García-Palacios, A., & Baños, R. (2017). Recent progress in virtual reality exposure therapy for phobias: A systematic review. Current Psychiatry Reports, 19, 42. doi: 10.1007/s11920-017-0788-4 [DOI] [PubMed] [Google Scholar]
  9. Brown, G. K., Beck, A. T., Newman, C. F., Beck, J. S., & Tran, G. Q. (1997). A comparison of focused and standard cognitive therapy for panic disorder. Journal of Anxiety Disorders, 11(3), 329–345. doi: 10.1016/s0887-6185(97)00014-5 [DOI] [PubMed] [Google Scholar]
  10. Cardoş, R. A., David, O. A., & David, D. O. (2017). Virtual reality exposure therapy in flight anxiety: A quantitative meta-analysis. Computers in Human Behavior, 72(C), 371–380. doi: 10.1016/j.chb.2017.03.007 [DOI] [Google Scholar]
  11. Choy, Y., Fyer, A. J., & Goodwin, R. D. (2007). Specific phobia and comorbid depression: A closer look at the national comorbidity survey data. Comprehensive Psychiatry, 48(2), 132–136. doi: 10.1016/j.comppsych.2006.10.010 [DOI] [PubMed] [Google Scholar]
  12. Christensen, L., & Mendoza, J. L. (1986). A method of assessing change in a single subject: An alteration of the RC index. Behavior Therapy, 17, 305–308. [Google Scholar]
  13. Cornelisz, I., Cuijpers, P., Donker, T., & Van Klaveren, C. (2020). Addressing missing data in randomized clinical trials: A causal inference perspective. PLoS One, 15, e0234349. doi: 10.1371/journal.pone.0234349 [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Craske, M. G., Treanor, M., Conway, C. C., Zbozinek, T., & Vervliet, B. (2014). Maximizing exposure therapy: An inhibitory learning approach. Behavior Research and Therapy, 58, 10–23. doi: 10.1016/j.brat.2014.04.006 [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Curtis, G., Magee, W., Eaton, W., Wittchen, H., & Kessler, R. (1998). Specific fears and phobias: Epidemiology and classification. British Journal of Psychiatry, 173(3), 212–217. doi: 10.1192/bjp.173.3.212 [DOI] [PubMed] [Google Scholar]
  16. de Brey, H. (1983). A cross-national validation of the client satisfaction questionnaire: The Dutch experience. Evaluation and Program Planning, 3(6), 395–400. doi: 10.1016/0149-7189(83)90018-6 [DOI] [PubMed] [Google Scholar]
  17. Devilly, G. J., & Borkovec, T. D. (2000). Psychometric properties of the credibility/expectancy questionnaire. Journal of Behavior Therapy and Experimental Psychiatry, 31, 73–86. doi: 10.1016/S0005-7916(00)00012-4 [DOI] [PubMed] [Google Scholar]
  18. Donker, T., Cornelisz, I., Van Klaveren, C., Van Straten, A., Carlbring, P., Cuijpers, P., & Van Gelder, J. L. (2019). Effectiveness of self-guided app-based virtual reality cognitive behavior therapy for acrophobia: A randomized clinical trial. JAMA Psychiatry, 76(7), 682–690. doi: 10.1001/jamapsychiatry.2019.0219 [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Donker, T., Van Straten, A., Marks, I., & Cuijpers, P. (2009). A brief Web-based screening questionnaire for common mental disorders: Development and validation. Journal of Medical Internet Research, 11(3), e19. doi: 10.2196/jmir.1134 [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Evans, C. (1998). Reliable and clinically significant change. [cited 2022 Nov 11] Retrieved from http://www.psyctc.org/stats/rcsc.htm. [Google Scholar]
  21. Fehribach, J. R., Toffolo, M. B. J., Cornelisz, I., Van Klaveren, C., Van Straten, A., Van Gelder, J. L., & Donker, T. (2021). Virtual reality self-help treatment for aviophobia: Protocol for a randomized controlled trial. JMIR Research Protocols, 10(4), e22008. doi: 10.2196/22008 [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Fodor, L. A., Coteț, C. D., Cuijpers, P., Szamoskozi, S., David, D., & Cristea, I. A. (2018). The effectiveness of virtual reality based interventions for symptoms of anxiety and depression: A meta-analysis. Scientific Reports, 8(1), 10323. doi: 10.1038/s41598-018-28113-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Freeman, D., Haselton, P., Freeman, J., Spanlang, B., Kishore, S., Albery, E., … Nickless, A. (2018). Automated psychological therapy using immersive virtual reality for treatment of fear of heights: A single-blind, parallel-group, randomised controlled trial. The Lancet. Psychiatry, 5(8), 625–632. doi: 10.1016/S2215-0366(18)30226-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Furukawa, T. A., & Leucht, S. (2011). How to obtain NNT from Cohen's d: Comparison of two methods. PLoS One, 6, e19070. doi: 10.1371/journal.pone.0019070 [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Geraets, C. N. W., Van der Stouwe, E. C. D., Pot-Kolder, R., & Veling, W. (2021). Advances in immersive virtual reality interventions for mental disorders – a new reality? Current Opinion in Psychology, 41, 40–45. doi: 10.1016/j.copsyc.2021.02.004 [DOI] [PubMed] [Google Scholar]
  26. Hamra, G., MacLehose, R., & Richardson, D. (2013). Markov chain Monte Carlo: An introduction for epidemiologists. International Journal of Epidemiology, 42(2), 627–634. doi: 10.1093/ije/dyt043 [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Jacobson, N. S., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59(1), 12–19. [DOI] [PubMed] [Google Scholar]
  28. Krijn, M., Emmelkamp, P. M., Olafsson, R. P., Bouwman, M., van Gerwen, L. J., Spinhoven, P., … van der Mast, C.A. (2007). Fear of flying treatment methods: Virtual reality exposure vs. Cognitive behavioral therapy. Aviation Space Environmental Medicine, 78(2), 121–128. PMID: 17310883. [PubMed] [Google Scholar]
  29. Kroenke, K., Spitzer, R. L., & Williams, J. B. W. (2001). The PHQ-9: Validity of a brief depression severity measure. Journal of General Internal Medicine, 16(9), 606–613. doi: 10.1046/j.1525-1497.2001.016009606.x3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Lee, D. S. (2009). Training, wages, and sample selection: Estimating sharp bounds on treatment effects. The Review of Economic Studies, 76(3), 1071–1102. doi: 10.1111/j.1467-937X.2009.00536.x [DOI] [Google Scholar]
  31. Lindner, P., Dafgård, P., Miloff, A., Andersson, G., Reuterskiöld, L., Hamilton, W., & Carlbring, P. (2021). Is continued improvement after automated virtual reality exposure therapy for spider phobia explained by subsequent in-vivo exposure? A first test of the lowered threshold hypothesis. Frontiers in Psychiatry, 20, 645273. doi: 10.3389/fpsyt.2021.645273 [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Mertens, V. C., Moser, A., Verbunt, J., Smeets, R., & Goossens, M. (2017). Content validity of the credibility and expectancy questionnaire in a pain rehabilitation setting. Pain Practice, 17(7), 902–913. doi: 10.1111/papr.12543 [DOI] [PubMed] [Google Scholar]
  33. Miloff, A., Lindner, P., Dafgård, P., Deak, S., Garke, M., Hamilton, W., … Heinsoo, J. (2019). Automated virtual reality exposure therapy for spider phobia vs. in-vivo one-session treatment: A randomized non-inferiority trial. Behavior Research and Therapy, 118, 130–140. doi: 10.1016/j.brat.2019.04.004 [DOI] [PubMed] [Google Scholar]
  34. Morina, N., Ijntema, H., Meyerbröker, K., & Emmelkamp, P. M. G. (2015). Can virtual reality exposure therapy gains be generalized to real-life? A meta-analysis of studies applying behavioral assessments. Behavior Research and Therapy, 74, 18–24. doi: 10.1016/j.brat.2015.08.010 [DOI] [PubMed] [Google Scholar]
  35. Nousi, A., Van Gerwen, L., & Spinhoven, P. (2008). The flight anxiety situations questionnaire and the flight anxiety modality: Norms for people with fear of flying. Travel Medicine and Infectious Disease, 6(5), 305–310. doi: 10.1016/j.tmaid.2008.06.001 [DOI] [PubMed] [Google Scholar]
  36. Price, M., & Anderson, P. (2007). The role of presence in virtual reality exposure therapy. Journal of Anxiety Disorder, 21, 742–751. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Rothbaum, B. O., Anderson, P., Zimand, E., Hodges, L., Lang, D., & Wilson, J. (2006). Virtual reality exposure therapy and standard (in vivo) exposure therapy in the treatment of fear of flying. Behaviour Therapy, 37(1), 80–90. doi: 10.1016/j.beth.2005.04.004 [DOI] [PubMed] [Google Scholar]
  38. Rozental, A., Castonguay, L., Dimidjian, S., Lambert, M., Shafran, R., Andersson, G., & Carlbring, P. (2018). Negative effects in psychotherapy: Commentary and recommendations for future research and clinical practice. BJPsych Open, 4(4), 307–312. doi: 10.1192/bjo.2018.42 [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Schonlau, M., & Zou, R. Y. (2020). The random forest algorithm for statistical learning. The Stata Journal, 20(1), 3–29. doi: 10.1177/1536867X20909688 [DOI] [Google Scholar]
  40. Schubert, T., Friedmann, F., & Regenbrecht, H. (2001). The experience of presence: Factor analytic insights. Presence: Teleoperators and Virtual Environments, 10(3), 266–281. doi: 10.1162/105474601300343603 [DOI] [Google Scholar]
  41. Schulz, K. F., Altman, D. G., & Moher, D., & the CONSORT Group. (2010). CONSORT 2010 Statement: Updated guidelines for reporting parallel group randomised trials. BMC Medicine, 8, 18. doi: 10.1186/1741-7015-8-18 [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Skolnick, R. B., Schare, M. L., Wyatt, K. P., & Tillman, M. A. (2012). Aviophobia assessment: Validating the flight anxiety situations questionnaire as a clinical identification measure. Journal of Anxiety Disorders, 26(8), 779–784. doi: 10.1016/j.janxdis.2012.07.006 [DOI] [PubMed] [Google Scholar]
  43. Trumpf, J., Margraf, J., Vriends, N., Meyer, A. H., & Becker, E. S. (2010). Specific phobia predicts psychopathology in young women. Social Psychiatry and Psychiatric Epidemiology, 45(12), 1161–1166. doi: 10.1007/s00127-009-0159-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Van Almen, K., & Van Gerwen, L. J. (2013). Prevalence and behavioral styles of fear of flying. Aviation Psychology and Applied Human Factors, 3(1), 39–43. doi: 10.1027/2192-0923/a000035 [DOI] [Google Scholar]
  45. Van Gerwen, L. J., Spinhoven, P., Van Dyck, R., & Diekstra, R. F. W. (1999). Construction and psychometric characteristics of two self-report questionnaires for the assessment of fear of flying. Psychological Assessment, 11(2), 146–158. doi: 10.1037/1040-3590.11.2.146 [DOI] [Google Scholar]
  46. Winkler, P., Formanek, T., Mlada, K., Kagstrom, A., Mohrova, Z., Mohr, P., & Csemy, L. (2020). Increase in prevalence of current mental disorders in the context of COVID-19: Analysis of repeated nationwide cross-sectional surveys. Epidemiology and Psychiatric Sciences, 29, E173. doi: 10.1017/S2045796020000888 [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Wittkampf, K. A., Naeije, L., Schene, A. H., Huyser, J., & van Weert, H. C. (2007). Diagnostic accuracy of the mood module of the patient health questionnaire: A systematic review. General Hospital Psychiatry, 29(5), 388–395. doi: 10.1016/j.genhosppsych.2007.06.004 [DOI] [PubMed] [Google Scholar]
  48. Wolitzky-Taylor, K. B., Horowitz, J. D., Powers, M. B., & Telch, M. J. (2008). Psychological approaches in the treatment of specific phobias: A meta-analysis. Clinical Psychology Review, 28, 1021–1037. doi: 10.1016/j.cpr.2008.02.007 [DOI] [PubMed] [Google Scholar]
  49. World Health Organization. (2018). Mental health atlas 2017. Geneva, Switzerland: World Health Organization. [cited 2022 May 23]. Retrieved from https://www.who.int/publications/i/item/9789241514019. [Google Scholar]
  50. World Health Organization. (2020). The impact of COVID-19 on mental, neurological and substance use services: Results of a rapid assessment. Geneva, Switzerland: World Health Organization.[cited: 2022 May 23]. Retrieved from https://www.who.int/publications/i/item/978924012455. [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

For supplementary material accompanying this paper visit https://doi.org/10.1017/S0033291722003531.

S0033291722003531sup.zip (1.2MB, zip)

click here to view supplementary material

Articles from Psychological Medicine are provided here courtesy of Cambridge University Press

RESOURCES