A mobile application for panic disorder and agoraphobia: Insights from a multi-methods feasibility study

Abstract

Background

Panic disorder with and without agoraphobia (PD) is a common psychological disorder. Internet-based interventions have the potential to offer highly scalable low-threshold evidence-based care to people suffering from PD. GET.ON Panic is a newly developed internet-based intervention addressing symptoms of PD. In order to transfer the training into the daily life of the individuals, we integrated mobile components in the training and created a so-called hybrid online training. The development and beta-testing of such a training requires a novel interdisciplinary approach between IT specialists and psychologists. From this point of view, we would like to share our experiences in this exploratory paper.

Methods

This initial feasibility study (N = 10) offers, on the one hand, a brief overview of the interdisciplinary development phase of the mobile application and on the other hand, provides first insights into the usage, usability and acceptance of this mobile application using qualitative interview data as well quantitative measures of 8 completing participants. For these reasons, we used a pre-posttest design without a control group. Furthermore, we present initial clinical outcomes of the intervention on e.g. panic symptom severity, depressive symptoms as well additional anxiety measures. Finally, we end with implications for further research in the relatively new field of mobile mental health.

Results

Overall, usability, user satisfaction, motivational value and technology acceptance of the app were perceived as high. The usage of app components was diverse: The use of interoceptive exposure exercises and daily summaries on anxiety and mood was highest while using in-vivo exposure exercises and monitoring panic symptoms was perceived as difficult. Furthermore, participants showed after the training less clinical symptoms as at baseline-assessment.

Discussion

The current feasibility study contributes to an in-depth understanding of the potential of mobile technology in e-mental health. Overall, the GET.ON Panic app appears to be an acceptable and motivational part of a CBT-based hybrid online training for PD that has the potential to promote training success. After some suggested adjustments have been made, the efficacy should be investigated in a randomized controlled trial.

Highlights

• •This paper presents lessons learned from an interdisciplinary research work that could be beneficial to others developing mental health apps.
• •Qualitative as well quantitative data lead to an in-depth understanding of the potential of mobile technology in e-mental health.
• •Supplementing an iCBT training for PD with a mobile application has the potential to improve adherence, usability and usage of the training.
• •Furthermore, the hybrid online training seems to improve clinical outcomes. However, further research is needed.

1. Introduction

Panic disorder with and without agoraphobia (PD) is with a life-time prevalence of 1.1%−3.7% a common anxiety disorder (Jonge et al., 2016; Kessler et al., 2006) that causes serious impairments for the individual (Wittchen et al., 2011) as well high economic costs from a societal perspective (Batelaan et al., 2007; Ophuis et al., 2018). Cognitive-behavioral therapy (CBT) is a widely used effective treatment for PD (Pompoli et al., 2016; Sánchez-Meca et al., 2010), which – in the last decade – has found increasing use in internet-based interventions (iCBT) and has proven to be acceptable and effective for the treatment of PD (Andrews et al., 2018; Andrews et al., 2010; Olthuis et al., 2015) as well other mental disorders such as depression or anxiety disorders in general (Andersson et al., 2014; Carlbring et al., 2006; Lewis et al., 2018; Richards and Richardson, 2012).

Considering an annual increase of 20% of new mobile-broadband subscriptions to a worldwide number of 4 billion people using mobile internet, compared to a 9% growth rate for fixed-broadband subscriptions per year ([ITU]International Technology Union, 2017), smartphones are promising to facilitate access to evidence-based health care for a large number of people. Recently, researchers have started to explore the potential of smartphones in the field of e-mental health (Boschen and Casey, 2008; Ehrenreich et al., 2011; Eonta et al., 2011; Heron and Smyth, 2010; Proudfoot et al., 2013). Compared to interventions that are exclusively delivered via a device that connects to stationary internet (such as PC or laptop computer), an assimilation into the everyday life can be fostered with the use of mobile technology such as smartphones (Enock and McNally, 2014; Harrison et al., 2011; Price et al., 2014). Particularly in the case of PD, the use of mobile tools in treatment is promising, as the treatment intervenes where panic attacks take place, namely in the everyday lives of those affected (Christoforou et al., 2017; Ivanova et al., 2016; Proudfoot et al., 2013). Mobile-based interventions show advantages such as a high accessibility to the internet, independence of time and location regarding treatment delivery, provision of context-sensitive feedback, real-time data collection as well as the use of sensor technology for data collection (Heron and Smyth, 2010; Proudfoot, 2013).

However, the emerging field of m-mental health can still be described as being in its infancy as many available apps lack scientific evidence (Donker et al., 2013; Sucala et al., 2017) or show poor content quality (Singer et al., 2015). For these reasons, in addition to studies evaluating treatment efficacy, initial exploratory studies are needed for building foundations to assess usability and feasibility of innovative tools in treatment of mental problems as well as gaining and sharing further knowledge in this field. Examples of such in-depth feasibility studies in the field of m-mental health include the work of Morrison et al. (2014) providing data (N = 13) on quantitative app usage as well qualitative interviews regarding the desktop- and app-based weight management POWeR Tracker. As one of the main results the aforementioned authors highlight the benefit of a multicomponent, hybrid intervention format. Supplementing a web-based program with a mobile application leads to higher awareness of achievement of eating and physical activity goals, which was also confirmed by qualitative evidence. Another exploratory investigation on an intervention based on Acceptance and Commitment Therapy (ACT) that integrated mobile- and desktop-based technology suggested that participants (N = 11) suggested an increase in ACT-relevant outcomes as psychological flexibility and valued action; scores on depression, anxiety, stress and global satisfaction did not change over time (Ly et al., 2012). Further, participants' experience of the intervention was investigated with the help of qualitative questionnaires indicating that the mobile application has the potential to transfer the training into the daily life of the client, while the psycho-educational part delivered via a desktop-based tool was perceived as too text-based. For further research the authors suggest an integration of mobile technology in order to increase adherence to online health trainings. Further, Morris et al. (2010) presented five case studies on an app for emotional self-awareness in order to understand participants' use of the newly developed mobile application. More examples of feasibility studies include mobile interventions for depression (Burns et al., 2011), bipolar disorder (Bardram et al., 2013; Bardram et al., 2012), schizophrenia (Depp et al., 2010) and promoting health behaviors as self-monitoring of caloric balance (Tsai et al., 2007) or understanding medication labels (Grindrod et al., 2014).

Due to tremendous potential of m-health interventions and the lack of studies in the field of PD, the aim of the present feasibility study is to explore usage, usability and acceptance of the newly developed GET.ON Panic app as part of a hybrid online training GET.ON Panic. Furthermore, we present initial clinical outcomes of the training on panic symptom severity, depressive symptoms as well additional anxiety measures and to draw conclusions for future development of mobile interventions for research and practice.

2. Methods

2.1. Participants

The 10 participants were recruited from the waiting list of interested individuals for participation in online trainings for PD. They learned about the possibility to take part in the GET.ON online training via newspaper or radio reports. The average age of the participants was 41,5 years (SD = 14,4 years, N = 10) with an equal representation in gender (five female, five male participants). With regard to the participants' ethnical background, seven are Caucasian, one is Asian and two did not make a specification. Seven of ten participants have a college degree or higher, two have finished a vocational training and one has finished secondary school. Experience with psychotherapy was mentioned by nine of the ten participants. Eight of them had received psychotherapy for PD prior to the study. Seven of the participants used an iPhone with an iOS operating system and three of them participated with an Android smartphone. For further information see Table 1. For better readability, we have decided to nickname the study participants. These names are unrelated to their real name.

Table 1.

Overview of the demographics of the 10 participants.

Nickname	Gender	Age	Family status	Diagnosisc	Smartphone
Sid	M	38	Married	PD, OCD	iPhone 4sa
Joy	W	36	Unmarried	PD/A, SAD	Sony Ericsson prob
Ken	M	64	Unmarried	PD	iPhone 4a
Liz	W	20	Unmarried	PD/A	iPhone 4a
Amy	W	56	Separated	PD/A, GAD	iPhone 3GSa
Guy	M	60	Married	PD	iPhone 5a
Dan	M	28	Married	PD/A	iPhone 4 Sa
Joe	M	33	Married	PD/A	Motorola xt890b
Yue	W	35	Married	PD/A	Samsung Galaxy S3b
Ash	W	45	Separated	PD/A	iPhone 4a

PD = Panic Disorder; OCD = Obsessive Compulsive Disorder; PD/A = Panic Disorder with Agoraphobia; SAD = Social Anxiety Disorder; GAD = General Anxiety Disorder.

^a= iOS operating system.

^b= Android operating system.

^c= According to SCID-I interview.

2.2. Intervention

The GET.ON Panic app that is described and evaluated in this paper was part of the hybrid online training GET.ON Panic combining a desktop and a mobile component based on cognitive behavioral therapy (CBT) for PD. The hybrid online training consists of 6 modules with a recommendation of working on one module per week. After every module, participants received feedback on their training progress by a trained psychologist. The first module is mainly an introduction of the training and the mobile application. Furthermore, psycho-education concerning panic, agoraphobia and avoidance are offered. In the second module, participants start with interoceptive exposure exercises in order to provoke similar bodily symptoms as in a panic attack with the intention of getting used to those symptoms. In-vivo exposure exercises are part of the third module. Participants confront themselves with their anxiety provoking situations in real life. The fourth and fifth modules focus on cognitive restructuring exercises with an emphasis on the role of irrational beliefs with regard to panic-associated topics. The sixth and last module is about relapse prevention and finally deals with an evaluation of the training in terms of attaining individual goals. Overall, the desktop components are primarily used to provide text- and video-based psycho-education as well as exercises that require extensive writing. The mobile application GET.ON Panic app consists of two main components: a mobile diary and a mobile exposure guide.

The mobile diary is used to document panic attacks and to record daily summaries. When the mobile application is started, users are presented with the main screen (see Fig. 1a) from which the desired component and function can be activated quickly. For further description of the hybrid online training see the study protocol for the RCT (Ebenfeld et al., 2014). To document a panic attack with the GET.ON Panic app, users are asked to answer four questions on a Likert-like scale from 1 to 10 (see Fig. 1b). For this purpose, we created a novel input component based on the single-dimension mood-scale presented by Morris et al. (2010). The number moves with the finger and the intensity of the background color changes in order to provide an additional visual feedback (high values have a high intensity and vice-versa). In addition, we wanted users to be able to give individual meaning to a panic event in a non-prescribed way. We anticipated that some users are more visually oriented while others tend to prefer text. Therefore, two additional options have been included in the documentation of panic events. First, we encourage users to take a photo of either the situation where the event occurred or something that is related to the situation. Second, they may enter written notes in order to remember feelings or specific aspects of the situation. For daily summaries, a fixed-schedule diary was implemented (Bolger et al., 2003). Participants were asked to choose a fixed time each day, preferably in the evening, where they reflect on the past day and fill in a daily summary. In order to visualize a client's development and progress, the application plots the daily summaries over time (see Fig. 1d). Furthermore, the plot illustrates how using the training exercises can reduce panic symptoms by depicting, for each day, how many exposure exercises have been performed.

Fig. 1.

Screenshots of the GET.ON Panic app.

a. The main screen of the app.

b. Rating anxiety when documenting a panic attack.

c. Introduction to an interoceptive exposure exercise.

d. Daily summary graph.

e. Feedback after an in-vivo exposure exercise.

The exposure guide offers support for performing interoceptive and in-vivo exposures. It is constructed as a wizard, which leads the user through a sequence of steps while at the same time offering instructions and orientation (see Fig. 1c). In addition, at certain points, the user is asked to answer questions in order to measure if the exposure has been performed correctly (for example the level of anxiety, the degree of avoidance and the severity of body-symptoms). After an in-vivo exposure, users are offered feedback to see if the exercise worked and to reflect on the situation (see Fig. 1e). Furthermore, they have the possibility to take of photo of the mastered situation. Users can review these photos in a photo gallery together with the feedback and a description of the situation.

2.3. Development of the app

High credibility is crucial to facilitate behavioral change (Fogg, 2003), whereas credibility is defined as the combination of perceived trustworthiness and perceived expertise and highlights the importance of a product's visual appearance for the perceived first-hand and long-term experience (Fogg, 2003; Fogg et al., 2009). For these reasons, the GET.ON Panic app was developed in an iterative process in a multidisciplinary team of IT scientists, visual and interaction designers and psychologists from science as well clinical practice. We started each iteration by creating a paper mockup for a specific feature that the app should fulfill. The mockup was refined until the functionality was presented to our satisfaction. Following this, we have created a functional prototype that can be run on a smartphone. In the next step, this prototype was extensively tested and gradually improved in several feedback loops in the multidisciplinary development team. Instead, we decided to use psychotherapists and researchers with experience in the treatment of PD. In addition, we conducted regular but informal tests with team members who were not directly involved in the app development. For further presentation of the app, see also Kleine Stegemann et al. (2013).

2.4. Measures

2.4.1. Screening

Interested individuals were screened at the beginning of the study according to in- and exclusion criteria identical to those described in the study protocol of the planned RCT (Ebenfeld et al., 2014). This screening consisted of demographic data (e.g. age, gender, education, smartphone model), the Panic and Agoraphobia Scale (PAS) (Bandelow, 1995, Bandelow, 1997) and Item 9 of the BDI-II that screens for suicidality (Beck et al., 1996; Hautzinger et al., 2006). Furthermore, the Structured Clinical Interview for DSM-IV-TR axis-I disorders (SCID-I) (Wittchen et al., 1997) was used to get a detailed sample description (e.g. presence or absence of PD, agoraphobia, other anxiety disorders and a current depressive episode). The interview was conducted by a trained interviewer via telephone (Irvine et al., 2012; Rohde et al., 1997).

2.4.2. Clinical outcomes

Panic symptoms were measured with the PAS (Bandelow, 1995, Bandelow, 1997; Bandelow et al., 2000). The PAS is a 13-item questionnaire which is divided into five subscales addressing panic attacks, agoraphobic avoidance, anticipatory anxiety, limitations in daily life, and health concerns (e.g., fear of physical harm or fear of an organic cause). These subscales can be evaluated separately or as a total score, ranging from 0 to 52. Higher scores indicate a higher level of panic symptoms. The psychometric properties of the scale are good, with Cronbach's alpha of 0.86 (Bandelow, 1995). A score of 0–8 indicated no clinical relevant symptoms, a score of 9–28 a moderate level of symptoms, and a score of 29 or higher indicated severe symptomatology (Bandelow, 1995). Agoraphobic cognitions were assessed with the Agoraphobic Cognitions Questionnaire (ACQ) (Chambless et al., 1984; Ehlers and Margraf, 2001). The 14-item questionnaire has a total score range between 14 and 70. Internal reliability is reported as α = 0.80 (Craske et al., 1986). Bodily sensations were measured with the Body Sensation Questionnaire (BSQ) (Chambless et al., 1984; Ehlers and Margraf, 2001), a 17-item self-report questionnaire. The total score of the BSQ ranges from 17 to 85 and has proven a good internal reliability of α = 0.87 (Chambless et al., 1984). Agoraphobic avoidance was measured with the Mobility Inventory (MI) (Chambless et al., 1984; Ehlers and Margraf, 2001). The 27 items of the MI cover the most important agoraphobic situations. Each situation is rated for both when person is alone and when they are accompanied by someone else. The two total scores range from 27 to 135, respectively. The internal consistency is reported to be α = 0.94 (alone) and α = 0.91 (accompanied) (Chambless et al., 1984). Furthermore, the Hamilton Anxiety Rating Scale (HAM-A) (Hamilton, 1959; Weyer, 2005), a 14-item observer-rating was conducted. For these reasons an adaption of the structured interview guide for the Hamilton Anxiety Rating Scale (SIGH-A) (Shear et al., 2001). The total score of the HAM-A and SIGH-A ranges between 0 and 30. The interview has shown good inter-rater and test-retest reliability of ICC = 0.99, α = 0.86 (Shear et al., 2001). Depressive symptoms were assessed with the well-established German version1 of the Centre for Epidemiological Studies Depression Scale (CES-D) (Hautzinger and Bailer, 1993; Radloff, 1977). The 20 items were answered on a 4-point Likert scale regarding the previous week. The total score ranges from 0 to 60. Internal consistency is reported as Cronbach's α = 0.89 (Hautzinger and Bailer, 1993).

2.4.3. Usability

Usability of GET.ON Panic app was examined using the System Usability Scale (SUS). The ten-item scale ranging from 0 to 100 gives a an overview of subjective assessments of the usability (Brooke, 1996). A reliability analysis conducted by Bangor et al. (2008) found the internal consistency of the SUS to be Cronbach's α = 0.91. The survey was administered only at post-trial. At the time of the study, there was no standard German translation of the questionnaire. As a consequence, we involved experts from the field to create our own German version of the SUS using back and forward translation to ensure the correct translation.

2.4.4. Technology acceptance

Research on technology acceptance is an established part of information systems science, which strives to explain factors that affect acceptance of software systems. The technology acceptance model (TAM) and its successor TAM2 are commonly used models that are based on theory of reasoned action (Davis, 1989; Venkatesh and Davis, 2000). While the conducted pilot-study is not powered sufficiently for a full-size TAM analysis, we used the subscales to assess acceptance of our app. Specifically, these include the dimensions “Perceived Usefulness” and “Perceived Usability”. Furthermore, researchers have noted that acceptance is not only motivated by extrinsic aspects but also by intrinsic factors (Davis, 1989; Heijden, 2004). For this reason, we included a third scale for the dimension “Perceived Enjoyment”. We developed items for the three different scales based on the questionnaires published by Venkatesh and Davis (2000) as well as the items on perceived enjoyment as proposed by Heijden (2004). In this process, we incorporated researchers from the field of clinical psychology as well as information science. The survey was conducted at post-trial.

2.4.5. Compliance to app

To assess compliance with GET.ON Panic app, the number of interactions with the app was recorded. In this context, we defined an interaction as the activation of a function of the app by its user. Due to technical limitations, low-level interactions, such as single gestures or page transitions, have not been stored. Furthermore, it has to be noted that we did not include “view-only” interactions, such as reading the diary or looking at photos. From the recorded data, several measures were derived. First, the number of interactions per module and over the course of the intervention was used to evaluate app usage at a global level. Next, the average number of interactions per day has been calculated in order to account for varying intervention durations amongst participants. Finally, we determined the total and average number of interactions per participant to evaluate individual differences.

2.4.6. User satisfaction

User satisfaction of the training was measured with a self-designed questionnaire based on the German version of the Client Satisfaction Questionnaire (Attkisson and Zwick, 1982; Schmidt et al., 1989). The adapted version of the questionnaire consists of 8 items and the total score ranges from 0 to 32. Internal consistency of the original German version questionnaire has been reported as good (Schmidt et al., 1989).

2.4.7. Feasibility interviews

A semi-structured interview with the participants was conducted at post-trial via telephone. The interview consisted of 59 open questions covering the training in general, the specific functions of GET.ON Panic app, integration of app and online training as well as enjoyment.

2.5. Data analysis

All data contained answers from 8 participants and were analyzed on completer-only principals due to drop out of 2 participants. The quantitative analyses were done using R (package = “lsr”) (Navarro, 2014; R Core Team, 2013). We calculated the changes in pre and post scores by performing t-tests. Furthermore, we calculated Cohen's d effect sizes for the clinical outcome measures. The qualitative data was analyzed based on thematic analysis according to Braun and Clarke (2006). For these reasons, the recorded interviews were transcribed and coded. Subsequently, all important aspects related to the research questions were identified and clustered to identify relevant topics and general patterns.

3. Results

3.1. Adherence

Eighty percent of the participants completed the questionnaires and finished the training after lesson 6. Two participants (Yue and Ash) dropped out after module 2 and module 5 respectively and did not take part in the post assessment. We were not able to reach them anymore.

3.2. Clinical outcomes

Using two-sided unpaired t-test we did not find a significant change pre and post intervention for any clinical outcome measures. Table 2 gives an overview of the data.

Table 2.

Clinical outcome (n = 8).

	M(SD) pre	M(SD) post	M(SD) pre-post	t-testa	Effect sizeb
PAS	17.25 (6.98)	13.75 (5.42)	3.5 (6.05)	t(df) = 1.12 NS	d = 0.56
HAM-A	17.62 (7.41)	11 (8.68)	6.62 (5.29)	t(df) = 1.64 NS	d = 0.82
ADS	21.88 (12.04)	15.88 (12.67)	6 (10.93)	t(df) = 0.97 NS	d = 0.49
ACQ	1.71 (0.50)	1.39 (0.42)	0.32 (0.36)	t(df) = 1.39 NS	d = 0.70
BSQ	2.58 (0.76)	2.26 (0.54)	0.32 (0.55)	t(df) = 0.96 NS	d = 0.48
MI-Al	2.02 (0.59)	1.81 (0.56)	0.20 (0.34)	t(df) = 0.71 NS	d = 0.35
MI-Ac	1.63 (0.58)	1.55 (0.50)	0.08 (0.23)	t(df) = 0.31 NS	d = 0.15

PAS=Panic and Agoraphobia Scale; HAM-A = Hamilton Anxiety Rating Scale; ADS = Allgemeine Depressions-Skala; ACQ = Agoraphobic Cognitions Questionnaire; BSQ = Body Sensation Questionnaire; MI-Al = Mobility Inventory (alone); MI-Ac = Mobility Inventory (accompanied). NS=Not significant.

^a=Independent two-sided t-test.

^b=Cohen's d pre-post (within) (0.30 for small effect, 0.50 for medium effect, 0.8 for large effect).

3.3. Usability

Results from the System Usability Scale are summarized in Table 3. The mean score of 84.06 indicates that there have been no serious usability issues. According to Bangor et al. (2008), GET.ON Panic app shows a good usability that is well within the acceptable range. The mean scores for all items were above midpoint of the scale for positive and below midpoint for negative items (Table 4). Of note is that participants found the app generally easy to learn and easy to use. On the other hand, item 4 has a mean score of 2.38 which indicates that some participants experienced technical issues. This is in line with problems reported by clients to our technical support (Table 5). All problems except the touch issues reported by one participant (Joy) could be solved and participants were able to continue using the app.

Table 3.

System Usability Scale (n = 8), range 0–100.

Participant	SUS score
Sid	97.50
Joy	82.50
Ken	77.50
Liz	100.00
Amy	92.50
Guy	62.50
Dan	90.00
Joe	70.00
	M = 84.06 (SD = 13.36)

Table 4.

Item ratings of the System Usability Scale (n = 8).

Item	Mean (SD)
1. I think that I would like to use this app frequently.	4.00 (0.76)
2. I found the app unnecessarily complex.	1.50 (0.76)
3. I thought the app was easy to use.	4.38 (0.52)
4. I think that I would need the support of a technical person to be able to use this app.	2.38 (1.69)
5. I found the various functions in this app were well integrated.	4.13 (0.83)
6. I thought there was too much inconsistency in this app.	1.50 (1.07)
7. I would imagine that most people would learn to use this app very quickly.	4.38 (0.74)
8. I found the app very cumbersome to use.	1.25 (0.71)
9. I felt very confident using the app.	4.50 (0.76)
10. I needed to learn a lot of things before I could get going with this app.	1.13 (0.35)

Note: All items rated on 1–5 Likert scale with 1 = strongly disagree and 5 = strongly agree.

Table 5.

Technical problems as reported by participants (n = 8).

Participant	Technical Problem(s)
Sid	–
Joy	Touch gestures did not work properly for in-vivo exposures.
Ken	App became unresponsive after accidentally activating the copy function.
Liz	–
Amy	App crashed because there was no space left on device.
Guy	App terminated a few times during an in-vivo exposure exercise.
Dan	–
Joe	Needed to re-install app after device was reset.

3.4. Technology acceptance

The results from the assessment of technology acceptance for GET.ON Panic app are illustrated in Table 6. In general, participants scored high on all items of the TAM survey. They did not only find the mobile application to be useful for the training (subscale “Perceived Usefulness”, M = 6.13, SD = 0.75, Likert scale 1–7), but also indicated that they enjoyed using the app (subscale “Perceived Enjoyment”, M = 5.89, SD = 0.91, semantic differential 1–7). Furthermore, the score for perceived usability (M = 6.09, SD = 1.24, Likert scale 1–7) is congruent with the results from the system usability scale.

Table 6.

Results from the TAM survey (n = 8).

Subscale: perceived usefulness	M = 6.13 (SD = 0.75)
1. The app makes the training more effective for me.	M = 6.13 (SD = 1.13)
2. The app makes it easier for me to implement the training.	M = 6.13 (SD = 0.83)
3. The app makes the training more intense for me.	M = 6.38 (SD = 0.52)
4. I find the app to be useful for the training.	M = 6.00 (SD = 1.07)
5. The app makes the training easier for me.	M = 6.00 (SD = 0.76)

Subscale: perceived enjoyment	M = 5.89 (SD = 0.91)
6. Boring – interesting	M = 6.38 (SD = 0.74)
7. Unpleasant – pleasant	M = 5.13 (SD = 1.36)
8. Dull – exciting	M = 6.00 (SD = 0.93)
9. Disgusting – enjoyable	M = 6.00 (SD = 1.07)

Subscale: perceived usability	M = 6.09 (SD = 1.24)
10. My interaction with the app is clear and understandable.	M = 6.00 (SD = 1.41)
11. Interacting with the app does not require a lot of mental effort.	M = 6.00 (SD = 1.41)
11. I find the app to be easy to use.	M = 6.25 (SD = 1.16)
12. I find it easy to get the app to do what I want it to do.	M = 6.13 (SD = 1.25)

Note: Items for perceived usefulness and perceived usability rated on 1–7 Likert scale with 1 = strongly disagree and 7 = strongly agree; For perceived enjoyment, semantic differentials have been used with a scale from 1 to 7.

3.5. Compliance to app

The eight participants spent a mean of 51.88 days in the training (SD = 12.11). During this time, they used GET.ON Panic app on average 1.56 (SD = 0.96) times per day. The use of individual app functions varies amongst participants with daily summaries and interoceptive exposures being the most popular parts. On the other hand, the function for in-vivo exposures was not well received and the number of documented panic events seems to be rather low (see Table 7). Fig. 2 shows that, on average, app use was highest in the second module (2.34 times per day) and remained – at a slightly lower level – fairly constant for modules three to six. During the last module, however, the app was used significantly less by all participants. Examination of app use and clinical outcomes shows no significant relation between average use per day and clinical outcomes (Fig. 3). This is illustrated, for example, by Sid, who used the app 2.62 times per day but improved less than Liz, who used the app only 0.91 times per day. The PAS score of Ken, who performed the most interoceptive exposure amongst all participants (125 times) and also used the app most often even increased by 5 points.

Table 7.

App use for clients (n = 8).

Participant	Diarya (n)	Daily summaryb (n)	Daily summary compliancec	Interoceptive exposured (n)	In-vivo exposuree (n)	Average daily usef (n)
Sid	9	36	86%	65	0	2.62
Joy	4	7	14%	5	0	0.31
Ken	2	6	15%	125	0	3.24
Liz	2	39	83%	1	1	0.91
Amy	8	41	79%	29	0	1.50
Guy	6	50	76%	43	9	1.64
Dan	4	33	45%	24	0	0.86
Joe	0	36	86%	22	0	1.38
Mean (SD)	4.38 (3.11)	31.00 (15.95)	60.50 (31.31)	39.25 (40.18)	1.63 (3.16)	1.56 (0.96)

^aNumber of reported anxiety related events during the training period (e.g. panic attack or avoidance behavior).

^bNumber of completed daily summaries within the training period.

^cCompliance is reported as percentage of the number of days in training on which the participant filled out the daily summary with the app.

^dNumber of completed interoceptive exposure exercises during the training period.

^eNumber of completed in-vivo exposure exercises during the training period.

^fAverage daily use, calculated as the sum of interactions divided by number of days in training.

Fig. 2.

Average daily use app over the course of the training (n = 8).

Fig. 3.

App use and clinical outcomes (n = 8).

Note:¹Average daily use, calculated as the sum of interactions divided by number of days in training. ²Difference of pre and post score; positive scores represent improvement.

3.6. User satisfaction

All participants scored high on the user satisfaction questionnaire with a mean of 30.13 (SD = 1.64). Given that the maximum achievable score is 32, we can assume a high level of satisfaction of the participants with the training in general.

3.7. Interviews

Based on thematic analysis (Braun and Clarke, 2006) five key themes were identified. Visual feedback and photo function were themes that directly address the functions of the app. Everyday suitability, usability (design) and interaction of app and online training were also functional but from a more general perspective. In addition, motivation and self-efficacy were clustered as intrapersonal themes.

3.7.1. Feasibility

The training without the mobile component was described as difficult to complete “because you just do not have the computer with you” as Joy stated. The exposure exercises on the app were perceived as “feasible” (Guy), “easy” (Amy) and “comprehensible” (Joe). One of the perceived advantages by Liz was that “the anxiety provocation exercises [interoceptive exposure exercises] could be done in real time by means of the app”. Joy was not able to integrate the exercises in her daily life due to technical problems (as discussed in the usability section). Six of the eight participants would have preferred a reminder for the exercises and daily summary. Most of the participants were able to fill in the daily summary most of the time (see compliance to app). In the following, reasons for non-compliance to the diary will be specified. Joy reported that she forgot to fill in the diary “if there wasn't an anxiety event on this day or because I was distracted by other things”. Ken also mentioned time-related problems that lead to forgetting the daily summary. Regarding the registration of anxiety events the participants reported problems with performing this task. Reasons for this were “I haven't had any panic attacks at all during the course of the training” (Joe) or “during the moment of panic, I didn't have a clear head and the time for this” (Joy). Regarding the in-vivo exposures a typical problem was that participants forgot to finish the exercise by pushing the ‘finish’ button with the consequence that the data was not recorded. Others reported that the app functions as tutorial and motivation, but the actual in-vivo was performed in a non-structured way without the app. With regard to the interoceptive exposure participants reported less problems with performing this task according to the app. Only Dan stated that “it became annoying to answer all the questions twice a day”.

3.7.2. Motivation

In the interview all participants indicated that motivation was an important factor of the mobile application. Dan described that “technical gadgets are per definition very motivational”. Guy stated: “Using the app was fun because I am an app-oriented person”. Especially in terms of conducting the interoceptive exposure exercises the mobile application was perceived as valuable. All of them stated that they would not have done the exercises that often without the app. One perceived the app as “mental support” during the exercises and “The fact that you can see what you have done is very motivating.”. Regarding motivational elements in the app the participants expressed contrary opinions. Three participants said that they would appreciate to have more motivational elements. “More motivational elements would be a good loosening up for the training.” Two of the eight participants experienced the motivational elements (such as the photo function) as “way too much” and “not desirable”.

3.7.3. Usefulness

One objective of the self-help training was to provide people who are suffering from panic with a better understanding of their anxiety. Nearly all participants reported that especially the interoceptive exposure exercises lead to a better understanding of how the body reacts during a panic attack. “You can make the difference with interoceptive exposure exercises. Also, the daily summary facilitates the own estimate”, as one participant states.

3.7.4. Visual feedback

Five interviewees reported that they liked the visual representation of the daily summaries and that it helped them to become more aware of their course of panic attacks. One said “that the perception of an anxiety attack is quite different at the moment while it happens and afterwards” and another one stated “retrospectively, I realized that nothing frightening would really happen”. Dan experienced the visualization of daily summaries as “reasonable but without added value” due to the fact that he knew “how the course of anxiety will look like”. Only Guy said that “the visual feedback was not that helpful. A simplified visualization with bar charts would be preferable”. The graphical representation of the exposure exercises was also perceived as useful by nearly all of the users. “The graphical feedback of the exposure exercise was helpful to understand ‘the thing as a whole’”. One remark of Dan was that he would have preferred all questions on a single page.

3.7.5. Photo function

The photo function as part of the documentation of the anxiety events as well during the in-vivo exposure lead to controversial statements. Most of the participants stated that they understood the meaning of the photo function but they did not use it. In the interviews we found explanations like “taking a picture wouldn't have a benefit” or “a nice gadget, but I wasn't able to take a picture of the anxiety provoking stimulus” or “anxiety events weren't catchable enough to capture in a photo” or “taking pictures wasn't important for me, the graphical feedback was more important to me”. Another one said “I just took a picture, it was a symbolization of the moment - how I felt there”. All participants who found it difficult to take a picture of an anxiety provoking stimulus agreed that they would have preferred to take a picture of a positive association which was not related to an anxiety-trigger.

3.7.6. Interaction of app und online training

One part of the interview focused on the integration of the app and the browser-based training part into a hybrid online training format. All participants said that the combination of both was overall coherent and very motivational. As a suggestion for improvement “it would be great if you would open the app and automatically see the exercises concerning module 3” (Joe).

4. Discussion

Previous research has highlighted the potential of smartphones to treat mental health problems (Boschen and Casey, 2008; Ehrenreich et al., 2011; Eonta et al., 2011; Heron and Smyth, 2010; Proudfoot et al., 2013). In this paper, we presented the mobile application GET.ON Panic app and examined its development, feasibility, usability, technology acceptance and first indices on clinical effectiveness amongst 8 participants suffering from PD symptoms.

4.1. Principal findings

Below we discuss the most important findings that can be drawn from this study.

4.1.1. App development

We developed the mobile application with an interdisciplinary team of researchers and clinicians from the fields of psychology, computer science and visual design in an iterative process. We have perceived this as a very efficient way of working. Clear areas of responsibility, different perspectives and frequent exchanges led to a common understanding of the objectives and little “frictional losses”.

4.1.2. Feasibility and usability

Most participants viewed the GET.ON Panic app as a feasible means of training. The usage was relatively high with daily summaries and interoceptive exposures exercises as the mostly used and most valued parts of the app. All participants stated they would not have done these exercises that often without the app. The app thus seems to enhance adherence to the training and to fulfill a motivational function. High scores on user satisfaction, usability and technology acceptance support these findings.

However, some parts of the app offer room for improvement. Overall, two components of the app were perceived as difficult. First, the participants reported problems with performing the real-time registration of anxiety events. Reasons for this could be that they currently do not suffer from panic attacks at all. For this case, they received the instruction instead to register the avoidance behavior whenever it occurs. However, this instruction seems difficult to follow per se, since avoidance is usually not conscious behavior and therefore difficult to notice and register. As further explanation it can be stated that registering a panic attack while actually having a panic attack is perceived as difficult. Participants reported they were too excited to enter data immediately and registering in this case was not perceived as supportive. Second, performing in-vivo exposures guided by the app led to difficulties. This exercise was introduced in one of the modules of the browser-based training. Here, the participants have to build their anxiety hierarchy by ordering their anxiety-provoking moments from weak to strong. Further, they have to enter this hierarchy in the app again. This was perceived as ineffective. A synchronization of data between app and training was desired in order to enhance the usability. In addition, it seems that the instruction of the app was difficult to follow. After successful exposure task, the participants were asked to stop the exercise on their smartphone display. However, it is suggested that many participants forgot this step due to excitement while and after exposure.

Using the app seems to facilitate daily routines during the training such as performing interoceptive exposure exercises or monitoring symptoms as overall anxiety or mood. In contrast, measuring data that is less predictable and more event-related (e.g. as in-vivo exposures or monitoring panic attacks) was perceived as more difficult. These perceptions were in line with the findings of Morrison et al. (2014). They stated that their POWeR tracker app was associated with greater usage for specific, repetitive behavior.

4.1.3. Integration in daily life

All participants said to perceive the combination of both, a mobile tool and a browser-based training as coherent and very motivational. In particular, elements of the app, such as (interoceptive) exposure exercises and receiving graphical feedback after performing this task helped them to get an understanding of their bodily sensations during anxiety moments. As an idea for improvement some participants only wished one reminder for the daily recurring exercises in order to ensure a better integration of these exercises in their daily routines. Other elements led to more controversial statements. For example, the usage of the photo function as part of the documentation of the anxiety events or during the in-vivo exposure has not added any therapeutic value to the participants. Most of them did not use it or consider it as a fun gadget. In contrast to our initial assumptions, it seems that a photo function is not an effective means to enhance the transfer of the training into the daily-lives of the participants.

4.1.4. Effects of app on clinical outcomes

In view of the study design, we can neither make reliable statements about the clinical effectiveness of the app nor of the hybrid online training and its mechanisms of change. However, due to the fact that most participants improved on the clinical outcomes like panic, anxiety and depressive symptoms, albeit, non-significant overall, we assume the hybrid training has the potential to reduce panic symptoms as well as further clinical outcomes such as depression. Since we did not find any correlation between clinical improvement and app usage, we cannot further specify the added value of the app compared to browser-based training alone in terms of clinical efficacy. Moreover, in the recent meta-analysis of Andrews et al. (2018), including only browser-based trainings for PD (e.g. Carlbring et al., 2006; Ballegooijen et al., 2013; Wims et al., 2010), a large mean effect size of hedge g = 1.31 was found, which also indicates that the benefit of the app in terms of clinical efficacy needs further investigation.

4.2. Strengths and limitations

One strength of this study is the mixed method design with taking qualitative and quantitative data into consideration. This allows us to draw conclusions about how affected people rated, used and interacted with an app for PD. As a limitation, this exploratory study design is not able to meet the demand for generalizability of the results to a larger population. Furthermore, conclusions about the clinical effectiveness are also not possible due to a missing control group and insufficient power of the study. As a final remark, it should be noted that two of the ten participants dropped out of the study without giving any reasons. We were not able consider their opinions on feasibility. This could bias the present interpretation of the results.

4.3. Future work

As a short-term implication we would like to implement minor improvements such as layout issues or integration of a prompts for entering the daily summaries. Looking further ahead, the input of data should be facilitated in the long term, especially at times when participants are in high states of arousal, as it is common during panic attacks or in vivo exposures. In these moments it may be inconvenient to enter data via the touchpad keyboard. To improve usability, data entry could be done using the integrated voice recorder that transmits voice-to-text. Furthermore, the current app has not yet explored the potential of integrated context sensors. For further long-term app development, it would be worthwhile to use GPS data to provide intelligent context-sensitive exposure tasks. People would become informed via an app notification when there is a previously defined anxiety-provoking location close to them, as, for example, a shopping mall or a high tower. They would be invited to perform an exposure task in real-time and real-place environment. This gamified component might foster the motivation of clients. As a final note on the development procedure of a hybrid online training, we favour an automatic synchronization of the entered data between the mobile app and the browser-based training to avoid the need for duplicate input on both devices and instead allow a continuous training process with an improved usability. As closing general remark when planning to develop a hybrid online training, we recommend an automatic synchronization of entered data between the mobile app and the browser-based training to avoid taking double entrees and instead offer a consecutive training process. Finally, with regard to future research an RCT is necessary to assess the clinical efficacy of the GET.ON Panic app as part of a hybrid online training.

4.4. Conclusion

The combination of quantitative and qualitative data leads to further insights into how participants use a health care app. Findings from this study suggest that supplementing an iCBT training for PD with a mobile application has the potential to improve adherence, usability, usage as well clinical outcome of the training. From a patient's perspective, using the app seems fun, is engaging and facilitates daily routines during the training such as performing interoceptive exposure exercises or monitoring anxiety or depressive symptoms on a daily basis. In contrast, measuring data that is less predictable and more event-related (e.g. in-vivo exposures or monitoring panic attacks) was perceived as more difficult. With regard to further app development, this leads to the idea that the input of data during excitement, for example during panic attacks or in-vivo exposure exercises should be less effort for the individual. This can be achieved, for example, by entering data via voice and not via typing on the smartphone's touchscreen. Furthermore, the integration of intelligent context-sensitive tools, such as prompting real-time content in relevant moments could lead to further improved usage of the app. Finally, a following RCT is necessary to address this research question.

Funding

The research presented in this paper was part of the Innovation-Incubator of the Leuphana University Lueneburg. The Innovation-Incubator was funded by the European Union and the state of Lower Saxony (ERDF: CCI 2007DE161PR001).

Author contributions

All authors contributed to the study design. LE, DL, DE, HR and MB developed the hybrid online training for treating people with PD with or without agoraphobia (GET.ON Panic). SKSt developed the app (GET.ON Panic app). BF supervised the technical aspects of the intervention. LE and SKSt performed the statistical analyses and drafted the manuscript. DL and MB revised the manuscript. All authors read and approved the final manuscript.

Declaration of competing interest

DL, DE and BF are shareholders, LE is an employee of the transfer institute (GET.ON Institute) that currently commercializes the GET.ON Panic training. However, during developing and evaluating the app and the hybrid online training GET.ON Panic the company was not yet incorporated.