Cognitive Bias Modification for Threat Interpretations: Using Passive Mobile Sensing to Detect Intervention Effects in Daily Life

Abstract

Background:

Social anxiety disorder is associated with distinct mobility patterns (e.g., increased time spent at home compared to non-anxious individuals), but we know little about if these patterns change following interventions. The ubiquity of GPS-enabled smartphones offers new opportunities to assess the benefits of mental health interventions beyond self-reported data.

Objectives:

This pre-registered study (https://osf.io/em4vn/?view_only=b97da9ef22df41189f1302870fdc9dfe) assesses the impact of a brief, online cognitive training intervention for threat interpretations using passively-collected mobile sensing data.

Design:

Ninety-eight participants scoring high on a measure of trait social anxiety completed five weeks of mobile phone monitoring, with 49 participants randomly assigned to receive the intervention halfway through the monitoring period.

Results:

The brief intervention was not reliably associated with changes to participant mobility patterns.

Conclusions:

Despite the lack of significant findings, this paper offers a framework within which to test future intervention effects using GPS data. We present a template for combining clinical theory and empirical GPS findings to derive testable hypotheses, outline data processing steps, and provide human-readable data processing scripts to guide future research. This manuscript illustrates how data processing steps common in engineering can be harnessed to extend our understanding of the impact of mental health interventions in daily life.

Introduction

Treatment outcome monitoring tracks a client’s progress throughout treatment, alerting clinicians to patient deterioration or stalled progress if it occurs. Although regular use of outcome monitoring is associated with improved treatment outcomes (Brattland et al., 2018), only 39% of American Psychological Association (APA) members reported sometimes, frequently, or routinely using assessment measures to monitor client progress in treatment (Wright et al., 2017). Outcome monitoring typically relies on clients’ responses to trait self-report questionnaires. This approach imposes a dual burden: busy clinicians must remember to administer questionnaires and track client responses, and clients must take the time to answer.

Personal smart devices, such as smartphones and smartwatches, enable unobtrusive, in-the-moment data collection via a variety of native sensors (e.g., GPS, accelerometer, light).

Historically, researchers have measured intervention success through trait self-report questionnaires, which are limited by a number of biases including the client’s lack of insight, memory distortions when reporting retrospectively (Shiffman et al., 2008), and demand effects to appear either ill or improved. Smartphone sensors provide a more objective, feature-rich depiction of how an individual is actually functioning on a moment-to-moment basis (Rabbi et al., 2011). While subjective evaluations of treatment progress are valuable, passive sensors provide additional, rich data that can be used to evaluate aspects of interventions while imposing minimal client and clinician burden (Harari et al., 2017). Given that mental health interventions aim to influence a person’s actual behavior and not only their self-reported behavior, passive sensors may serve as a complementary tool in outcome monitoring. The current study aims to test if passive mobility patterns that are associated with social anxiety disorder change as a function of a brief, one-week online cognitive training intervention.

Passive Sensing and Mental Health

Passively-collected, behavioral data are increasingly linked to mental illnesses, such as bipolar disorder (Grünerbl et al., 2015), post-traumatic stress disorder (Place et al., 2017), and general stress and loneliness (Ben-Zeev, et al., 2015). The growing body of passive sensing literature shows that passively collected mobility patterns, specifically, can differentiate between within-person states (e.g., depressive vs. manic states; Grünerbl et al., 2015) and between-person mental health traits (e.g., time at home varies by social anxiety level; Boukhechba et al., 2018). Sensor technologies present a promising new way to conduct outcome monitoring by comparing patterns in passively collected data leading up to, during, and after a mental health intervention. However, while researchers are beginning to leverage ecological momentary assessment (EMA) self-reports to track intervention-related changes in emotional experiences and behaviors in daily life (e.g., Clarke et al., 2016; Daniel et al., 2020), few studies have investigated how interventions are tied to changes in passive features. Extending treatment outcome monitoring to include the monitoring of passively-collected mental health-related behaviors affords the opportunity to assess objectively observable, daily-life effects of a mental health intervention with little to no client burden.

Passive Mobility Data and Social Anxiety Disorder

Avoidance is a defining feature of social anxiety disorder (SAD; American Psychiatric Association, 2013). In daily life, SAD has been associated with a number of passive data features that suggest social avoidance. For example, higher trait social anxiety symptoms are associated with a greater likelihood to spend more time at one’s own home (Chow et al., 2017) and less time at other people’s homes (Boukhechba, et al., 2018). Further, college students who are more socially anxious have been shown to visit fewer distinct locations than less anxious students (Boukhechba, et al., 2017). Although movement patterns can be driven by a number of factors that are independent of social anxiety and avoidance (e.g., weather, season, employment), the evidence for connections between social anxiety and movement patterns supports the potential utility of leveraging passive mobility data to assess one aspect of how social anxiety manifests in daily life. For example, it is possible that changes in a person’s passively-sensed movement patterns from pre- to post-intervention could indicate whether or not the intervention had an impact on objective behavior patterns that are associated with social anxiety disorder.

Cognitive Bias Modification for Interpretation

Social anxiety disorder is associated with rigidly interpreting ambiguous situations as socially threatening (e.g., Amir et al., 2005). This tendency, called negative interpretation bias, is a cognitive process that is theorized to maintain symptoms of social anxiety, including maladaptive social avoidance (Hofmann, 2007). To illustrate how negative interpretation bias might maintain social avoidance, imagine the following scenario: You wave to a neighbor and they do not wave back. You now have a choice of what meaning you assign to your neighbor’s behavior (i.e., how you interpret the ambiguous non-response from your neighbor). If you assign a negative meaning (e.g., they don’t want to be friends with me) versus a benign meaning (e.g., they didn’t see me), you will be less likely to try to strike up a conversation with your neighbor (i.e., you will be more avoidant). Cognitive bias modification for interpretations (CBM-I) aims to reduce negative interpretation bias by presenting socially anxious people with many brief scenarios that each introduce a potentially socially-threatening situation (e.g., going to a dinner party). The outcome of each scenario remains ambiguous until the final word, which is presented as a word fragment that the person must complete to resolve the emotional ambiguity. By routinely presenting participants with word fragments that assign a benign rather than a threatening meaning, CBM-I makes salient that the emotional ambiguity can be resolved in non-threatening ways. As a result, people are expected to be more willing to tolerate ambiguous social situations, which may then improve social anxiety symptoms like avoidance (Hofmann, 2007; Rapee & Heimberg, 1997).

Given that CBM-I has been shown to effectively reduce both negative interpretation bias and social anxiety symptoms (see Jones & Sharpe, 2017, for a meta-analytic review), this intervention provides an opportunity to evaluate changes to passive data patterns in daily life following an intervention for socially anxious individuals. Using the realist evaluation framework (Punton et al., 2020) to describe this intervention logic mechanistically, we consider an ambiguous social threat for someone with social anxiety to be the context in which the intervention works. In this context, we expect the CBM-I intervention to bring about changes to movement patterns (the outcome) via changes in threat interpretation bias (the mechanism). Specifically, we expect that changing this rigidly negative interpretive style to be more flexible, so that a person does not assign threat meanings as often or as intensely, will facilitate a greater willingness to enter into ambiguous social situations and/or to stay in social situations for longer. Consequently, this may change how a person moves about their environment. In fact, previous analyses in this data set found that the CBM-I intervention did indeed significantly reduce negative interpretation bias relative to the control condition, supporting mechanism engagement in this sample (Daniel et al., 2020). However, in these previous analyses, self-reported outcome measures showed largely null intervention effects. The present study tests a different series of outcomes, focusing on passively-sensed sources of data rather than participant self-report.¹

Hypotheses

The current study assesses changes in passively sensed mobility data following a low dose (six daily sessions, approximately 15 minutes each) of an online CBM-I intervention. This intervention was randomized to half of a sample scoring high on trait social anxiety symptoms during the middle of a 5-week mobile phone monitoring study, where passive GPS data was continuously collected for two weeks prior to the intervention period and for two weeks following the intervention period. Hypotheses were pre-registered on the Open Science Framework (OSF https://osf.io/em4vn/?view_only=b97da9ef22df41189f1302870fdc9dfe). Given that CBM-I aims to reduce social anxiety symptoms like avoidance through reducing negative interpretation bias, we expect that passive mobility patterns will look less stereotypically avoidant (i.e., entering into more locations, spending less time at home, etc.) following the intervention compared to prior to the intervention. We expect that the passive mobility patterns for individuals who do not receive CBM-I (i.e., the EMA-only monitoring group) will not systematically change over time to the same extent, given these individuals are expected to show less improvement in their social anxiety symptoms in the intervention’s absence. We note that some improvement is still anticipated for the EMA-only group, for two reasons. First, Truong et al. (2017)’s work suggests that the act of monitoring has some therapeutic benefits. Second, previous analyses completed on self-reported anxiety outcomes in this dataset indicated comparable improvement for all participants’ in-the-moment anxiety throughout the five-week study (see Daniel et al., 2020).

Length of Homestay

Spending long periods of time at home (and therefore away from public and other social spaces) may indicate social avoidance, a core feature of social anxiety. Therefore, following Chow and colleagues (2017), we expect that individuals assigned to CBM-I (vs. an EMA-only control condition) will show a relatively greater decrease in the amount of time they spend at home following the Week 3 intervention (by comparing the proportion of time spent at home throughout the two weeks immediately prior to the intervention to the proportion of time spent at home throughout the two weeks immediately following the intervention). We expect this pattern of results to hold during weekdays (H1a), during weekday evenings (H1b), and during weekends (H1c).

Time Spent at Others’ Houses

Helpful interventions for social anxiety are expected to reduce social isolation. Therefore, following Boukhechba and colleagues (2018), we expect that individuals assigned to CBM-I (vs. EMA-only) will show a relatively greater increase in the amount of time that they spend at others’ houses following the intervention, both during weekday evenings (H2a) and during weekends (H2b).

Location Entropy

Entropy measures how a person’s time is distributed over different types of locations (Saeb et al., 2016). Entropy calculated using a frequency-based metric captures the distribution of the number of times that an individual visits different types of locations, where lower entropy signals fewer locations visited. Social anxiety is characterized by avoidance of social situations, and social anxiety severity has been associated with visiting fewer locations in daily life (Boukhechba et al., 2017). Helpful interventions are thus expected to increase the breadth of locations into which socially anxious individuals enter. Namely, we expect that individuals assigned to CBM-I (vs. EMA-only) will show a relatively greater increase in frequency-based entropy following the intervention (H3a), suggesting increased exploration of different locations.

Entropy can also be calculated using a time-based metric, which captures the distribution of time a person spends at different types of locations (Saeb et al., 2016). Lower time-based entropy indicates less time spent across a variety of locations. It is possible that a helpful intervention could increase the amount of time that an individual remains in socially threatening situations and/or decrease the amount of time that an individual spends in non-social situations, making this an interesting metric through which to investigate intervention-related change. However, it is not clear in which locations social threat might be perceived for a given individual, making it challenging to establish clear hypotheses. Consequently, this analysis is exploratory.

Circadian Movement

Circadian movement refers to the way in which a participant’s pattern of movement varies over time. It can be defined in two ways: (1) the degree to which a person’s movement pattern following an intervention deviates from their baseline pattern prior to the intervention (Saeb et al., 2015), and (2) how regular (vs. irregular) a person’s movement patterns are over a specified period of time (Wang et al., 2018).

When circadian movement is defined as (1), greater deviation from the baseline pattern following the intervention signals stronger intervention-linked changes to their pattern of movement. We expect that individuals assigned to CBM-I (vs. EMA-only) will show a greater deviation from their baseline pattern in the weeks following the intervention (H4a).

When circadian movement is defined as (2), we can calculate one regularity index for the two weeks leading up to the intervention and another regularity index for the two weeks following the intervention. By comparing the two regularity indices, we can test if a person’s pattern becomes more or less regular following the intervention compared to their pre-intervention regularity index. We expect that regularity in a socially anxious individual’s life can be unhealthy at either extreme: Rigid adherence to a schedule might indicate constricted behavior and an inability to engage in spontaneous social outings, while erratic behavior might indicate emotional instability and an inability to sustain healthy patterns in life (i.e., work and sleep cycles, etc.). We expect that individuals assigned to CBM-I (vs. EMA-only) will show a differential change to their regularity index following the intervention (H4b), but this hypothesis is non-directional.

Methods

Participants

One-hundred and fourteen participants scoring at or above 29 on the Social Interaction Anxiety Scale (SIAS; Mattick & Clarke, 1998) consented to participate in the study in exchange for course credit and/or payment. The a priori cutoff score of 29 was selected to limit enrollment to participants who were experiencing moderate to severe levels of social anxiety symptoms prior to beginning the intervention study, where a score of 29 represents approximately 25% of a standard deviation below the average score observed in a sample diagnosed with social phobia (M = 34.6, SD = 16.4; Mattick & Clarke, 1998). After enrolling, participants were randomly assigned to either receive the CBM-I intervention (n = 59) or to have no intervention added (EMA-only group, n = 55).

Participants were excluded from analyses if GPS data was not collected (n = 10) or if they were assigned to the CBM-I group but did not initiate the first CBM-I training session (n = 5), leaving a final intent to treat sample of N = 98 (n = 49 in the CBM-I group and n = 49 in the EMA-only group).² Demographic information for the overall sample and broken down by intervention group is provided in Table 1.

Table 1.

Sample Demographics

Identity Characteristic	CBM-I (n = 49)	EMA-only (n = 49)	Overall (N = 98)
Age	M = 20.22 SD = 3.12	M = 20.69 SD = 2.95	M = 20.50 SD = 3.03
Gender
Female	36 (73.57%)	36 (73.57%)	72 (73.47%)
Male	13 (26.53%)	13 (26.53%)	26 (26.53%)
Race
White	32 (65.31%)	34 (69.39%)	65 (66.33%)
Asian	7 (14.29%)	10 (20.41%)	17 (17.35%)
Black	4 (8.16%)	2 (4.08%)	6 (6.12%)
Middle Eastern	1 (2.04%)	0 (0%)	1 (1.02%)
Multiracial	5 (10.20%)	3 (6.12%)	8 (8.16%)
Ethnic Identity
Latinx/Hispanic	2 (4.08%)	1 (2.04%)	3 (3.06%)
Not Latinx/Hispanic	46 (93.88%)	48 (97.56%)	94 (95.92%)
Prefer not to answer	1 (2.04%)	0 (0%)	1 (1.02%)
Social Anxiety Severity at Baseline	M = 45.94 SD = 9.57	M = 46.62 SD = 10.75	M = 46.28 SD = 10.12

Note. CBM-I = Cognitive bias modification for interpretations. EMA = Ecological momentary assessment.

Study Procedure

Participants were recruited through advertisements sent to university email listservs for undergraduate and graduate students, through a psychology research participant pool, and through community flyers and online postings seeking “socially anxious individuals aged 18-45 to participate in a 5-week smartphone monitoring study”. Participants provided written informed consent to participate in a five-week smartphone monitoring study “to test a computer program designed to train different ways of thinking, and understand what people think and feel as they interact with their surroundings.” The university’s ethics review board approved the study.

Eligible participants who enrolled in the study completed two in-lab sessions separated by approximately five weeks, each lasting approximately one and a half hours and composed of trait questionnaires, computer tasks, and a speech stressor task. Participants also downloaded a mobile phone application (MetricWire; https://metricwire.com/) for the five-week EMA portion of their study. Research assistants trained participants on the EMA protocol (six randomly timed surveys per day, one end-of-day survey, and one end-of week survey for five weeks) and all EMA survey items.³ MetricWire continuously and passively collected participants’ location using global positioning system (GPS) coordinates throughout the five-week study. Though not used in the current study, MetricWire also passively collected accelerometer data, activity type data (e.g., riding a bike vs. walking), and pedometer (i.e, number of steps) data.⁴

CBM-I Protocol

Enrolled participants were randomly assigned to either the CBM-I intervention group or to the EMA-only group. In addition to completing the daily phone surveys, the CBM-I intervention group was instructed to complete six total sessions of an online ambiguous scenarios training program (following Mathews & Mackintosh, 2000) during the third week of the five-week study. Each CBM-I session included 30 ambiguous scenarios and individuals resolved the emotional meaning of each scenario by filling in missing letter(s) in the last word, which are presented as a word fragment (e.g., “You arrange to meet up with a friend you have not seen for many years. You drive to the station to pick him up. When you arrive, you know he will find spending time with you f_n.”). To reduce negative interpretation bias, or the tendency to rigidly assume ambiguous social situations will resolve negatively, 90% of the scenarios were resolved positively. Participants were encouraged to complete one session each day for six days straight during Week 3 of the study, though participants could stop at any point or take more than six days to complete the training program. Each session took approximately 15 minutes (so the full intervention dose was approximately 90 minutes).

Measures

Trait Social Anxiety Symptoms

Trait social anxiety symptom severity was assessed using the Social Interaction Anxiety Scale (SIAS; Mattick & Clarke, 1998). Prior to enrolling in the study, participants rated their agreement with 20 statements on a 0 (“Not at all characteristic of me”) to 4 (“Extremely characteristic of me”) Likert scale, with higher total scores indicating greater symptom severity. The average SIAS score across the full sample was 46.28 (SD = 10.12), which is nearly one standard deviation above the average SIAS score observed in a sample of individuals diagnosed with social anxiety disorder (M = 34.6, SD = 16.4; Mattick & Clarke, 1998). Internal consistency was excellent (α = .96) in the present sample.

GPS Location Data

GPS location coordinates were sampled each time the MetricWire app detected a significant change in location (i.e., the participant moved more than 50 meters from the previously logged location).⁵ GPS coordinates were used to quantify all outcomes of interest: length of homestay, time spent at others’ houses, location entropy, and circadian movement. Detailed explanation of all GPS data pre-processing steps and feature extraction are included in the online supplement. Human-readable processing scripts are available on OSF (https://osf.io/em4vn/?view_only=b97da9ef22df41189f1302870fdc9dfe).

Analytic Approach

We constructed a series of linear mixed-effects models using the lme4 package (Bates et al., 2015) in R version 3.4.3 (R core team, 2013). P values were obtained using the lmerTest package in R (Kuznetsova et al., 2017). Fixed effects of interest were time (prior to the intervention = 0 vs. following the intervention = 1), study condition (EMA-only = 0 vs. CBM-I = 1), and their interaction. Participants were treated as random effects with a random intercept and no random slopes. This problem is formulated as the model equation below, where Y_ij is the model outcome for each participant at each timepoint, w_1ij is the level 1 time predictor variable that varies within-and between-person, and w_2i is the level 2 condition predictor variable that varies between-person.

$$Y_{ij}={\gamma }_{00}+{\gamma }_{01}{w}_{1ij}+{\gamma }_{02}{w}_{2i}+{\gamma }_{03}{w}_{1ij}{w}_{2i}+{\mu }_{0i}+e_{ij}$$

Outcomes were derived from passively collected GPS data and were characterized according to the steps detailed in the online supplement. Briefly, mobility characteristics were calculated for each participant for each outcome of interest and were then coded with respect to whether they occurred during the first two weeks of data collection or during the final two weeks of data collection (i.e., the two weeks immediately following the Week 3 intervention). Normality was tested by visual inspection. Following recent guidelines for reporting effect sizes from multilevel models (Nakagawa et al., 2017), we calculated two R² indices that account for fixed and random effects in multilevel modeling: marginal R²_m (i.e., the proportion of the total variance explained by the fixed effects), and conditional R²_c (i.e., the proportion of the total variance explained by both fixed and random effects).

Results

Length of Homestay

Weekday Days

Regarding Hypothesis 1a, we observed a significantly different change in the proportion of time spent at home between 6am and 4pm on Mondays through Fridays for participants in the EMA-only condition relative to the CBM-I condition over the course of the study (b = .09, SE = .03, p = .002). However, the direction of this time-by-condition effect was in an unexpected direction, such that participants in the EMA-only condition showed a larger decrease in the proportion of time they spent at home, compared to other places, during weekday days following the intervention compared to individuals in the CBM-I only condition. Overall, participants spent on average 2.45 hours at home (SD = 3.73 hours) each weekday between 6am and 4pm.

Weekday Evenings

Regarding Hypothesis 1b, we observed a significant main effect of time on differences in the proportion of time spent at home on Mondays, Tuesdays, Wednesdays, and Thursdays from 4pm to 12am during the final two weeks of monitoring compared to the first two weeks of monitoring (b = −.03, SE = .02, p = .038), such that all participants spent less time at home during weekday evenings relative to other locations in the final two weeks of the study compared to the first two weeks of the study. We observed no main effect of condition (b = −.01, SE = .02, p = .633) nor any time-by-condition effect (b = .02, SE = .02, p = .320). Overall, participants spent on average 1.40 hours at home (SD = 1.89 hours) each weekday evening between 4pm and 12am.

Weekends

Regarding Hypothesis 1c, we observed no time (b = −.03, SE = .03, p = .335), condition (b = −.01, SE = .04, p = .739), or time-by-condition (b = .03, SE = .04, p = .482) differences for the proportion of time spent at home between Friday from 4pm until Sunday at 12am. This indicates that, contrary to hypothesis, the amount of time participants spent at home throughout the weekends was comparably stable over time for both groups. Overall, participants spent on average 13.73 hours at home (SD = 13.55 hours) throughout each weekend.

Time Spent at Others’ Houses

Weekday Evenings

Inconsistent with Hypothesis 2a, we observed no time (b = −.03, SE = .02, p = .187), condition (b = .05, SE = .03, p = .078), or time-by-condition (b = −.02, SE = .03, p = .559) differences in the proportion of time spent at other’s houses relative to one’s own home between 4pm and 12am on Mondays through Thursdays. This indicates that, contrary to hypothesis, the amount of time participants spent at other people’s homes throughout the weekday evenings was comparably stable over time for both groups. Overall, participants spent on average .88 hours at others’ houses (SD = 2.35 hours) each weekday evening between 4pm and 12am.

Weekends

Inconsistent with Hypothesis 2b, we observed no time (b = .03, SE = .04, p = .544), condition (b = .04, SE = .05, p = .484), or time-by-condition (b = −.05, SE = .06, p = .383) differences in the proportion of time spent at other’s houses relative to one’s own home between Friday from 4pm until Sunday at 12am. This indicates that, contrary to hypothesis, the amount of time participants spent at other people’s homes throughout the weekend was also comparably stable over time for both groups. Overall, participants spent on average 11.24 hours at others’ houses (SD = 18.78 hours) throughout each weekend.

Location Entropy

Frequency-based Entropy

Regarding Hypothesis 3a, we observed an unexpected significant main effect of time on frequency-based entropy (b = −.11, SE = .05, p = .041), such that all participants visited fewer locations in the final two weeks of the study compared to the first two weeks of the study. We observed no main effect for condition (b = .12, SE = .08, p = .164) nor any time-by-condition effect (b = −.00, SE = .08, p = .948).

Time-based Entropy

This exploratory analysis yielded no time (b = −.08, SE = .05, p = .154), condition (b = .09, SE = .07, p = .200), or time-by-condition (b = −.03, SE = .08, p = .671) differences on time-based entropy of locations. This indicates that the amount of time participants spent across different locations was comparably stable over time for both groups.

Circadian Rhythm

Deviation from Baseline Pattern

Inconsistent with Hypothesis 4a, we observed no condition effect on the degree to which a participant’s circadian rhythm in the two weeks following the intervention deviated from their circadian rhythm in the two weeks prior to the intervention (b = 1.26, SE = .67, p = .062).

Regularity Index

Inconsistent with Hypothesis 4b, we observed no time (b = −.05, SE = .03, p = .137), condition (b = .03, SE = .04, p = .395), or time-by-condition (b = −.01, SE = .04, p = .895) differences on regularity indices. This indicates that, contrary to hypothesis, the extent to which participants’ schedules followed a regular pattern over the two weeks prior to the intervention did not change in the two weeks following the intervention, across all participants.

Changes in Social Anxiety Symptoms and Mobility Patterns

We conducted a series of secondary analyses to test whether change in social anxiety symptoms, regardless of intervention condition, was associated with changes in mobility metrics. We specified eight latent change score models in the lavaan R package (Rosseel, 2012), and identified the path of interest in all models as the covariance between latent change in SIAS and latent change among each of the above mobility metrics (with the exception of deviation from baseline circadian rhythm pattern, as this was measured only once). The covariance paths were not significant in any model (ps ranging from 0.211 to 0.922).

Discussion

Although previous analyses on this dataset found that individuals assigned to CBM-I showed uniquely reduced negative interpretation bias following the intervention (i.e., there was evidence of target engagement, Daniel et al., 2020), we did not observe any systematic intervention effects on passive mobility patterns in the present paper. The current null results are consistent with previous results in these data, which showed a general null effect of the intervention on active self-reported measures at both trait (i.e., social anxiety symptom severity) and momentary (i.e., momentary fear of negative evaluation) levels. Although we did observe a few significant effects in the present analyses, we caution against overinterpreting any one of those effects given the large number of tests that we conducted and the clear pattern that we have established in this paper and in Daniel et al. (2020): the CBM-I intervention as it was delivered in the current sample was largely unsuccessful at uniquely shifting behavior in daily life, at least in the short-term and insofar as how behavior was measured in the current study.

Although patterns in passive mobility data have been shown to differentiate between relatively higher and lower trait socially anxious individuals (i.e., time spent at one’s own home: Chow et al., 2017; time at other people’s homes: Boukhechba, et al., 2018; number of distinct locations visited: Boukhechba, et al., 2017), we did not observe evidence to suggest that the current intervention significantly shifted social anxiety symptoms or their associated mobility patterns. That said, previous research has been able to detect within-person differences over time using passive mobility data (Grünerbl et al., 2015). Further, we did not observe robust intervention effects on self-reported social anxiety symptoms in previous analyses. Taken together, we believe that the low dose of CBM-I may not have been strong enough to bring about robust mobility pattern changes across the overall intervention group. Idiographic analyses in those who demonstrated a strong decrease in negative interpretation bias, rather than all people randomly assigned to the intervention, might uncover interesting person-level effects. Further, within-person fluctuations captured by daily-life data may offer meaningful insights about treatment progress (Roczniewska et al., 2020). As such, we encourage readers to not take the pattern of null results described here as a shortcoming of the passive data outcome monitoring approach itself.

Potential reasons for the lack of robust intervention effects include: (1) the intervention dose was relatively low and participants who did not complete the full dose were retained in analyses, (2) training scenarios covered multiple threat domains (vs. exclusively targeting socially ambiguous situations) and were not personalized to the individual’s concerns, (3) this is not a treatment-seeking sample and so participants may not have been motivated to change their anxious thinking and behaviors, (4) two weeks post-intervention may have been too brief a period of time to observe any subtle intervention effects, especially if those effects build over time, and (5) likely limited statistical power. Beyond potential explanations unique to the current sample, it is also notable that the effectiveness of CBM-I in socially anxious populations has been mixed (e.g., Brettschneider et al., 2015). Thus, despite CBM-I offering a scalable way to increase access to care for those suffering with SAD, it remains to be seen how to best and most consistently leverage the potential impact of CBM-I so that patterns in mobility data that have been associated with SAD symptom severity (Boukhechba et al., 2017; Boukhechba et al., 2018; Chow et al., 2017) may be altered as a function of the intervention. For example, researchers have begun testing CBM-I in primary care (Beard et al., 2011) settings. Further, incorporating tele-coaching support (Baumeister et al., 2014) or motivational interviewing components into the intervention protocol may improve intervention effectiveness by increasing participant buy-in and more successful application of the principles of change taught in CBM-I.

Despite the null results for this particular intervention, this paper provides a robust analytical framework within which researchers and clinicians can assess intervention effects in daily life using readily available and passively-collected GPS data. This is one way the field can move towards recapturing the study of behavior in psychological research (Doliński, 2018). Although we cannot publicly share our data given the identifiable nature of GPS, we openly provide our data pre-processing scripts on OSF (https://osf.io/em4vn/?view_only=b97da9ef22df41189f1302870fdc9dfe) so that others can use these resources. Importantly, 81% of US adults now own a smartphone (Pew Research Center, 2019) and smartphone-enabled GPS data can differentiate between within- (e.g., Grünerbl et al., 2015) and between-person mental health traits (e.g., Boukhechba et al., 2018). Further, research groups have been improving our ability to predict state anxiety in daily life using smartphone-based passive sensing (Fukazawa et al., 2019; Jacobson et al., 2020), contributions which can be integrated into the current analytical framework. Thus, the impact of this framework and these resources are two-fold. First, researchers can be better positioned to assess the in-the-moment effects of any mental health intervention deployed through clinical trials. Second, clinicians can be better positioned to monitor client progress with minimal burden for either party.

Considerations for Future Researchers Using GPS Data

To improve future studies that use GPS data to trace meaningful behavioral changes over time, we offer the following suggestions: First, attend to the GPS polling technique used by your sampling software. Some sampling applications (like the one used in the current study) collect a new GPS data point only when the device has moved a set distance (i.e., 50 meters) from the previous GPS data point. Although this sampling approach minimizes battery drain and theoretically reduces redundant data points by capturing only novel movement, it also creates challenges to data integrity. First, meaningful social movement could occur within 50 meters (e.g., two neighboring dorm rooms could be within 50 meters). Second, it is not always clear if the absence of a data point indicates a meaningful lack of participant movement or missing data (consequent to the participant’s phone running out of battery, the technology failing, the participant turning off the GPS sensor, etc.). Although we took steps to mitigate bias that may have been introduced by any confounding between lack of movement and missing data (see the online supplement for detailed steps), it is more straightforward to sample GPS continuously, at a constant rate. While this approach does reduce battery life, the resulting data will more clearly capture any potentially relevant movement and highlight any sampling error.

Second, attend to the specific geographic characteristics of the communities in which you collect data. In the current study, all participants lived in the same Southeastern US college town. The majority of participants were college students and, as such, many participants’ living, social, dining, and working locations were typically in very close proximity with each other, raising challenges about how to infer one life domain from another using only GPS data. We addressed this challenge by processing data according to the steps outlined in the online supplement. Although the specific steps we took were useful for processing data collected in this one city, a major benefit of online interventions and mobile assessment is that researchers are not confined to local participant pools. Researchers who wish to expand to geographically diverse (and therefore more representative) samples must also consider how to best apply these processing steps to GPS points obtained across both rural and densely populated urban areas. Having local, expert knowledge of the different locations and establishments that exist in any given area can increase the speed and accuracy of this data processing pursuit.

Third, attend to the study’s duration. Given GPS data can be collected passively, with little to no burden on the participant, researchers could consider collecting data for longer periods of time pre-and post-intervention than we did here. Studies that rely more exclusively on passive data collection could be well positioned to test a range of theories aimed at understanding bidirectional changes over time at different time scales (Ram et al., 2014).

Conclusion

Despite the dearth of significant CBM-I intervention outcomes in the current sample, the potential value of routine outcome monitoring using low-burden, passively sensed GPS data is notable. In this manuscript, we provide a framework for using a readily available source of objective, behavioral data to measure intervention effectiveness in daily life.

Table 2.

Model Estimates for Length of Homestay Models

	Weekday Days				Weekday Evenings				Weekends
Predictors	b	CI	t	p	b	CI	t	p	b	CI	t	p
(Intercept)	0.25	0.20 – 0.30	9.28	<0.001	0.19	0.16 – 0.23	10.90	<0.001	0.26	0.21 – 0.31	9.51	<0.001
Time	−0.03	−0.08 – 0.01	−1.51	0.131	−0.03	−0.06 – −0.00	−2.07	0.038*	−0.03	−0.09 – 0.03	−0.96	0.335
Condition	−0.03	−0.10 – 0.05	−0.70	0.487	−0.01	−0.06 – 0.04	−0.48	0.633	−0.01	−0.09 – 0.06	−0.33	0.739
Time x Condition	0.09	0.03 – 0.15	3.08	0.002**	0.02	−0.02 – 0.06	1.00	0.320	0.03	−0.05 – 0.11	0.70	0.482
Observations	1960				1568				392
Marginal R2 / Conditional R2	0.005 / 0.179				0.003 / 0.166				0.002 / 0.230

Note. CI = 95% Confidence Interval. b = unstandardized beta values. Weekday days is defined as 6am and 4pm on Mondays through Fridays. Weekday evenings is defined as Mondays through Thursdays from 4pm to 12am. Weekends is defined as Friday from 4pm until Sunday at 12am.

^*= p < .05,

^**= p < .01.

Table 3.

Model Estimates for Time Spent at Others’ Houses Models

	Weekday Evenings				Weekends
Predictors	b	CI	t	p	b	CI	t	p
(Intercept)	0.10	0.06 – 0.14	4.89	<0.001	0.18	0.11 – 0.26	4.76	<0.001
Time	−0.03	−0.06 – 0.01	−1.32	.187	0.03	−0.06 – 0.11	.61	0.544
Condition	0.05	−0.01 – 0.11	1.77	0.076	0.04	−0.07 – 0.14	.70	0.483
Time x Condition	−0.02	−0.07 – 0.04	−.58	0.559	−0.05	−0.17 – 0.06	−.87	0.382
Observations	1568				392
Marginal R2 / Conditional R2	0.009 / 0.140				0.002 / 0.267

Note. CI = 95% Confidence Interval. b = unstandardized beta values. Weekday evenings is defined as Mondays through Thursdays from 4pm to 12am. Weekends is defined as Friday from 4pm until Sunday at 12am.

Table 4.

Model Estimates for Entropy Models

	Frequency-based Entropy				Time-based Entropy
Predictors	b	CI	t	p	b	CI	t	p
(Intercept)	0.97	0.86 – 1.09	16.56	<0.001	0.68	0.58 – 0.77	13.60	<0.001
Time	−0.11	−0.21 – −0.00	−2.04	0.041*	−0.08	−0.18 – 0.03	−1.43	0.154
Condition	0.12	−0.05 – 0.28	1.39	0.164	0.09	−0.05 – 0.23	1.28	0.200
Time x Condition	−0.00	−0.15 – 0.14	−0.07	0.948	−0.03	−0.18 – 0.12	−0.42	0.671
Observations	195				195
Marginal R2 / Conditional R2	0.036 / 0.605				0.029 / 0.440

Note. CI = 95% Confidence Interval. b = unstandardized beta values.

^*= p < .05

Table 5.

Model Estimates for Circadian Rhythm Models

	Deviation from Baseline				Regularity Index
Predictors	b	CI	t	p	b	CI	t	p
(Intercept)	3.62	2.69 – 4.55	7.60	<0.001	0.34	0.28 – 0.39	12.31	<0.001
Time	--	--	--	--	−0.05	−0.11 – 0.01	−1.49	0.137
Condition	1.26	−0.06 – 2.58	1.87	0.062	0.03	−0.04 – 0.11	0.85	0.395
Time x Condition	--	--	--	--	−0.01	−0.09 – 0.08	−0.13	0.895
Observations	1372				196
Marginal R2 / Conditional R2	0.013 / 0.328				0.022 / 0.389

Note. CI = 95% Confidence Interval. b = unstandardized beta values. Statistics are not provided for Time and Time x Condition effects in Deviation from Baseline model because these terms are not included in the overall model.

Data availability statement:

Data supporting the analyses are not openly available due to the identifying nature of GPS data. Feature extraction and data analysis scripts are available at https://osf.io/em4vn/ (Daniel et al., 2019). Non-identifiable data from the overall study collection are available at https://osf.io/eprwt/ (Daniel & Teachman, 2019).