Skip to main content
PMC home page
JMIR mHealth and uHealth logoLink to JMIR mHealth and uHealth
. 2021 Oct 26;9(10):e20638. doi: 10.2196/20638

A Mobile Sensing App to Monitor Youth Mental Health: Observational Pilot Study

Lucy MacLeod 1,#, Banuchitra Suruliraj 2,#, Dominik Gall 3, Kitti Bessenyei 1, Sara Hamm 1, Isaac Romkey 1, Alexa Bagnell 1, Manuel Mattheisen 1, Viswanath Muthukumaraswamy 1, Rita Orji 2,#, Sandra Meier 1,✉,#
Editor: Lorraine Buis
Reviewed by: Christina Ralph-Nearman, Aravind Sesagiri Raamkumar, Danielle Hartigan, Suresh Kumar Mukhiya, Luis Castro, Alexander Adams
PMCID: PMC8579216  PMID: 34698650

Abstract

Background

Internalizing disorders are the most common psychiatric problems observed among youth in Canada. Sadly, youth with internalizing disorders often avoid seeking clinical help and rarely receive adequate treatment. Current methods of assessing internalizing disorders usually rely on subjective symptom ratings, but internalizing symptoms are frequently underreported, which creates a barrier to the accurate assessment of these symptoms in youth. Therefore, novel assessment tools that use objective data need to be developed to meet the highest standards of reliability, feasibility, scalability, and affordability. Mobile sensing technologies, which unobtrusively record aspects of youth behaviors in their daily lives with the potential to make inferences about their mental health states, offer a possible method of addressing this assessment barrier.

Objective

This study aims to explore whether passively collected smartphone sensor data can be used to predict internalizing symptoms among youth in Canada.

Methods

In this study, the youth participants (N=122) completed self-report assessments of symptoms of anxiety, depression, and attention-deficit hyperactivity disorder. Next, the participants installed an app, which passively collected data about their mobility, screen time, sleep, and social interactions over 2 weeks. Then, we tested whether these passive sensor data could be used to predict internalizing symptoms among these youth participants.

Results

More severe depressive symptoms correlated with more time spent stationary (r=0.293; P=.003), less mobility (r=0.271; P=.006), higher light intensity during the night (r=0.227; P=.02), and fewer outgoing calls (r=−0.244; P=.03). In contrast, more severe anxiety symptoms correlated with less time spent stationary (r=−0.249; P=.01) and greater mobility (r=0.234; P=.02). In addition, youths with higher anxiety scores spent more time on the screen (r=0.203; P=.049). Finally, adding passively collected smartphone sensor data to the prediction models of internalizing symptoms significantly improved their fit.

Conclusions

Passively collected smartphone sensor data provide a useful way to monitor internalizing symptoms among youth. Although the results replicated findings from adult populations, to ensure clinical utility, they still need to be replicated in larger samples of youth. The work also highlights intervention opportunities via mobile technology to reduce the burden of internalizing symptoms early on.

Keywords: mobile sensing, youth, psychiatry, feasibility, mobile phone

Introduction

Background

Internalizing disorders are the most common psychiatric problems among youth in Canada, with an estimated 6-month prevalence of 11% to 14% and more girls being affected than boys [1]. Over the last 3 decades, a steep increase in the prevalence of internalizing disorders has been observed; unfortunately, this trend is likely to continue [1]. Internalizing disorders typically have an early onset in childhood or adolescence and are characterized by high rates of relapse and chronicity, often resulting in substantial impairment across the lifespan. They are among the top 10 global causes of years lived with disability [2], and severely affect psychosocial development [1], family and social functioning, long-term health outcomes [3], and increased the risk of death by suicide [4]. Unfortunately, only 20% of youth with these disorders receive adequate mental health care [5]. If health professionals cannot predict and efficiently prevent new onset and poor outcomes of internalizing disorders in youth, then this field faces a crisis. Particularly, intervening at critical moments—that is—during mental health crises, including times of risk of suicide, self-harm, psychotic breakdown, substance use relapse, bullying, and interpersonal loss can have a major impact on improving youth mental health.

Current methods of predicting internalizing disorders usually rely on subjective symptom ratings obtained at discrete time points during routine clinical care (eg, clinical monitoring and screening). However, clinical decision-making based on such subjective information is challenging, as changes in symptoms might be subthreshold, context-dependent, or too subtle to be captured using subjective patient ratings. Symptom recall in patients might also not be accurate, potentially biased by current symptom severity [6] and environmental stressors [7]. Moreover, subjective self-reports often fail to reflect symptom variability and context reactivity. Therefore, novel prediction tools need to be developed to meet the highest standards of reliability, feasibility, scalability, and affordability. Given the large proportion of youths who experience mental health crises and do not receive treatment [8], these prediction tools must also encompass those who are not in traditional mental health care.

The recent proliferation of mobile sensing technologies offers health care professionals unprecedented access and insights into youth behavior and mood. Of profound clinical importance is the ability of these technologies to unobtrusively record aspects of youth behaviors in their daily lives with the potential to make inferences about their mental health states [9]. Crucially, these technologies make it possible to obtain detailed moment-by-moment information on youth behaviors and mood while requiring only minimal, if any, active involvement of the users [10]. Smartphones may be especially helpful prediction tools, not only because they afford a wide variety of types of behavioral data that can be automatically recorded by their built-in sensors, but also because they are integral to a youth’s life and nearly 80% of youths own at least 1 smartphone. Thus, smartphones might allow long-term clinical monitoring of internalizing disorders at low cost and high scalability. Collecting high-quality, passive mobile sensing data might further enable the generation of predictive algorithms for solving previously intractable problems and identifying risk states before they manifest as full-blown internalizing disorders.

Previous Work

The utility of mobile sensing approaches to predict internalizing symptoms has been examined in multiple studies involving adult populations. Several mobile sensor features mirror internalizing symptoms in these studies [11-14]. In particular, GPS data have been shown to be of value in the prediction of internalizing symptoms. For example, the number of different locations visited was found to be negatively associated with social anxiety [11,12] and depression [15-17]. Study participants with social anxiety and depression were also found to avoid public areas and engage in less leisure activities, choosing instead to spend more time at home after school [11,12]. Studies exploring phone use data have reported the number of outgoing calls and messages to negatively correlate with internalizing symptoms [13,15,18]. In addition, ambient light data, an indicator of quality of sleep, were found to be associated with internalizing symptoms [19]. However, to date, mobile sensing studies in the context of youth have been limited [20]. We are only aware of a small pilot study exploring the utility of an Android mobile sensing app among 11 youth patients diagnosed with a major depressive disorder [21]. In that study, the daily number of steps, SMS frequency, and average call duration across a 2-week window were found to be highly correlated with the clinical symptoms of depression. However, in contrast to many studies among adults [17,22], clinical symptoms of depression correlated positively with entropy among youth [21]. This is in line with findings from Jongs et al [22], who also described strong age-related differences in entropy and other GPS-related measures of mobility in a sample aged 18-90 years. Similarly, phone use patterns have been shown to vary between youths and adults. Using a mobile sensing approach, Christensen et al [23] observed that younger age was significantly associated with more screen time. Importantly, screen time has been suggested to be differentially related to internalizing symptoms across age groups [24]. Thus, it is important to explore whether the mobile sensor features found to mirror internalizing symptoms in adults also predict such symptoms in youth.

Study Aims

Accordingly, this study aims to test the utility of passively collected smartphone sensor data (ie, incoming and outgoing calls, text messages, and GPS data), gathered over 14 days via a mobile sensing app, as the predictors of internalizing symptoms in youth. Studies have shown that internalizing symptoms are common in the general population [25], and the combination of patient and general population samples is meaningful for examining this pathology on a continuum. Therefore, this study aims to collect data from both clinical and nonclinical youth populations. Moreover, the study also uses a mobile sensing app that works on both iOS and Android operating systems, ensuring the generalizability of our results. By including iOS and Android users from clinical and nonclinical settings, we have advanced the inclusivity of current mobile sensing studies in youth [21]. In addition, we hypothesize that mobile sensing data will be associated with internalizing symptoms among youths aged 10-21 years. As outlined, the only study using mobile sensing to predict internalizing symptoms in youth included only 11 participants, thus requiring further evaluation in a larger and more diverse sample. This is especially important as studies have suggested age-related differences in mobile sensing patterns. In contrast to previous studies, we also aim to correct for clinical comorbidities, thus enabling the identification of symptom-specific associations of mobile sensing data. Finally, given the novel nature of these data, we have some exploratory aims, namely, whether the data indicate any nuanced biomarkers that may be worth examining in future iterations of this work.

Methods

Overview

Youth participants from clinical and nonclinical populations were recruited for a 14-day observational study, in which a custom app was installed on their personal Android or iOS phone. Self-report measures of anxiety and depression were collected at the beginning of the study. Throughout the duration of the study, the smartphone app passively recorded multiple indices of the youths’ daily life behaviors in an unobtrusive manner. The indices were selected based on findings that demonstrated their potential to make inferences about mental health states [26,27]. These indices included geolocation, sleep, phone use, calls, and messages. A set of features were designed and used to extract higher-level information from these data. Statistical analyses were performed to determine whether a significant relationship existed between participants’ self-reported symptoms of anxiety or depression and these features. The Sciences Research Ethics Board of the Izaak Walton Killam Health Centre in Halifax, (Nova Scotia, Canada) approved this study, which also included the collection of location data.

Study Procedure

Youth were recruited via posters and social media from the outpatient psychiatry clinic at Izaak Walton Killam Health Centre and the local community in Halifax between February 2020 and July 2020. Youths interested in the study were referred over a link to the lab’s screening page. The study inclusion criteria were as follows: subjects should (1) be between 10 and 21 years old, (2) reside in Canada, (3) be fluent in English, and (4) have an Android or iOS phone that would be in use as the primary mobile phone during the study period (14 days). There were no exclusion criteria for this study. Youth participants who met the inclusion criteria could read a description of the study, which included an informed consent guide. There was no deception nor were the study aims omitted for the participants. All participants provided written informed consent before participation. For youths aged 14 or younger, we required, in addition to their own assent, one of their parents to consent to their study participation. After providing consent, the participants were asked to complete a short web-based survey regarding their clinical symptomatology.

Next, the youth participants were sent instructions to install the app and provided with individualized log-in credentials for using it. Once installed, the study app guided the youths through a short setup, during which they were asked to provide the app with the necessary permissions to access their data, followed by a log-in. Immediately following the setup and log-in, the app began to continuously collect data in the background. No further actions or interactions with the study app were performed until the end of the study, and data were uploaded for 14 days. At this time, the youths received a notification, informing them that the study had ended and directed them to uninstall the app from their phone. A study ID was assigned to each participant, and documents connecting the ID and the participant’s name, demographics, and clinical data were kept separate. The data on the phone were anonymous and only identifiable through the participants’ study ID. To encourage youths to participate in our study, we offered them a monetary reward of Can $20.00 (US $15.82). The REDCap (Research Electronic Data Capture) platform [28] was used for web-based screening, consent, and clinical surveys. REDCap is a secured web-based app that allows researchers to collect, store, and manipulate research data.

Self-report Assessment

Youth participants completed two self-report measures of internalizing symptoms in digital form at the beginning of the 14-day study. Anxiety symptoms were assessed using the Screen for Child Anxiety-Related Emotional Disorders (SCARED) [29]. SCARED is a self-report instrument for children and their parents, including 41 items that screen for several types of 4 anxiety disorders, namely, generalized anxiety disorder, separation anxiety disorder, panic disorder, and social anxiety disorder [30]. Both the child and parent versions of SCARED in clinical samples demonstrated good internal consistency (α=.74-.93), test-retest reliability (intraclass correlation coefficients 0.70-0.90), and discriminative validity (both between anxiety and other disorders and within anxiety disorders) [29,30]. We administered a 33-item version of SCARED, excluding items related to separation anxiety disorders.

Depression symptoms were assessed using the Center for Epidemiological Studies Depression Scale for Children (CES-DC), which includes 20 items. CES-DC screens for behavioral and cognitive components of depression [31]. The sensitivity of the CES-DC is 85% and can be considered good [32]. In addition, the CES-DC showed results similar to the results of other highly validated instruments [33,34].

To be able to adjust for comorbid externalizing symptomatology, we also required the youths to complete the Adult Attention-Deficit Hyperactivity Disorder Self-Report Scale (ASRS) with adolescent prompts. On the basis of an 18-item scale, the ASRS assesses hyperactivity or impulsivity and inattentiveness symptoms. The ASRS demonstrated a good internal consistency (α=.80) and reliability (α=.79) in adolescent samples [35]. The SCARED, CES-DC, and ASRS instruments asked youth participants to evaluate their symptoms over the past 3 months.

Smartphone Data Collection

A mobile sensing app was designed and created to collect all the study data. The PROSIT (Predicting Risk and Outcomes of Social Interactions) tool captures multiple indices of a youth’s daily life behaviors via the naturalistic use of a smartphone (Figure 1). In this study, we used the following features of the PROSIT tool: first, the tool collected GPS data every time the youths moved within a 20-meter radius, preventing the identification of their home address or precise geographical location to preserve privacy. Second, the app logged call events (ie, time and date of the call, duration, and contact of both incoming and outgoing calls), and screen on or off events (ie, time and date). Finally, we sampled ambient light data as an indicator of sleep using the PROSIT tool. Calls and phone use were event-based sensor streams, whereas GPS and ambient light data were sampled as time series. Upon installation, the PROSIT tool continuously ran in the background and collected sensor and use data. The data were first stored locally and then automatically uploaded every hour to a remote server when the phone was in an idle state, and the Wi-Fi was connected. To protect user privacy, all collected sensitive personal data, such as contact details (ie, names and phone numbers), were anonymized during data collection by the app through the built-in cryptographic hash functions, thus guaranteeing that user identity was never recorded either locally or remotely. As the tool is not yet distributed through the official mobile app stores from Apple and Google, a TestFlight-based distribution for iOS and downloadable Android packages for Android was used in this study. The iOS version of the tool was built in the native Swift language using XCode 13 (Apple), whereas the Android version was built using Java and Kotlin languages using Android Studio 4.1 (Google). Importantly, the PROSIT tool also captured steps, notifications, installed apps, app use time, typed text, and music listened to. However, we decided not to include these measures in the current analyses, as these data were slightly differently sampled among Android and iOS users because of certain iOS-specific restrictions. More information on the PROSIT tool is presented in Multimedia Appendix 1 [13,36-50].

Figure 1.

Figure 1

Open in a new tab

Sensors used to capture behavioral traits.

Participant

A total of 161 youth participants downloaded the PROSIT tool into their phone, 6.8% (11/161) withdrew from the study (only 3/161, 1.8% due to privacy concerns), 15.5% (25/161) provided less than 14 days of data, and 1.8% (3/161) did not provide data on sensors relevant to this study. Therefore, this study focused on 75.7% (122/161) of the initial youth participants.

Youth participants were, on average, 18 (SD 2.76) years old and 78.6% (96/122) were female. A total of 5.7% (7/122) participants identified themselves as indigenous, 5.7% (7/122) as black Canadian, 6.5% (8/122) as Asian, and the other 81.9% (100/122) were identified as non-Hispanic Caucasians. As a marker of the youths’ socioeconomic status, we obtained information on maternal education: 4.1% (5/122) of the youths’ mothers had not finished high school, 14.7% (18/122) had finished high school, 23.7% (29/122) had received further education, 54.1% (66/122) had attended university, and 3.2% (4/122) cases lacked this information. A total of 78.6% (96/122) youths were iOS users and 21.3% (26/122) were Android users. In total, 24.5% (30/122) of the youth participants had a lifetime diagnosis of depression, 31.9% (39/122) had a lifetime diagnosis of generalized anxiety disorder, and 9.8% (12/122) had a lifetime diagnosis of social phobia. Regarding the self-report measures of internalizing symptoms, SCARED mean 33.02 (SD 14.79) and range 3-62, the CES-DC mean 32.59 (SD 15.23) and range 1-58. During the study period, 9.8% (12/122) youths were diagnosed with depression, 11.4% (14/122) were diagnosed with generalized anxiety, and 4.9% (6/122) were diagnosed with social phobia. Finally, we excluded two participants from the call analyses and one participant from the GPS analyses, as they were extreme outliers.

Feature Extraction

Raw unobtrusive mobile sensing data with respect to mobility, social interactions, phone use, and sleep were aggregated into daily summaries. As outlined above, we logged each GPS data point when the displacement from the last point was larger than 20 m. From these data, we extracted the number of locations visited per hour for at least 2 minutes. The distance between stationary states is defined by the Haversine distance function, which calculates the distance between a pair of coordinates denoted by latitude and longitude. We extracted the total distance traveled (times traveled a day), normalized entropy, and time spent at one location (ie, stationary at work or at home), 2-9 locations (ie, moving), and over 10 locations (ie, transportation) per hour for each participant on a daily basis. Normalized entropy was defined as

Normalized Entropy = −∑Ni pilog pi(2) / log N (1)

where each i=1, 2,..., N represented a location cluster (in a 150-m radius), N denoted the total number of location clusters, and pi was the percentage of time the participant spent at the location cluster i. High entropy indicated that the participant spent time more uniformly across different location clusters, whereas lower entropy indicated a greater inequality in the time spent across the clusters. Unlike entropy, the normalized entropy is invariant to the number of clusters; thus, it depends solely on the distribution of the visited location clusters. The value of normalized entropy ranges from 0 to 1, where 0 indicates that all location data points belong to the same cluster, and 1 implies that they are uniformly distributed across all clusters. For each phone call, we logged the timestamp, type (incoming, outgoing, and missing), and duration. For each user, we computed the total number of phone calls (connected, incoming, and outgoing) and total call duration on a daily basis. The raw mobile phone screen on or off events were transformed into two features: (1) the total number of times the screen was turned on per day and (2) the total amount of screen time per day (calculated as the difference between the number of screen on or off events). From the ambient light data, we logged changes in light intensity and calculated the time spent at each light intensity level. Next, we computed the total average light intensity, time spent in the lowest and highest light intensity ranges during nighttime per day (11 PM-7 AM), which was used as an indicator of sleep quality.

Statistical Analysis

After completing feature extraction and exclusion of extreme outliers, we performed the Spearman correlation analysis of sensor data over 14 days with SCARED and CES-DC scores of youth participants at a dimensional level. All analyses were adjusted for age, sex, and maternal education as the indicators of socioeconomic status, comorbidities, and the smartphone operating system. In addition, we included the month of assessment to account for potential impact of the pandemic on the results. We also fitted linear regressors by comparing models with and without passively collected smartphone sensor data as predictors. Finally, we assessed the discriminant validity of the predicted internalizing symptoms; for example, we ran correlations between the SCARED scores based on smartphone biomarkers and CES-DC and ASRS scores.

Results

Mobile Sensing Data Characteristics

The mean and SD of mobile sensing features across youth participants are presented in Table 1. To predict the assessment of internalizing symptoms (SCARED and CES-DC), we used averages across individuals as predictor variables.

Table 1.

Descriptive statistics of mobile sensor data and features.

Feature Values, mean (SD)
Mobility

Time sedentary 18.82 (3.60)

Time moving 4.02 (3.56)

Distance traveled 6.01 (3.56)

Entropy 0.48 (0.20)
Social interaction

Call duration 15.22 (13.63)

Incoming calls 1.81 (0.94)

Outgoing calls 2.11 (1.20)

Connected calls 2.30 (1.25)
Phone usage

Screen use time 249.58 (109.28)

Screen use unlocks 53.66 (38.71)
Sleep

Ambient light intensity 304,883 (95,828)
Clinical symptoms

SCAREDa 33.02 (14.79)

CES-DCb 32.59 (15.23)
Open in a new tab

aSCARED: Screen for Child Anxiety-Related Emotional Disorders.

bCES-DC: Center for Epidemiological Studies Depression Scale for Children.

Relationship Between Smartphone Data and Psychometric Scores

We used the pairwise Spearman coefficient as an indicator of the correlations between mobile sensing data and clinical self-report measures of internalizing symptoms (SCARED, CES-DC). The mobile sensing data were represented by mobility, social interaction, phone use, and sleep-related features extracted using methods described in Feature Extraction and Statistical Analysis of the Methods section. The analysis and correlations are shown in Tables 2 and 3.

Table 2.

Spearman correlation coefficients between internalizing symptoms and mobility features.

Feature Anxiety Depression

Correlation coefficient (r) P value Correlation coefficient (r) P value
Distance travelled 0.230 .02 −0.271 .006
Time moving −0.248 .01 0.293 .003
Time sedentary 0.234 .02 −0.266 .007
Entropy 0.076 .46 −0.042 .69
Open in a new tab

Table 3.

Spearman correlation coefficients between internalizing symptoms and social interactions, phone use, sleep-related features.

Feature Anxiety Depression

Correlation coefficient (r) P value Correlation coefficient (r) P value
Call duration 0.024 .81 −0.035 .73
Incoming calls 0 .99 −0.072 .48
Outgoing calls −0.069 .50 −0.223 .03
Connected calls −0.106 .31 −0.110 .29
Ambient light intensity −0.136 .12 0.227 .02
Screen use 0.203 .049 −0.113 .28
Open in a new tab

Table 2 shows the correlations between mobility and internalizing symptom scores, where mobility is captured using GPS data. From the table, we observe that youths with more severe depression and higher CES-DC scores spent more time stationary (r=0.293; P=.003) and were significantly less mobile (r=−0.271; P=.006). In contrast, less time spent stationary (r=−0.249; P=.01) and higher mobility (r=0.234; P=.02) were associated with more anxiety symptoms as measured by SCARED. No correlations between normalized entropy values and internalizing symptoms were observed.

The correlations of social interactions, phone use, and sleep-related features with internalizing symptom scores are shown in Table 3. The results indicate that higher depression scores are significantly correlated with lower social interaction levels, such as fewer outgoing phone calls (r=−0.223; P=.03). Higher anxiety scores also correlated with fewer outgoing phone calls (r=−0.293; P=.003); however, this correlation became insignificant after adjusting for comorbid depression (r=−0.069; P=.50). In addition, higher ambient light intensity during the night, which is a proxy for poorer sleep quality, was correlated with higher depression scores (r=0.227; P=.02), but not anxiety scores (r=−0.15; P=.12). Conversely, higher smartphone screen use was only correlated with higher anxiety scores (r=0.203; P=.049).

Thus, internalizing symptoms assessed by clinical instruments were significantly correlated with mobility levels and social interactions captured by passive smartphone sensor data. We also observed significant correlations between other daily living contexts (ie, ambient light intensity and smartphone screen use) and internalizing symptoms.

The inclusion of mobility, social interactions, phone use, and sleep-related features significantly improved the fit of models predicting more severe depression and higher CES-DC scores (F17,62=2.036; P=.04), as well as more severe anxiety and higher SCARED scores (F17,62=2.256; P=.02). Finally, predicted SCARED scores from smartphone biomarkers correlated with higher depression scores (r=0.258; P=.01), but not ASRS scores (r=0.062; P=.55). Consistently, predicted CES-DC scores correlated with higher anxiety scores (r=0.258; P=.01), but not ASRS scores (r=0.020; P=.85).

Discussion

Principal Findings

Our results support the hypothesis that passively collected smartphone sensor data can provide sufficient information for successfully predicting internalizing symptoms among youth. The proposed approach worked well for youths who were usually heavy smartphone users. Remarkably, we were able to include youths as young as 10 years of age in our completely remote study design, demonstrating the feasibility of mobile sensing among youth. The data also underlined high acceptability; even youths with severe clinical symptomology reported only minimal issues in the use of the app. Dropouts due to privacy concerns can be considered low (3/161, 1.9%).

The study indicated that there were significant correlations between internalizing symptoms and smartphone biomarkers. Lower levels of mobility, fewer social interactions, and higher ambient light intensity during the night, used as a proxy for poorer sleep quality, were correlated with higher depression symptoms. Higher levels of mobility, fewer social interactions, and higher smartphone screen use were correlated with higher anxiety symptoms among youth. The study also showed that by adding passively collected smartphone sensor data to prediction models, we were able to achieve better model fits. Finally, the results indicate the potential of smartphone biomarkers to have a discriminative nature by predicting other internalizing symptoms but not adult attention-deficit hyperactivity disorder symptoms.

Here, we expand on the only previous study among youth showing that light intensity and smartphone screen use in addition to mobility and social interactions might be valuable features for predicting internalizing symptoms among youth [21]. In addition, we confirmed the potential discriminative validity of smartphone biomarkers observed among adults in a youth sample [51]. Although our results on the number of outgoing calls negatively correlating with internalizing symptoms are in line with previous studies in adults [13,15,18], differences in mobility patterns of youths with high anxiety symptoms as compared with high depression symptoms have not been described thus far [11,14,21]. However, these findings have to be interpreted considering the recent pandemic and the behavioral changes accompanying it [52]. It is easy to argue that the time spent outside during a pandemic might increase anxiety symptoms.

Limitations and Future Directions

This study has some limitations that provide directions for future research. First, the conclusions must be considered preliminary, given the relatively small sample size of 122 youths and require further replication. Larger samples will also allow for more power to assess potentially moderating effects (eg, sex, trauma exposure), especially as the cost-effectiveness of mobile sensing by using the patients’ own smartphones is fostering the scalability of the approach. However, in contrast to many previous studies [11], we were able to work with a relatively heterogeneous real-world sample of youths, including patients with severe symptomatology, which may reduce the generalizability of our findings. Second, through the use of passively collected smartphone sensor data, we may miss relevant information on social interactions; for example, it can be difficult to infer the quality of the social contact based on the metadata of call logs. Third, our study focused on identifying between-person differences; the identified predictors might not be of value in predicting within-person variability of internalizing symptoms. Fourth, it is not possible to make any claims regarding causation between mobility, screen time, sleep, social interactions, and internalizing symptoms. It cannot be determined if fewer social interactions lead to more internalizing symptoms or vice versa. Fifth, our study was conducted during the recent pandemic that led to a multitude of behavioral changes [52,53]; thus, the results might not be generalizable to a postpandemic situation. However, our study findings were largely consistent with those of previous studies conducted in adult populations, supporting the reliability of mobile sensing results.

The next steps should focus on exploring how youth patients, their parents, and clinicians would prefer to be notified about the youths’ passively collected smartphone sensor data, what inferences can be drawn about youths’ mental health states from these data, and what actions they would find useful and acceptable in response to these data. For example, early warning signs can be identified from smartphone biomarkers, triggering early interventions. In addition, the enthusiasm for exciting new possibilities associated with passively collected smartphone sensor data must be carefully evaluated regarding the challenges of privacy and data security.

Conclusions

In sum, our study expands recent efforts to use passively collected smartphone sensor data for predicting internalizing symptoms among youth. Such accurate behavioral assessments might be especially important for youths with internalizing symptoms, given that the occurrence of these symptoms is more frequent than is typically reported. However, this method of harnessing naturally occurring behavioral data is certainly relevant for identifying and better understanding a range of maladaptive thoughts and behaviors that underlie psychiatric conditions more broadly. Finally, it is equally important to be sensitive and treat passively collected smartphone sensor data as sensitive health information and ensure adequate privacy and security controls are in place before wider use.

Abbreviations

ASRS

Adult Attention-Deficit Hyperactivity Disorder Self-Report Scale

CES-DC

Center for Epidemiological Studies Depression Scale for Children

PROSIT

Predicting Risk and Outcomes of Social Interactions

REDCap

Research Electronic Data Capture

SCARED

Screen for Child Anxiety-Related Emotional Disorders

Multimedia Appendix 1

The PROSIT (Predicting Risk and Outcomes of Social Interactions) app.

Footnotes

Conflicts of Interest: None declared.

References

  • 1.Melkevik O, Hauge LJ, Bendtsen P, Reneflot A, Mykletun A, Aarø LE. Associations between delayed completion of high school and educational attainment and symptom levels of anxiety and depression in adulthood. BMC Psychiatry. 2016 Mar 15;16:64. doi: 10.1186/s12888-016-0765-1. https://bmcpsychiatry.biomedcentral.com/articles/10.1186/s12888-016-0765-1 .10.1186/s12888-016-0765-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Walker ER, McGee RE, Druss BG. Mortality in mental disorders and global disease burden implications: a systematic review and meta-analysis. JAMA Psychiatry. 2015 Apr;72(4):334–41. doi: 10.1001/jamapsychiatry.2014.2502. http://europepmc.org/abstract/MED/25671328 .2110027 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Merikangas KR, He J, Burstein M, Swanson SA, Avenevoli S, Cui L, Benjet C, Georgiades K, Swendsen J. Lifetime prevalence of mental disorders in U.S. adolescents: results from the National Comorbidity Survey Replication--Adolescent Supplement (NCS-A) J Am Acad Child Adolesc Psychiatry. 2010 Oct;49(10):980–9. doi: 10.1016/j.jaac.2010.05.017. http://europepmc.org/abstract/MED/20855043 .S0890-8567(10)00476-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Meier SM, Mattheisen M, Mors O, Mortensen PB, Laursen TM, Penninx BW. Increased mortality among people with anxiety disorders: total population study. Br J Psychiatry. 2016 Sep;209(3):216–21. doi: 10.1192/bjp.bp.115.171975. http://europepmc.org/abstract/MED/27388572 .S0007125000244863 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Hintzpeter B, Klasen F, Schön G, Voss C, Hölling H, Ravens-Sieberer U, BELLA study group Mental health care use among children and adolescents in Germany: results of the longitudinal BELLA study. Eur Child Adolesc Psychiatry. 2015 Jun;24(6):705–13. doi: 10.1007/s00787-015-0676-6. [DOI] [PubMed] [Google Scholar]
  • 6.Wells JE, Horwood LJ. How accurate is recall of key symptoms of depression? A comparison of recall and longitudinal reports. Psychol Med. 2004 Aug;34(6):1001–11. doi: 10.1017/s0033291703001843. [DOI] [PubMed] [Google Scholar]
  • 7.Goldfarb EV, Mendelevich Y, Phelps EA. Acute stress time-dependently modulates multiple memory systems. J Cogn Neurosci. 2017 Nov;29(11):1877–94. doi: 10.1162/jocn_a_01167. [DOI] [PubMed] [Google Scholar]
  • 8.Thornicroft G, Chatterji S, Evans-Lacko S, Gruber M, Sampson N, Aguilar-Gaxiola S, Al-Hamzawi A, Alonso J, Andrade L, Borges G, Bruffaerts R, Bunting B, de Almeida J, Florescu S, de GG, Gureje O, Haro JM, He Y, Hinkov H, Karam E, Kawakami N, Lee S, Navarro-Mateu F, Piazza M, Posada-Villa J, de Galvis Y, Kessler RC. Undertreatment of people with major depressive disorder in 21 countries. Br J Psychiatry. 2017 Feb;210(2):119–24. doi: 10.1192/bjp.bp.116.188078. doi: 10.1192/bjp.bp.116.188078.bjp.bp.116.188078 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Torous J, Larsen ME, Depp C, Cosco TD, Barnett I, Nock MK, Firth J. Smartphones, sensors, and machine learning to advance real-time prediction and interventions for suicide prevention: a review of current progress and next steps. Curr Psychiatry Rep. 2018 Jun 28;20(7):51. doi: 10.1007/s11920-018-0914-y.10.1007/s11920-018-0914-y [DOI] [PubMed] [Google Scholar]
  • 10.Torous J, Kiang MV, Lorme J, Onnela J. New tools for new research in psychiatry: a scalable and customizable platform to empower data driven smartphone research. JMIR Ment Health. 2016 May 5;3(2):e16. doi: 10.2196/mental.5165. http://mental.jmir.org/2016/2/e16/ v3i2e16 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Boukhechba M, Chow P, Fua K, Teachman BA, Barnes LE. Predicting social anxiety from global positioning system traces of college students: feasibility study. JMIR Ment Health. 2018 Jul 04;5(3):e10101. doi: 10.2196/10101. https://mental.jmir.org/2018/3/e10101/ v5i3e10101 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Chow PI, Fua K, Huang Y, Bonelli W, Xiong H, Barnes LE, Teachman BA. Using mobile sensing to test clinical models of depression, social anxiety, state affect, and social isolation among college students. J Med Internet Res. 2017 Mar 03;19(3):e62. doi: 10.2196/jmir.6820. http://www.jmir.org/2017/3/e62/ v19i3e62 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Place S, Blanch-Hartigan D, Rubin C, Gorrostieta C, Mead C, Kane J, Marx BP, Feast J, Deckersbach T, Pentland AS, Nierenberg A, Azarbayejani A. Behavioral indicators on a mobile sensing platform predict clinically validated psychiatric symptoms of mood and anxiety disorders. J Med Internet Res. 2017 Mar 16;19(3):e75. doi: 10.2196/jmir.6678. http://www.jmir.org/2017/3/e75/ v19i3e75 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Saeb S, Lattie EG, Kording KP, Mohr DC. Mobile phone detection of semantic location and its relationship to depression and anxiety. JMIR Mhealth Uhealth. 2017 Aug 10;5(8):e112. doi: 10.2196/mhealth.7297. http://mhealth.jmir.org/2017/8/e112/ v5i8e112 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Sarda A, Munuswamy S, Sarda S, Subramanian V. Using passive smartphone sensing for improved risk stratification of patients with depression and diabetes: cross-sectional observational study. JMIR Mhealth Uhealth. 2019 Jan 29;7(1):e11041. doi: 10.2196/11041. https://mhealth.jmir.org/2019/1/e11041/ v7i1e11041 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Pratap A, Atkins DC, Renn BN, Tanana MJ, Mooney SD, Anguera JA, Areán PA. The accuracy of passive phone sensors in predicting daily mood. Depress Anxiety. 2019 Jan;36(1):72–81. doi: 10.1002/da.22822. http://smj.sma.org.sg/5101/5101sa1.pdf . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Saeb S, Zhang M, Karr CJ, Schueller SM, Corden ME, Kording KP, Mohr DC. Mobile phone sensor correlates of depressive symptom severity in daily-life behavior: an exploratory study. J Med Internet Res. 2015 Jul 15;17(7):e175. doi: 10.2196/jmir.4273. http://www.jmir.org/2015/7/e175/ v17i7e175 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Wang Y, Ren X, Liu X, Zhu T. Examining the correlation between depression and social behavior on smartphones through usage metadata: empirical study. JMIR Mhealth Uhealth. 2021 Jan 06;9(1):e19046. doi: 10.2196/19046. https://mhealth.jmir.org/2021/1/e19046/ v9i1e19046 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Narziev N, Goh H, Toshnazarov K, Lee SA, Chung K, Noh Y. STDD: Short-Term Depression Detection with passive sensing. Sensors (Basel) 2020 Mar 04;20(5):1396. doi: 10.3390/s20051396. https://www.mdpi.com/resolver?pii=s20051396 .s20051396 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Melbye S, Kessing LV, Bardram JE, Faurholt-Jepsen M. Smartphone-based self-monitoring, treatment, and automatically generated data in children, adolescents, and young adults with psychiatric disorders: systematic review. JMIR Ment Health. 2020 Oct 29;7(10):e17453. doi: 10.2196/17453. https://mental.jmir.org/2020/10/e17453/ v7i10e17453 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Cao J, Truong AL, Banu S, Shah AA, Sabharwal A, Moukaddam N. Tracking and predicting depressive symptoms of adolescents using smartphone-based self-reports, parental evaluations, and passive phone sensor data: development and usability study. JMIR Ment Health. 2020 Jan 24;7(1):e14045. doi: 10.2196/14045. https://mental.jmir.org/2020/1/e14045/ v7i1e14045 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Jongs N, Jagesar R, van Haren NE, Penninx BW, Reus L, Visser PJ, van der Wee NJ, Koning IM, Arango C, Sommer IE, Eijkemans MJ, Vorstman JA, Kas MJ. A framework for assessing neuropsychiatric phenotypes by using smartphone-based location data. Transl Psychiatry. 2020 Jul 01;10(1):211. doi: 10.1038/s41398-020-00893-4. doi: 10.1038/s41398-020-00893-4.10.1038/s41398-020-00893-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Christensen MA, Bettencourt L, Kaye L, Moturu ST, Nguyen KT, Olgin JE, Pletcher MJ, Marcus GM. Direct measurements of smartphone screen-time: relationships with demographics and sleep. PLoS One. 2016 Nov 9;11(11):e0165331. doi: 10.1371/journal.pone.0165331. http://dx.plos.org/10.1371/journal.pone.0165331 .PONE-D-16-25126 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Sharifian N, Kraal AZ, Zaheed AB, Sol K, Morris EP, Zahodne LB. Measurement invariance of social media use in younger and older adults and links to socioemotional health. Innov Aging. 2021 Mar 11;5(2):igab009. doi: 10.1093/geroni/igab009. http://europepmc.org/abstract/MED/33928196 .igab009 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Merikangas KR. Vulnerability factors for anxiety disorders in children and adolescents. Child Adolesc Psychiatr Clin N Am. 2005 Oct;14(4):649–79, vii. doi: 10.1016/j.chc.2005.06.005.S1056-4993(05)00063-5 [DOI] [PubMed] [Google Scholar]
  • 26.Rohani DA, Faurholt-Jepsen M, Kessing LV, Bardram JE. Correlations between objective behavioral features collected from mobile and wearable devices and depressive mood symptoms in patients with affective disorders: systematic review. JMIR Mhealth Uhealth. 2018 Aug 13;6(8):e165. doi: 10.2196/mhealth.9691. http://mhealth.jmir.org/2018/8/e165/ v6i8e165 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Glenn T, Monteith S. New measures of mental state and behavior based on data collected from sensors, smartphones, and the Internet. Curr Psychiatry Rep. 2014 Dec;16(12):523. doi: 10.1007/s11920-014-0523-3. [DOI] [PubMed] [Google Scholar]
  • 28.Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009 Apr;42(2):377–81. doi: 10.1016/j.jbi.2008.08.010. http://linkinghub.elsevier.com/retrieve/pii/S1532-0464(08)00122-6 .S1532-0464(08)00122-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Birmaher B, Khetarpal S, Brent D, Cully M, Balach L, Kaufman J, Neer SM. The Screen for Child Anxiety Related Emotional Disorders (SCARED): scale construction and psychometric characteristics. J Am Acad Child Adolesc Psychiatry. 1997 Apr;36(4):545–53. doi: 10.1097/00004583-199704000-00018.S0890-8567(09)62526-0 [DOI] [PubMed] [Google Scholar]
  • 30.Monga S, Birmaher B, Chiappetta L, Brent D, Kaufman J, Bridge J, Cully M. Screen for Child Anxiety-Related Emotional Disorders (SCARED): convergent and divergent validity. Depress Anxiety. 2000;12(2):85–91. doi: 10.1002/1520-6394(2000)12:2<85::AID-DA4>3.0.CO;2-2. [DOI] [PubMed] [Google Scholar]
  • 31.Faulstich ME, Carey MP, Ruggiero L, Enyart P, Gresham F. Assessment of depression in childhood and adolescence: an evaluation of the Center for Epidemiological Studies Depression Scale for Children (CES-DC) Am J Psychiatry. 1986 Aug;143(8):1024–7. doi: 10.1176/ajp.143.8.1024. [DOI] [PubMed] [Google Scholar]
  • 32.Garrison CZ, Addy CL, Jackson KL, McKeown RE, Waller JL. The CES-D as a screen for depression and other psychiatric disorders in adolescents. J Am Acad Child Adolesc Psychiatry. 1991 Jul;30(4):636–41. doi: 10.1097/00004583-199107000-00017.S0890-8567(09)64332-X [DOI] [PubMed] [Google Scholar]
  • 33.Aebi M, Metzke CW, Steinhausen H. Prediction of major affective disorders in adolescents by self-report measures. J Affect Disord. 2009 May;115(1-2):140–9. doi: 10.1016/j.jad.2008.09.017.S0165-0327(08)00390-X [DOI] [PubMed] [Google Scholar]
  • 34.Dierker LC, Albano AM, Clarke GN, Heimberg RG, Kendall PC, Merikangas KR, Lewinsohn PM, Offord DR, Kessler R, Kupfer DJ. Screening for anxiety and depression in early adolescence. J Am Acad Child Adolesc Psychiatry. 2001 Aug;40(8):929–36. doi: 10.1097/00004583-200108000-00015.S0890-8567(09)60341-5 [DOI] [PubMed] [Google Scholar]
  • 35.Green JG, DeYoung G, Wogan ME, Wolf EJ, Lane KL, Adler LA. Evidence for the reliability and preliminary validity of the Adult ADHD Self-Report Scale v1.1 (ASRS v1.1) Screener in an adolescent community sample. Int J Methods Psychiatr Res. 2019 Mar;28(1):e1751. doi: 10.1002/mpr.1751. http://europepmc.org/abstract/MED/30407687 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Paluska SA, Schwenk TL. Physical activity and mental health: current concepts. Sports Med. 2000 Mar;29(3):167–80. doi: 10.2165/00007256-200029030-00003. [DOI] [PubMed] [Google Scholar]
  • 37.Gomes da Silva S, Arida RM. Physical activity and brain development. Expert Rev Neurother. 2015;15(9):1041–51. doi: 10.1586/14737175.2015.1077115. [DOI] [PubMed] [Google Scholar]
  • 38.Lederbogen F, Kirsch P, Haddad L, Streit F, Tost H, Schuch P, Wüst S, Pruessner JC, Rietschel M, Deuschle M, Meyer-Lindenberg A. City living and urban upbringing affect neural social stress processing in humans. Nature. 2011 Jun 22;474(7352):498–501. doi: 10.1038/nature10190.nature10190 [DOI] [PubMed] [Google Scholar]
  • 39.Tost H, Reichert M, Braun U, Reinhard I, Peters R, Lautenbach S, Hoell A, Schwarz E, Ebner-Priemer U, Zipf A, Meyer-Lindenberg A. Neural correlates of individual differences in affective benefit of real-life urban green space exposure. Nat Neurosci. 2019 Sep;22(9):1389–93. doi: 10.1038/s41593-019-0451-y.10.1038/s41593-019-0451-y [DOI] [PubMed] [Google Scholar]
  • 40.Tarokh L, Saletin JM, Carskadon MA. Sleep in adolescence: physiology, cognition and mental health. Neurosci Biobehav Rev. 2016 Nov;70:182–88. doi: 10.1016/j.neubiorev.2016.08.008. http://europepmc.org/abstract/MED/27531236 .S0149-7634(16)30266-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.Walker MP, Harvey AG. Obligate symbiosis: sleep and affect. Sleep Med Rev. 2010 Aug;14(4):215–7. doi: 10.1016/j.smrv.2010.02.003.S1087-0792(10)00024-9 [DOI] [PubMed] [Google Scholar]
  • 42.Wehr TA. Sleep-loss as a possible mediator of diverse causes of mania. Br J Psychiatry. 1991 Oct;159:576–8. doi: 10.1192/bjp.159.4.576.S0007125000030531 [DOI] [PubMed] [Google Scholar]
  • 43.Cheng P, Kalmbach DA, Tallent G, Joseph CL, Espie CA, Drake CL. Depression prevention via digital cognitive behavioral therapy for insomnia: a randomized controlled trial. Sleep. 2019 Oct 09;42(10):zsz150. doi: 10.1093/sleep/zsz150.5571183 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44.Lemola S, Perkinson-Gloor N, Brand S, Dewald-Kaufmann JF, Grob A. Adolescents' electronic media use at night, sleep disturbance, and depressive symptoms in the smartphone age. J Youth Adolesc. 2015 Feb;44(2):405–18. doi: 10.1007/s10964-014-0176-x. [DOI] [PubMed] [Google Scholar]
  • 45.Woods HC, Scott H. #Sleepyteens: social media use in adolescence is associated with poor sleep quality, anxiety, depression and low self-esteem. J Adolesc. 2016 Aug;51:41–9. doi: 10.1016/j.adolescence.2016.05.008.S0140-1971(16)30034-3 [DOI] [PubMed] [Google Scholar]
  • 46.Godsell S, White J. Adolescent perceptions of sleep and influences on sleep behaviour: a qualitative study. J Adolesc. 2019 Jun;73:18–25. doi: 10.1016/j.adolescence.2019.03.010.S0140-1971(19)30055-7 [DOI] [PubMed] [Google Scholar]
  • 47.Touitou Y, Touitou D, Reinberg A. Disruption of adolescents' circadian clock: the vicious circle of media use, exposure to light at night, sleep loss and risk behaviors. J Physiol Paris. 2016 Nov;110(4 Pt B):467–79. doi: 10.1016/j.jphysparis.2017.05.001.S0928-4257(17)30034-7 [DOI] [PubMed] [Google Scholar]
  • 48.Eichstaedt JC, Smith RJ, Merchant RM, Ungar LH, Crutchley P, Preoţiuc-Pietro D, Asch DA, Schwartz HA. Facebook language predicts depression in medical records. Proc Natl Acad Sci U S A. 2018 Oct 30;115(44):11203–8. doi: 10.1073/pnas.1802331115. http://www.pnas.org/cgi/pmidlookup?view=long&pmid=30322910 .1802331115 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49.Noel VA, Acquilano SC, Carpenter-Song E, Drake RE. Use of mobile and computer devices to support recovery in people with serious mental illness: survey study. JMIR Ment Health. 2019 Feb 20;6(2):e12255. doi: 10.2196/12255. https://mental.jmir.org/2019/2/e12255/ v6i2e12255 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 50.Low L-S, Maddage NC, Lech M, Sheeber LB, Allen NB. Detection of clinical depression in adolescents' speech during family interactions. IEEE Trans Biomed Eng. 2011 Mar;58(3):574–86. doi: 10.1109/TBME.2010.2091640. http://europepmc.org/abstract/MED/21075715 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 51.Jacobson NC, Summers B, Wilhelm S. Digital biomarkers of social anxiety severity: digital phenotyping using passive smartphone sensors. J Med Internet Res. 2020 May 29;22(5):e16875. doi: 10.2196/16875. https://www.jmir.org/2020/5/e16875/ v22i5e16875 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 52.Sun S, Folarin AA, Ranjan Y, Rashid Z, Conde P, Stewart C, Cummins N, Matcham F, Dalla Costa G, Simblett S, Leocani L, Lamers F, Sørensen PS, Buron M, Zabalza A, Guerrero Pérez AI, Penninx BW, Siddi S, Haro JM, Myin-Germeys I, Rintala A, Wykes T, Narayan VA, Comi G, Hotopf M, Dobson RJ, RADAR-CNS Consortium Using smartphones and wearable devices to monitor behavioral changes during COVID-19. J Med Internet Res. 2020 Sep 25;22(9):e19992. doi: 10.2196/19992. https://www.jmir.org/2020/9/e19992/ v22i9e19992 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 53.Huckins JF, daSilva AW, Wang W, Hedlund E, Rogers C, Nepal SK, Wu J, Obuchi M, Murphy EI, Meyer ML, Wagner DD, Holtzheimer PE, Campbell AT. Mental health and behavior of college students during the early phases of the COVID-19 pandemic: longitudinal smartphone and ecological momentary assessment study. J Med Internet Res. 2020 Jun 17;22(6):e20185. doi: 10.2196/20185. https://www.jmir.org/2020/6/e20185/ v22i6e20185 [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Multimedia Appendix 1

The PROSIT (Predicting Risk and Outcomes of Social Interactions) app.

mhealth_v9i10e20638_app1.docx (1.1MB, docx)

Articles from JMIR mHealth and uHealth are provided here courtesy of JMIR Publications Inc.

RESOURCES