Predicting Brain Functional Connectivity Using Mobile Sensing

Abstract

Brain circuit functioning and connectivity between specific regions allow us to learn, remember, recognize and think as humans. In this paper, we ask the question if mobile sensing from phones can predict brain functional connectivity. We study the brain resting-state functional connectivity (RSFC) between the ventromedial prefrontal cortex (vmPFC) and the amygdala, which has been shown by neuroscientists to be associated with mental illness such as anxiety and depression. We discuss initial results and insights from the NeuroSence study, an exploratory study of 105 first year college students using neuroimaging and mobile sensing across one semester. We observe correlations between several behavioral features from students’ mobile phones and connectivity between vmPFC and amygdala, including conversation duration (r=0.365, p<0.001), sleep onset time (r=0.299, p<0.001) and the number of phone unlocks (r=0.253, p=0.029). We use a support vector classifier and 10-fold cross validation and show that we can classify whether students have higher (i.e., stronger) or lower (i.e., weaker) vmPFC-amygdala RSFC purely based on mobile sensing data with an F1 score of 0.793. To the best of our knowledge, this is the first paper to report that resting-state brain functional connectivity can be predicted using passive sensing data from mobile phones.

1. INTRODUCTION

Neuroscience is the study of the nervous system. Within neuroscience, a topic of interest is how brain activity relates to behavior. To illustrate a few examples, researchers have found that activity in a brain region associated with how we process our social world is correlated with one’s self-reported number of friends [74], and social media use is related to the brain’s reward response to favorable self-gains [60]. Typically to assess a given behavior, researchers ask participants to self-report a variety of information (e.g., how many people they interacted with or use a scale to measure social media usage). While interesting and informative, it is unclear if this information is always accurate. In addition, self-report places a burden on people. Neuroscience is one of the fastest growing research areas with research ranging from the cellular to the person level, and the mobile sensing community is a natural partner to bring into the fold. From the examples above, it is clear how mobile sensing could provide unbiased accounts of the behaviors that participants would typically have to self-report. Moreover, within the neuroscience community, there is a growing interest in complementing inferential findings with predictive, validated models [102, 104]. Beyond a growing similarity in modeling approaches, these two communities have many research interests in common. One notable focus area is that of mental health. On the neuroimaging side, a great body of work has linked neural activity to mental health measures [61, 72, 94], and of course, using sensing data to better understand mental health is widely studied in the UbiComp community [42, 96, 101, 103]. The purpose of the current proof-of-concept study is to bring these two communities together and to demonstrate that mobile sensing can provide unbiased metrics that can be used to further our understanding of the brain.

As mentioned, mental health is an active area of research within both the neuroscience and UbiComp communities where mental illness impacts one in four people worldwide affecting mood, perception, thinking and behavior [7]. The most common types of mental illness are: depression, which is characterized by persistently depressed mood or loss of interest in activities, causing significant impairment in daily life; anxiety, which is characterized by feelings of worry or fear that are strong enough to interfere with one’s daily activities. College-aged students are more prone to experience their first mental health episode than at any other time in their lives with their brains continuing to develop into their mid-twenties [89]. Adapting to academic pressures, new environments and living conditions, social pressures, erratic sleep habits and other worries and stressors present challenges even for the most healthy and resilient individuals. Anxiety and depression are the top reasons that college students seek counseling: 47% of students at the counseling center presented concern with anxiety and 39% with depression [77].

Neuroscientists believe that abnormalities in how particular brain circuits function are associated with mental illness – that is, mental illness relates to brain functioning. Measuring brain activity is therefore a natural approach to understanding the mind and mental health. With the advent of brain imaging techniques such as functional Magnetic Resonance Imaging (fMRI), researchers are able to measure brain activity and functional connectivity in increasing detail. Using neuroimaging, brain functions associated with various mental illnesses have been widely studied [22, 48, 50, 63, 93]. Neuroscientists have found that mental illness is associated with brain circuit functioning and connectivity between specific regions of the brain resulting in abnormal mood, perception and behavior. They have used resting-state functional connectivity (RSFC) as one method to explore the network structure of the brain and its association with mental illness through temporal synchronization. Resting-state captures the synchrony between brain regions when no explicit task is conducted by the person being scanned. In parallel, computer scientists working in mobile sensing have found correlates and predictors of individual differences associated with sensing data from phones and wearables. Mobile sensing, ecological momentary assessment (EMA) [87] and machine learning allow researchers to infer complex human behavior (e.g., physical activity, sleep state) and context (e.g., social interaction, places visited) from passive sensing data from phones and wearables. There is also growing evidence that passive sensing data can predict a range of behavioral and physical symptoms associated individual differences and a number of mental illnesses including anxiety [14, 15, 42, 80], depression [16, 59, 81, 96, 99], bipolar disorder [2, 33] and schizophrenia [10, 95, 98].

Recently, researchers published the first findings from a study on brain imaging and mobile sensing in the neuroscience literature [44]. These findings inspire the work presented in this paper. The authors [44] find correlations between smartphone usage from a group of college students and their fMRI scans. This prior work is similar to ours in terms of combining fMRI data with mobile sensing. In this paper, we aim to not only correlate which mobile phone sensing features contribute to the functional connectivity between brain regions but to go one important step further and demonstrate for the first time that mobile behavioral features inferred from students’ phone data can predict the level of connectivity between brain regions; that is, we significantly go beyond the correlation findings in [44] to predict the functional connectivity using mobile sensing data from mobile phones. More specifically, we ask if mobile sensing from phones can predict brain functional connectivity associated with mental health of college students. We discuss initial results and insights from the NeuroSence Study, an exploratory study of 105 first year college students using neuroimaging, mobile sensing and ecological momentary assessment across one semester. Specifically, we study the brain resting-state functional connectivity (RSFC) between the ventromedial prefrontal cortex (vmPFC) and the amygdala, which is shown to be associated with anxiety [51], depression [20], social-cognitive function [106], sleep [24] and adverse childhood experience [37, 41]. Figure 1 shows the locations of vmPFC and amygdala in the brain. The vmPFC is reported to be related to decision-making [9, 38], reward evaluation [78], morality [32] and emotion regulation [23, 36]. Whereas, the amygdala is a more evolutionarily conserved part of the brain, which responds to fear [55, 67], threat [26, 34], facial expressions (e.g., fearful faces) [64] and emotional processing [4, 55]. Researchers have shown that the vmPFC and amygdala play a particularly important role in human anxiety [21, 29, 31, 50–52]. It is widely reported in studies on college campuses in the USA [96] that students experience a wide variety of stressors associated with many issues, such as, assignments, tests, exams and social issues. In addition, there is an increasing rate of anxiety and depression across college campuses [57, 77]. For this reason, we focus on the known association between mental wellbeing and the vmPFC and amygdala functional connectivity. The vmPFC inhibits the amygdala’s reaction to fear or threats that makes people anxious. Higher functional connectivity defined by the synchronization of neural activity between the vmPFC and amygdala regions is inversely correlated with anxiety. In other words, stronger connectivity relates to lower anxiety and vice versa. We use this as an important neurological insight in our study. The goal of the NeuroSence study is three fold.

Fig. 1.

Location of ventromedial prefrontal cortex (red) and right amygdala (green). We utilize mobile sensing data to predict the brain connectivity between these regions, as the functional connectivity between vmPFC and amygdala is known to be associated with various aspects of mental health. The Montreal Neurological Institute coordinates are (8, 36, −18) and (20, −3, −15) for the right vmPFC and amygdala, respectively.

• First, to investigate if human behavior inferred from mobile phones is associated with the RSFC between vmPFC and amygdala, which is known to relate to mental health (e.g., anxiety, depression).

• Second, to study various machine learning models to determine if we can coarsely predict the brain activity between the vmPFC and amygdala regions (i.e., higher connectivity relates to lower anxiety and lower connectivity relates to higher anxiety) using continuous mobile phone sensing features as a pragmatic starting point for this exploratory study.

• Third, to stimulate discussion on this new topic in the UbiComp and neuroscience communities, potentially opening the way for new research at the intersection of neuroimaging and mobile sensing.

We use a smartphone application to continuously collect students’ sensing and behavioral data from their phones over one semester as predictive variables for the study. The passive sensing app records a variety of data from sensors on the students’ phone, subsequently transferring them to a secure server for offline modeling and analysis. All students who consented to participate in our study took part in an fMRI session (i.e., scanning) and installed the NeuroSence data collection application on their personal phones at the start of the semester. Our analysis consists of two parts: an exploratory correlation analysis and performance evaluation of machine learning models. The exploratory correlation analysis allows us to identify sensing features that show a relationship with vmPFC-amygdala RSFC. In the performance evaluation part of analysis, a predictive model is trained using 8 different machine learning algorithms in order to investigate predictability. Furthermore, we determine which sensing features are useful in order to predict vmPFC-amygdala RSFC using a feature selection technique. Research indicates [20, 41, 50] that people with mental health issues (e.g., anxiety) have lower vmPFC-amygdala RSFC, as discussed above. To restate this finding: people with higher anxiety typically have lower resting-state functional connectivity between vmPFC-amygdala regions [50]. As a part of our preliminary approach to study the relationship between mobile sensing data and functional connectivity between vmPFC-amygdala, we split participants into two groups for analysis based on RSFC: a “higher” and a “lower” functional connectivity group. We take a simple approach in our exploratory analysis by treating the problem as a classification task that predicts whether a student belongs to the higher or lower RSFC group.

The NeuroSence study makes the following contributions.

• To the best of our knowledge, we are the first group to use behavioral features from mobile sensing to study the brain functioning of 105 first year college students over one semester, specifically, we propose to use mobile sensing to predict functional connectivity between brain regions.

• We discover a set of mobile sensing features that relate to vmPFC-amygdala RSFC, which is known by neuroscientists to be related to mental health (e.g., negative correlation with anxiety). Specifically, we find behavioral features from students’ mobile phones correlate with functional connectivity between vmPFC and amygdala brain regions shown in Figure 1, including the conversation duration a student is around (r=0.365, p<0.001), their sleep onset time (r=0.299, p<0.001) and the number of phone unlocks (r=0.299, p=0.029) they initiate.

• We train 8 different machine learning algorithms to predict whether a student belongs to the higher or lower vmPFC-amygdala RSFC group. The higher connectivity group relates to lower anxiety and vice versa. After applying a 10-fold nested cross validation with hyperparameter tuning, the support vector classifier achieves an F1 score of 0.793.

While our study is exploratory and results preliminary, and our findings applicable to a first year college student cohort at a 4 year university, we hope that the NeuroSence study stimulates discussion about this new topic at the interface between brain imaging and mobile computing.

2. BACKGROUND ON BRAIN IMAGING

In this section, we introduce the neuroscience basics necessary for researchers in the UbiComp community to best understand our work. The human brain is roughly 1.4 kg of tissue containing roughly 86 billion neurons to perform the complex processing tasks that humans perform every day [91]. Interaction among an enormous number of neurons in the brain allows us to learn, remember, recognize and think as humans. Neuroscientists have found some localization of brain functions; the role of specific brain regions is well studied. For instance, a region called the visual cortex plays a role in processing the information from vision. Similar to a node in a neural network that is discussed in the context of computer science, a neuron transmits information to another neuron. Communication between specific parts of the brain have been observed to be related to individual differences in behaviors or mental states.

Functional MRI (fMRI) is used to measure changes in blood flow related to brain activity [66]. Temporal changes in blood oxygenation in different regions of the brain closely mirror brain activity. By measuring the change in blood oxygenation after neural activity, the brain regions that are actively working during a particular period of time can be determined. In short, fMRI captures the dynamic changes in blood flow as a proxy for neuronal activity. Researchers frequently use fMRI to study cognitive and affective processes, both in healthy subjects and subjects with particular conditions, such as mental disorders [48, 50, 63].

A more recent trend is to take a network approach to studying the brain. In the NeuroSence study, we used such an approach; namely, resting-state functional connectivity (RSFC), which observes the temporal relationship between distinct brain regions over time when no explicit task is conducted by the person being scanned. The general idea is that these networks have developed over time and depict a lasting history of activating and deactivating together. RSFC is often done using Pearson’s correlation between the fMRI signal of different regions. While general brain organization measured with RSFC is similar across thousands of different individuals [12] and stable and robust over the short-term (hourly) and long-term (monthly) [86], there are small individual differences in connectivity, which can be reliably observed across time. Researchers have found that RSFC is associated with a variety of individual differences, such as, personality [3], depression [47, 48], anxiety [50, 107], autism [18], dementia [6] and Alzheimer’s disease [93]. Finally, another advantage of resting brain scans is that individuals who have difficulty performing certain tasks typically can complete resting-state fMRI scans.

3. RELATED WORK

In this section, we first discuss the relevant research on mobile sensing and mental health then discuss selective work on neuroimaging; specifically, we focus on research associated with the vmPFC-amygdala RSFC and mental health. Finally, we discuss recent work at the intersection of mobile sensing and brain imaging that inspires us.

3.1. Mobile Sensing and Mental Health

There is a growing number of papers related to mobile mental health sensing covering a wide range of health issues, such as, mood [35, 46, 56], stress [13, 35, 83], depression [16, 59, 81, 96, 99], anxiety [14, 15, 42, 80] as well as serious mental health conditions, such as, bipolar disorder [2, 33] and schizophrenia [10, 95, 98]. For detailed surveys on sensing for mental health see [1, 8, 62]. In what follows, we highlight the most relevant work to our study.

Important early work on using embedded sensors built into a specialized mobile device for mental health assessment called the mobile sensing platform [19] is reported in [76]. The authors study eight elderly adults living in a continuing care retirement community and demonstrate that speech and conversation occurrences extracted from audio data and physical activity can infer mental and social wellbeing. N = 8 is small in this study, but this work predates the availability of smartphones which opened the way for larger scale studies without the need for the development and deployment of specialized sensing platforms. The StudentLife study [96] shows that sensing features from smartphones correlate with depression, perceived stress, flourishing and loneliness. The StudentLife study installs a passive continuous sensing application on 48 undergraduate students’ phones collecting data across a ten week term at Dartmouth College. The authors [96] discover that several sensing features are correlated with mental health. For example, the frequency of conversation data during the day, as measured with the microphone on the phone, had a significant negative correlation with PHQ-9 depression scores. In other words, students who engage in more conversations report having fewer depressive symptoms. The researchers also demonstrate that the perceived stress scale negatively correlates with conversational duration (r=−0.357, p=0.026) and frequency (r=−0.394, p=0.013). The MONARCA project also reports on an early use of smartphones for mental health sensing [39, 68, 69, 79], specifically, bipolar disorder. They report [69] a number of correlations between the activity levels of subjects over different periods of the day and psychiatric evaluation scores associated with the mania-depression spectrum. Other authors present important insights, including the automatic inference of circadian stability as a measure to support effective bipolar management [2].

Mobility features from mobile phones and their associated semantics relate to anxiety and depression. A study at Northwestern University [82] shows that phone usage and mobility data collected from smartphones correlate with depressive symptoms measured by PHQ-9 [53]. Canzian et al. [16] report an important association between PHQ-9 and a wide variety of mobility features. They find that the maximum distance moved between two places greatly correlates with the severity of depression. In [43], the authors report the relationship between location information and social anxiety for college students. The GPS location data is classified into five categories: “home”, “work”, “religious”, “food & leisure” and “transportation” using Foursquare [25]. The researchers [43] computed two features: the time spent at each location category and transitions between locations (e.g., from “work” to “home”). Geolocation data collected over ten days correlated with scores on the social interaction anxiety scale [58]. The results suggest that more time spent at “religious” places relates to lower anxiety scores. Similarly, the transition frequency from “home” to “religious” places (r=−0.521, p=0.025) and from “work” to “religious” places (r=−0.656, p=0.002) is negatively correlated with social anxiety scores. The authors of [14] focus on social anxiety and mobility. A two-week study involving 228 undergraduate students indicate that students with high levels of anxiety tend to avoid public places. In addition, students with high anxiety engage less in leisure activities in the evenings compared to less anxious students.

In a follow up study to StudentLife the same group report on results from predicting the dynamics of depression across two terms for college students [98]. The authors detail depression prediction results against a weekly administered PHQ-4 with 69.1% precision and 81.5% recall. In [92], the authors study 126 participants using smartphone sensing data with the goal of classifying depressive symptom using PHQ-9. They report a classification accuracy of 60.1%. Another study [101] successfully classifies whether a user is depressed using WiFi meta-data with an F1 score of 0.85. In [98], researchers use mobile sensing data to predict the Brief Psychiatric Rating Scale (BPRS), a survey used to evaluate symptom severity for schizophrenia patients. Their regression model successfully predicts the score ±1.59 error on average for a 7-item BPRS. The study in [13] classifies whether a user is stressed with 72.28% accuracy (N=117) using the Random Forest algorithm. In [27], researchers also report on the use of smartphones to predict changes in anxiety as measured by the State-Trait Anxiety Inventory [90] with an F1 score of 0.742. In summary, while most of the reported studies are small scale and focus on different aspects of mental wellbeing, there seems to be an emerging sense that data collected passively from phones can be used to model mood.

3.2. Neuroimaging and Mental Health

There is a large amount of work on neuroimaging research that relates to mental health. With regards to vmPFC-amygdala functional connectivity, as shown in Figure 1, there is some work on its relationship to depression [20] but more literature is associated with its connection to anxiety [21, 29, 31, 50–52]. Anxiety is characterized by a failure to inhibit inappropriate fear [70]. Researchers are interested in understanding the mechanism of regulating the amygdala, the region that reacts to fear, threat, or uncertainty because aberrant activation of the amygdala is associated with anxiety. A number of researchers find that the vmPFC, a region involved in self-control, decision-making and risk evaluation, plays a critical role in regulating the amygdala [65]. Other findings report that the prefrontal cortex inhibits the amygdala via a top-down regulation mechanism [40, 49]. In [28], the authors report there is an inverse correlation between fear extinction recall and vmPFC-amygdala functional connectivity. Individuals with higher (i.e., stronger) functional connectivity between these regions are better at eliminating fear and reducing anxiety. Kim et al. [50] conduct a study involving 28 students and find that the RSFC between the right amygdala and vmPFC regions is negatively correlated with state anxiety. Connolly et al. report the vmPFC-amygdala RSFC is associated with major depression [20]. Depressed adolescents (N=48) have lower vmPFC-amygdala RSFC compared to healthy controls (N=53). The authors report the impairment of these regions supporting emotional regulation characterizes adolescent depression. In [106], researchers find that the vmPFC-amygdala RSFC is associated with socio-cognitive functioning. The study reports that RSFC among healthy individuals positively correlates with the performance of the theory of mind, which is the ability to understand others’ intention, emotion and beliefs [75]. Prior research links vmPFC-amygdala functional connectivity with sleep, suggesting that sleep deprivation (a common condition reported by students during the college years) is associated with lower vmPFC-amygdala functional connectivity [24, 105]. We build on these prior results and insights associated with the vmPFC-amygdala RSFC and anxiety, depression and sleep in our study.

Finally, one group of researchers [44] recently published the first findings from a study on mobile sensing data and brain imaging data. This early work inspires our study. In summary, Huckins et al. [44] present connections between mobile sensing technology and fMRI scans. The researchers scan 257 students at Dartmouth College and compute RSFC for the subgenual cingulate cortex (sgCC), a brain region related to depression. They generate a 3D map of the brain representing the functional connectivity between subgenual cingulate cortex and all other parts of the brain. Furthermore, they conduct correlation analysis between each functional connectivity value and smartphone usage (specifically unlock duration) in order to find which brain region is associated with depression and unlock duration. This prior work is similar to ours in terms of combining RSFC with mobile sensing. However, we aim to not only investigate which mobile phone sensing features contribute to vmPFC-amygdala RSFC but also to demonstrate for the first time that these mobile behavioral features inferred from students’ phone data can predict the level of connectivity between brain regions; that is, we go beyond finding correlates to predicting the RSFC between the vmPFC-amygdala regions using mobile sensing data.

4. NEUROSENCE STUDY DESIGN

In what follows, we describe the NeuroSence study design, ground truth (i.e., fMRI data), our mobile sensing system and feature generation.

4.1. Study Design

We recruit 105 undergraduate students (75 female, 30 male) at the start of the fall semester as they entered their first year at college with the goal of investigating mental health using mobile sensing and brain imaging. The mean age of participants at the beginning of the study is 18.2 years (standard deviation of 0.63). Among the participants, the majority are Caucasian (56%), 21% Asian, 16% multiracial, 3% African, 1% Hispanic and the rest 2% answered that they belong to other or unknown ethnicity group. After the subjects consent to participate in the study, two types of data are collected: brain connectivity using fMRI and mobile sensing data captured using students’ own Apple or Android phones. We use a Siemens MAGNETOM Prisma 3-Tesla scanner [88] for brain imaging of participants during the start of the fall semester when all subjects are scanned. Students are scanned at the start of the term and the NeuroSence app is installed on their phones to collect passive sensing data across one academic term (79 days). The NeuroSence study is approved by the study institution’s Institutional Review Board and the students receive monetary compensation for participating in the study.

4.2. Ground Truth: fMRI data

We use the fMRI data collected from scanning our subjects as ground-truth for correlation and predictive analysis using mobile sensing features, which we have discussed in detail in Section 5. During scanning, participants view a white fixation cross on a black background. The scanner collects both anatomical images of the brain, and the fMRI signal that represent the proxy of neural activity. Figure 2 shows anatomical brain images centered at the positions of the vmPFC and right amygdala of a co-author. With our scanner, we can collect the fMRI signal with a 2.4 × 2.4 mm in-plane resolution. The complete functional brain image is collected roughly every 1.2 seconds, acquiring a total of 605 volumes for 12 minutes of data acquisition. After scanning, preprocessing is performed to reduce artifacts (e.g., correcting head motion, removing physiological noise), increasing the signal-to-noise ratio of acquired fMRI data. For more details regarding the data collection of neuroimaging, please see the work by Huckins et al. [44]. Functional connectivity between the vmPFC and amygdala is calculated using the Pearson’s correlation (r) of the fMRI signal of these two regions over time. Only subjects with 5 minutes or more of low-motion data are included in the analysis to minimize the influence of motion. The r values are transformed using Fisher’s Z to improve the normality of the distribution.

Fig. 2.

An anatomical brain image of a co-author. The crosshairs are centered over the right vmPFC and amygdala.

4.3. Mobile Sensing System

We collect sensing data across an academic term and develop a dedicated smartphone application to collect behavioral data of users’ daily activities (Table 1). The NeuroSence application works efficiently on both iOS and Android phones, and acquires sensing data associated with physical activity and location using the API provided by each OS platform. The NeuroSence sensing platform is a tailored version of a mobile phone sensing platform that has been validated by a number of published studies. We use location data to estimate a subject’s location on campus, which is located in a college town where all first years live on a compact campus. Thus, we compute semantic location data by creating a dictionary table that links campus buildings to a number of different location categories; for example, the libraries are categorized as “study area” locations while cafeterias are categorized as a “food” location. We also infer which dormitory each student lives in, in order to compute their “home” location. The “greek” location category denotes Greek Letter Organization Houses (i.e., fraternities and sororities). Furthermore, our application detects sleep [17] and speech/conversation data [54, 76] by incorporating pre-trained classifiers using accelerometer and audio data from a microphone, respectively. The application does not record raw audio data in order to protect users’ privacy, but stores audio information when speech or conversation is detected, which is based on a two stage classifier: first a voice activity detector determines if speech is present then the next stage, a conversation classifier, determines if a conversation started and records its duration. We can not be completely sure if the subject is actively involved in the conversation, the inference of conversation is associated with the subject being around conversation. We use this as a proxy for social engagement or isolation.

Table 1.

Features generated from mobile sensing data

Sensing Type	Features
Activity	Duration of still, walking and running
Location	Time spent at home (dorm), other dorms, study, food, social, greek, religious and workout areas Number of places visited Distance travelled
Phone Usage	Unlock duration Number of unlocks
Microphone	Audio amplitude Conversation duration Number of conversation Ratio of voice (speech)
Sleep	Sleep start time Sleep end time Sleep duration

4.4. Mobile Sensing Features

In our study, we design behavioral features from mobile sensing data inspired by the previous research on mobile sensing [96, 97]. Table 1 shows the list of features computed in the NeuroSence study. As a basic strategy, we compute the average value (e.g., how many hours a user uses the phone) and the count (e.g., how many times a user unlocks the phone) for each sensor. In order to increase the interpretability of features and to better understand students’ behavior, we divide time across the day into three epochs similar to [97]. Epoch 0 represents the entire 24 hour day. The epochs 1, 2 and 3 denote, day (9AM–6PM), evening (6PM-0AM) and night (0AM-9AM), respectively. The separation of time into these epochs allows us to understand the user’s context at a finer granularity. For example, we can compute the number of steps, exercise duration or how much they are social (inferred from microphone data) within each epoch period. Furthermore, we compute the standard deviations to estimate the variability of a student’s behavior [100]. Typically, student behavior might be dependent on the day of the week (e.g., days they have class scheduled, tests, assignments due). We calculate the standard deviation for each day of the week from Monday, Tuesday, etc., and average those five values as variability score. This measure attempts to capture the variability of their week and ultimately across the semester. Similarly, we compute the regularity index for each sensing feature using negation of approximate entropy [73]. The regularity feature differs from the variability since it considers the unpredictability of changes over time-series data.

5. ANALYSIS

A key goal of our exploratory study is to investigate the relationship between mobile sensing features and vmPFC-amygdala RSFC and predict brain regions connectivity associated with various aspects of mental health (e.g., anxiety) from sensing data, as discussed in Section 3.2. In what follows, we describe our analysis and then in the next section our results. For data quality reasons, in particular, to prevent distortion of data caused by missing data we set some criteria based on prior data quality insights associated with mobile phone sensing [100]. First, we remove subjects from our data set who have either brain images or mobile sensing data missing. Specifically, in terms of mobile sensing data we exclude data with less than 18 hours [100] of data per day and less than 14 days of data during the term.

5.1. Correlation Analysis

We compute Spearman’s correlation to investigate the relationship between each mobile sensing feature and vmPFC-amygdala RSFC. Although the correlation coefficient does not explain the cause-effect relationship, we can examine the monotonic relationship in the observed data. In other words, we can recognize whether or not one variable increases or decreases as the other variable increases or decreases. Because the number of comparisons is high (i.e., > 100), the result may have multiple testing problems: including more false-positive errors. To avoid this issue, we use the Benjamini-Hochberg procedure [11], which regulates the false discovery rate (FDR) in our exploratory analysis. We report correlation along with FDR-adjusted p-values and standard p-values.

5.2. Prediction Analysis

A key aim of our study is to evaluate the performance of various machine learning models, trained to predict vmPFC-Amygdala RSFC from mobile sensing features. We formulate this research problem as a simple binary classification problem as a starting point for analysis and perform a median split on the RSFC data into higher or lower RSFC groups; that is, those subjects with higher and lower resting state functional connectivity between the vmPFC-Amygdala regions. We then train our model to predict if a person is in the higher or lower RSFC group. All sensing features are standardized before training, transforming the data distribution to have a mean of 0 and a standard deviation of 1. We select eight distinct algorithms to evaluate machine learning models and perform a grid search to find the best hyper-parameters for each model. Table 2 shows machine learning algorithms evaluated and hyper-parameters tuned. We use a nested cross validation (CV) scheme to prevent over-fitting, to build a generalized model and to restrict hyper-parameters from being solely adapted to the training data (see Figure 3). The outer cross validation for evaluating the performance of the machine learning model is 10-Fold, and the inner CV for hyper-parameter tuning is 3-Fold. We consider the nested 10-Fold cross validation as being better than leave-one-out cross validation (LOOCV) for two reasons. First, 10-Fold CV is more conservative than LOOCV in terms of performance since the former uses less training data and more testing data. Second, using LOOCV might cause overfitting of hyper-parameters. In other words, we lose the benefit of using the nested CV because the inner fold would be using almost the same size as the entire data (i.e., N-1). Please note that a record in our dataset represents one user’s data. We do not include the same individual’s data on both training and testing set while doing record-wise cross validation. We select the F1 score as a metric that indicates the performance of the model during training.

Table 2.

Machine learning algorithms and hyper-parameters.

Algorithm	Hyper-parameter	Values
KNN	n_neighbors	1, 3, 5, 7, 9
Linear SVC	C	0.1, 1, 10
SVC (RBF kernel)	C gamma	0.1, 1, 10 0.01, 0.1, 1, 10
Gaussian Naive Bayes	* no hyper-parameter
Bernoulli Naive Bayes	* no hyper-parameter
Logistic Regression	C penalty	0.1, 1, 10 l1, l2
Random Forest	max_depth min_sample_split	3, 5, 7 3, 5
XGBoost	max_depth min_child_weight gamma	3, 5, 7 1, 3, 5 0.01, 0.1, 1, 10

Fig. 3.

Overview of nested cross validation. We set an inner 3-Fold CV to select the best combination of hyper-parameters for training a model in the outer fold. The outer 10-Fold CV is used to evaluate the generalizability of the model. The machine learning algorithm trains the classifier using 90% of data and predicts the rest 10% data for ten times, where each data becomes the validation data once.

Furthermore, we evaluate 20 models using different random seeds when training each model and average the score to prevent bias from the initial seed selection. Training a model with one random seed could lead to bias because the random value is used inside a specific algorithm, and is used when splitting the training and test data during cross validation. We apply the sequential forwarding selection [45] as the feature selection method. Sequential forwarding selection is a method to decrease the dimensionality of the set of features by selecting a subset of features that improves the predictive performance of a model. Using sequential forwarding selection, we create a set X containing all features and an empty set Y to store the selected features. For each iteration, the algorithm picks one element in the set X and temporarily adds it to set Y. We then compute the performance of the model (i.e., F1 score) by training with the features in set Y, calculating how much that feature improves the model performance. We remove the best feature from X and add it to Y once we finish checking all features, and go to the next iteration and repeat the process.

6. RESULTS

In this section, we describe our preliminary results from the NeuroSence study. First, we show the distribution of vmPFC-amygdala RSFC across the complete cohort, followed by our correlation results and finally we present the evaluation of the machine learning models to predict higher or lower RSFC from mobile sensing features. In terms of the data quality criteria (discussed in Section 4), of the 105 students consenting to be part of the NeuroSence study only 92 opted to be scanned at the start of the semester. In addition, only 75 of the 92 scanned subjects met the mobile sensing data quality criteria [100] of 18 hours of sensing data per day for a minimum of 14 days across the term. Although we applied the median split for RSFC before filtering the participants, the number of higher RSFC and lower RSFC subjects is well balanced (i.e., specifically, 38 students in the higher RSFC group and 37 in the lower group). Figure 4 is a histogram representing the distribution of vmPFC-amygdala RSFC of all subjects who are scanned. The average value of RSFC is 0.12 with the median of 0.14. The vertical line denotes the median in Figure 4. The distribution approximates a normal distribution as the null hypothesis is not rejected (p=0.515) in the Shapiro-Wilk test of normality [84].

Fig. 4.

Distribution of vmPFC-amygdala resting-state functional connectivity among participants.

6.1. Correlation: Mobile Sensing and Neuroimaging

The correlation between each feature generated from mobile sensing data and vmPFC-amygdala RSFC is shown in Table 3. The physical activity of subjects correlates with vmPFC-amygdala RSFC. Specifically, we observe four regularity features (i.e., running/walking during the entire day, running during the daytime and walking in the evening) correlates to vmPFC-amygdala RSFC. All results show the same direction: more regular physical activity behavior of subjects relates to a higher RSFC. Among the five sensor types shown in Table 1, the microphone demonstrates several significant correlations, as shown in Table 3. The result suggest that RSFC increases when subjects are around more frequent conversation and conversation of increasing duration; that is, our inference using the microphone infers that a subject is “around” conversation. This could be a social encounter or a class lecture, we do not know for sure. We consider the inference of conversation frequency (i.e., the number of conversations per day) and duration (i.e., the length of each conversation) as a proxy for social engagement or isolation. The dynamics of these conversational features change over the term as students engage in increased workload and exams, etc. Furthermore, the variability of conversation duration shows a significant correlation, as shown in Table 3. The StudentLife study found that social engagement made students feel more connected, less lonely and more resilient with better mental wellbeing, positive emotions and potentially better academic performance [96, 97]. Considering location features, the time spent at “social” places shows a positive correlation, as shown in Table 3. Similar to conversational features, our result support that students who spend more time at social areas, where conversations are more likely, have a higher RSFC. We do not observe a significant relationship with spending time at “religious” places (RSFC: r=0.045, p=0.703) as existing literature suggests [43, 80] (i.e., spending time at “religious” locations negatively correlates with anxiety). The regularity of time spent at other students’ dorms demonstrates the most significant correlation among location-based features, as shown in Table 3. This result suggests that the regular pattern of visiting the friends’ dorm rooms is associated with a higher RSFC. We find that correlation associated with smartphone usage relates to the unlock duration between time epoch of 0 AM - 9 AM. During the same epoch, we also observe that the number of unlocks positively correlates with RSFC.

Table 3.

Correlation between sensing data and vmPFC-amygdala resting-state functional connectivity.

Sensor	Feature	Spearman’s r
Activity	Regularity of running (24H)	0.231
	Regularity of running (9AM-6PM) **	0.362
	Regularity of walking (24H)	0.229
	Regularity of walking (6PM-0AM)	0.285
Microphone	Audio amplitude (9AM-6PM)	−0.251
	Conversation duration (24H) **	0.365
	Conversation duration (0–9AM) **	0.330
	Conversation duration (9AM-6PM) *	0.312
	Conversation duration (6PM-0AM) **	0.363
	Number of conversation (24H) **	0.360
	Number of conversation (0–9AM) *	0.314
	Number of conversation (9AM-6PM)	0.255
	Number of conversation (6PM-0AM) **	0.366
	Ratio of voice (24H) **	0.365
	Ratio of voice (0–9AM)*	0.293
	Ratio of voice (9AM-6PM)	0.246
	Ratio of voice (6PM-0AM) **	0.348
	Variability of conversation duration (24H) **	0.368
	Variability of conversation duration (0–9AM) *	0.318
	Variability of conversation duration (9AM-6PM) **	0.355
	Variability of conversation duration (6PM-0AM)	0.281
	Variability of number of conversation (24H) **	0.358
	Variability of number of conversation (0–9AM) **	0.339
	Variability of number of conversation (6PM-0AM)	0.243
	Variability of number of ratio of voice (24H) *	0.318
	Variability of number of ratio of voice (0–9AM)	0.278
	Regularity of number of conversation (24H)	−0.261
Location	Distance travelled (0–9AM)	−0.245
	Time spent at “social” location	0.236
	Regularity of number of places visited (24H)	0.278
	Regularity of number of places visited (6PM-0AM)	0.255
	Regularity of time spent at “other dorms” location *	0.321
Phone usage	Unlock duration (0–9AM)	0.276
	Number of unlock (0–9AM)	0.253
	Variability of number of unlock (24H)	0.260
Sleep	Sleep start time *	0.299

standard p < 0.05;

bold standard p < 0.01;

^*FDR-adjusted p < 0.1;

^**FDR-adjusted p < 0.05

Finally, sleep onset time shows the most significant and only correlation among sleep features, implying RSFC is higher for students that go to sleep later at night. On academically challenging college campuses, many students have poor sleep habits going to bed late into the night. Surprisingly, we do not find significant correlations for sleep duration (r=−0.161, p=0.168) and sleep variability features in our study. While there is an error of +/− 30 minutes in the sleep classifier used in [17], the trends remain accurate.

6.2. Classification: Higher and Lower RSFC

As a result of training the models with eight distinct algorithms, the support vector classifier (SVC) with radial basis function (RBF) kernel using 23 features achieves the highest F1 score of 0.793. Table 4 shows the F1 score for each algorithm. Although some research finds that gradient tree boosting based algorithms (i.e., random forest, XGBoost) improve the performance when compared to other machine learning algorithms, they are unsuitable for our dataset because of its relatively small sample size. Although random forest and XGBoost have more hyper-parameters to be tuned requiring more training time than others, the SVC using 23 features performed best when classifying students into higher (i.e., stronger) or lower RSFC (i.e., lower) connectivity groups, showing a relatively even precision-recall balance. The 23 features are listed in Table 5. Note that the accuracy is 0.791, precision is 0.798, recall is 0.788 and AUROC is 0.811. Figure 5 and Figure 6 show the confusion matrix and ROC curve, respectively. All values are an average of 20 models from different random seeds, as described in Section 5. During cross validation, 17 out of 20 models selected the pair of 1.0 for C and 0.1 for gamma as the best hyper-parameter. The C and gamma are hyper-parameters used to adjust the penalty of the error term and complexity of the model, respectively. These pair of hyper-parameters are close to the default value of SVC implementation in scikit-learn [71]. Regarding the variance of the 20 models of the best SVC, the standard deviation of the F1 score is 0.03 and the mean of the standard deviation of 10-Fold cross validation is 0.17.

Table 4.

F1 score among eight machine learning algorithms using feature selection. The support vector classifier with RBF kernel achieves the highest F1 score of 0.793.

Algorithm	F1 score
KNN	0.756
Linear SVC	0.644
SVC (RBF kernel)	0.793
Gaussian Naive Bayes	0.654
Bernoulli Naive Bayes	0.736
Logistic Regression	0.717
Random Forest	0.672
XGBoost	0.641

Table 5.

The 23 features selected by sequential forwarding selection for support vector classifier. K indicates an iteration number in feature selection.

K	Accumulated features	F1
1	Audio amplitude (24H)	0.657
2	Audio amplitude (6PM-0AM)	0.655
3	Variability of time spent at “religious” location	0.642
4	Distance travelled (24H)	0.668
5	Variability of activity “walking” duration (6PM-0AM)	0.680
6	Number of place visited (6PM-0AM)	0.710
7	Variability of activity “still” duration (6PM-0AM)	0.775
8	Variability of number of place visited (24H)	0.698
9	Variability of audio amplitude (24H)	0.718
10	Number of unlock (0–9AM)	0.719
11	Regularity of time spent at “religious” location	0.727
12	Variability of audio amplitude (0–9AM)	0.719
13	Regularity of activity “running” duration (0–9AM)	0.746
14	Activity “still” duration (6PM-0AM)	0.743
15	Time spent at “religious” location	0.747
16	Variability of conversation duration (24H)	0.762
17	Distance travelled (0–9AM)	0.789
18	Variability of distance travelled (24H)	0.777
19	Activity “walking” duration (24H)	0.772
20	Regularity of time spent at “greek” location	0.759
21	Conversation duration (0–9AM)	0.766
22	Variability of number of unlock (6PM-0AM)	0.764
23	Regularity of number of unlock (6PM-0AM)	0.793

Fig. 5.

Confusion matrix of support vector classifier using 23 features. F1 score is 0.793.

Fig. 6.

ROC curve of support vector classifier using 23 features.

7. DISCUSSION

In this section, we discuss some insights from our results. We also discuss the limitation of our work.

7.1. Correlated Mobile Sensing Features

When considering all the different categories of mobile sensing features, the conversational features from phones are the most correlated ones with vmPFC-amygdala functional connectivity. Our results indicate that vmPFC-amygdala RSFC increases (i.e., is stronger) when students are around a conversation or spend more time in social or working locations where they typically engage with other students (e.g., spend time at friends’ dorms). Our results are consistent with previous studies that indicate that people who have substantial social connection tend to have better psychological wellbeing [30]. Our results show that social features are correlated with functional connectivity; the more the social engagement, the higher the functional connectivity. As discussed in the related work section, a number of neuroscientists find that higher vmPFC-amygdala functional connectivity is correlated with lower depression and anxiety [20, 50]. Based on these findings, we can hypothesize that more social engagement and stronger functional connectivity (as found in our study) is likely to be related to better mental wellbeing of students.

Regarding phone usage, vmPFC-amygdala RSFC correlates with the number of phone unlocks and unlock duration between 0 AM and 9 AM, but not with the other time epochs. We hypothesize that phone usage with social apps (e.g., texting) might be an important behavior that positively correlates with RSFC. During the night, we presume that most students rely on their phones to interact with their friends instead of face to face interaction. To prove this, we would need to understand which specific apps users interact with on their phones. Some apps are clearly more socially potent than others. We would also need to better understand the context of the conversational interaction: sitting through an hour long lecture (which would likely be inferred as a single conversation of one hour by our sensing platform) is different from a one hour conversation with close friends. We do not have this contextual information in our study.

In terms of sleep, previous studies [24, 105] find that sleep deprivation is related to weaker vmPFC-amygdala RSFC. Our analysis indicates that vmPFC-amygdala RSFC is higher (i.e., stronger) when students go to sleep later. We do not have any further data on why students go to sleep later but the two most common reasons discussed in the StudentLife study [96] are academic demands such as assignment due dates and exams and social events such as hanging out with friends and parties. Without the specific reasons, we cannot fully interpret if late sleep is motivated by work of social life or both (e.g., students working together on joint assignments). We could interpret our result along a number of lines. By linking it to behavior at night before sleep; for example, students attending social events (e.g., party, drinking) at the night may result in positive mental well-being. In this study, we have used a phone based sleep algorithm [17] that is described as a “best effort” estimator for sleep onset and duration. In a future study, we would use wearables for sleep stage analysis to determine what students do before sleep – e.g., work, academics, sports, social. Having deeper sleep data and more contextual information about the behavior before sleep would lead to better insights. Also, four regularity features correlate with vmPFC-amygdala RSFC in our dataset. Students who regularly walk or run prone to have higher RSFC. It is widely accepted that frequent aerobic exercise (i.e., running) plays a vital role in improving mental wellbeing [85]. Finally, note that our results have limitations. Specifically, we find many pairs of variables having significant p-values; however, the correlation coefficient (r) is modest at best. In fact, the maximum r-value we obtain is 0.368. Therefore, we cannot claim that our correlations are strong [5].

7.2. Classification

Although our goal to predict neural activity from mobile sensing is challenging, we consider the model that we trained as robust. In our analysis, we make our best effort to prevent over-fitting or to bias training data. As a drawback of machine learning, we need to sacrifice some interpretability. The SVC with RBF kernel using 23 features has the best performance among all the models, but we cannot easily comprehend why the model selects those specific features. The linear SVC provided lower performance but would be more interpretable. Note that the sequential forward selection features differ from the rankings of feature importance because the sequential forward selection is accumulating a feature that improves a model on the stacked features (i.e., audio amplitude in the first place) over each iteration. Interestingly, the time spent at religious locations is selected for all types of feature calculation (i.e., mean, variability and regularity). We do not see a relationship with vmPFC-amygdala RSFC in the correlation analysis; however, visits to religious places have been shown to have an association with mental wellbeing (i.e., anxiety) [43, 80].

In terms of performance, SVC results in the highest F1 score (0.793) using 23 features, whereas using the 6 or 7 features results in an F1 score of 0.710 and 0.775, respectively. Considering our dataset size, a smaller number of features may be more robust and generalizable. Furthermore, the relationships between features in high-dimensional data can be hard to interpret. Figure 7 presents the boundary of SVC with RBF kernel using the top two features (F1 score is 0.655 in this case). Although we can perceive a boundary drawn by Gaussian kernel, visualizing higher dimension is difficult. Alternatively, if we can get good performance in particular algorithms that return feature importance (e.g., tree-based models), we can understand the importance of each feature as it relates to prediction. We assume that the performance of our machine learning model (i.e., SVC) is reliable. Our model exceeds the 50% baseline, which is relevant to random prediction, reasoning that mobile sensing contains a signal for predicting brain activity. The F1 score of 0.793 (0.791 accuracy) is a strong result as it achieves a similar accuracy of binary classification performance reported in mobile mental health sensing studies, as discussed in Section 3.1. Using machine learning enables us to predict the target RSFC with higher performance. We believe that our classifier performance could be improved by increasing the number of subjects in the data set. We consider we have good power but increasing the number would be the next step as well as studying the replicability of our results at higher scale.

Fig. 7.

Predictive boundary drawn by support vector classifier using top two features. Green and orange markers represent the higher and lower vmPFC-amygdala RSFC group, respectively.

7.3. Limitations

Although we succeeded in training a model to predict the functional connectivity using mobile sensing data, we recognize a number of limitations of our work. First, all the subjects that participated in our study are first-year undergraduate students at a US university, with the majority being female students. Therefore, our results cannot necessarily be widely generalized (e.g., age, gender, first generation, socio-economics, nationality). We do not screen for students with mental health conditions. It would be interesting to consider this community as part of our cohort or a subcohort in the future. In summary, future work will consider a more diverse set of subjects and incorporate demographic information to train a generalized machine learning model. Second, identifying the function at a specific brain localization is critical. In other words, we have to understand that a particular brain region is not playing a single role. Based on prior research in neuroscience, we assume that vmPFC-amygdala RSFC is associated with mental health - anxiety, in particular - in the current study. However, the physiological phenomenon or mechanism that forms anxiety has not been elucidated yet. Third, while we selected amygdala and vmPFC as brain regions of interest, because of their relationship to mental health, it is possible that connectivity between other pairs of brain regions may have higher or lower accuracy, depending on if the regions communicate with each other directly or through other brain regions, along with their specific excitatory/inhibitory relationships. Future work will better understand this issue by considering the whole-brain level, but many things must be considered before this can become a reality, including the optimal set of features to try to train models on and selecting a specific model so that there can be consistency between pairs of brain regions.

8. CONCLUSION

In this paper, we discussed NeuroSence, an exploratory study investigating the relationship between brain imaging and mobile sensing from phones, a new topic in the UbiComp community. Our study uses mobile sensing to predict brain connectivity between the vmPFC and amygdala regions of the brain, which has been shown in the neuroscience literature to be associated with various aspects of mental health. Mobile sensing, which allows us to capture the user’s behavior in-the-wild, has the potential to brings new insights to neuroscience, where today research studies typical rely on self-report surveys. We hope our paper is a first step toward showing how mobile sensing provides deep longitudinal human behavioral data that provides additional contextual information when analyzing discreet and costly brain imaged data – offering new insights into brain imaged subjects (e.g., sleep patterns, social engagement, physical activity levels) not possible without continuous sensing. In this paper, we assessed brain connectivity using resting-state fMRI and collected behavioral data using a continuous sensing smartphone app over one academic semester. We identified a set of behaviors that correlate with vmPFC-amygdala connectivity. For example, conversational sensing data shows significant correlation (r=0.365, p<0.001) with brain function connectivity of the college students in the study. We also evaluated the performance of machine learning models to predict if a student belongs to a higher or lower vmPFC-amygdala RSFC group using mobile sensing features. The support vector classifier achieved an F1 score of 0.793. In this paper, we have demonstrated the feasibility of predicting people’s brain functioning by combining mobile sensing and fMRI data. Based on our insights from the NeuroSence study, we plan to conduct a larger scale, longitudinal study of college students.

CCS Concepts: • Human-centered computing → Ubiquitous and mobile computing.