Smartphone sensing of social interactions in people with and without schizophrenia

Abstract

Social impairment is a cardinal feature of schizophrenia spectrum disorders (SZ). Smaller social network size, diminished social skills, and loneliness are highly prevalent. Existing, gold-standard assessments of social impairment in SZ often rely on self-reported information that depends on retrospective recall and detailed accounts of complex social behaviors. This is particularly problematic in people with SZ given characteristic cognitive impairments and reduced insight. Ecological Momentary Assessment (EMA; repeated self-reports completed in the context of daily life) allows for the measurement of social behavior as it occurs in vivo, yet still relies on participant input. Momentary characterization of behavior using smartphone sensors (e.g., GPS, microphone) may also provide ecologically valid indicators of social functioning. In the current study we tested associations between both active (e.g., EMA-reported number of interactions) and passive (GPS-based mobility, conversations captured by microphone) smartphone-based measures of social activity and measures of social functioning and loneliness to examine the promise of such measures for understanding social impairment in SZ. Our results indicate that passive markers of mobility were more consistently associated with EMA measures of social behavior in controls than in people with SZ. Furthermore, dispositional loneliness showed associations with mobility metrics in both groups, while general social functioning was less related to these metrics. Finally, interactions detected in the ambient audio were more tied to social functioning in SZ than in controls. Findings speak to the promise of smartphone-based digital phenotyping as an approach to understanding objective markers of social activity in people with and without schizophrenia.

Social impairment is a cardinal feature of schizophrenia (SZ). Smaller social network size, reduced social activity, and loneliness are highly prevalent (Eglit et al., 2018; Macdonald et al., 2000), present prior to illness onset (Fulford et al., 2013; Gayer-Anderson & Morgan, 2013), and contribute to poorer outcomes, including increased hospitalization and lower quality of life (Erickson et al., 1998; Salokangas et al., 2006). Social skills deficits, social cognitive impairment, and negative symptoms (e.g., amotivation) are primary contributors to social dysfunction (Degnan et al., 2018; Fulford et al., 2017; Mueser et al., 1991). Given the relative stability of social impairment across the illness course in SZ (Velthorst et al., 2016), there is a need to improve understanding of modifiable contributors to social impairment.

Most research on social impairment in SZ has focused on limited social quantity, as defined by smaller social networks or few close contacts. Loneliness, on the other hand, is the subjective experience of aloneness and can occur in the context of either adequate or inadequate objective social connection (Cacioppo et al., 2003). On average, people with SZ report significantly higher levels of loneliness than those in the general population (Eglit et al., 2018; Lim et al., 2018), although there is at least a subgroup of people with SZ who demonstrate limited social drive and associated low levels of loneliness (Fulford et al., 2018). As such, the relationship between loneliness and social functioning in SZ is likely complex, with some levels of loneliness being adaptive (Fulford & Mueser, In press). Nevertheless, there is mounting evidence that both those with and without SZ who endorse high levels of loneliness are significantly more likely to experience poor health outcomes, including worse mental health (depression; (Ludwig et al., 2020)) and physical problems (e.g., high rates of cardiovascular disease and early mortality; (Badcock et al., 2019; Holt-Lunstad et al., 2015)). An improved understanding of the everyday contributors to loneliness in people with SZ can serve to inform the development of interventions to address this critical problem.

Our knowledge of social impairment in people with SZ is derived primarily from assessments in which a trained interviewer asks questions about the interviewee’s recent or distant past that the person with SZ (or a knowledgeable informant) is asked to reflect on with sufficient detail for the interviewer to rate. Questions often require the reporter to think back weeks or months and identify specific examples of social interactions, including their frequency and content. Unfortunately, retrospective recall biases impacted by memory impairments or current mood state, as well as a lack of characterization of the contextual correlates of social impairment (e.g., opportunities for social engagement), limit the information gleaned from these interviews on day-to-day social functioning (Sabbag et al., 2012). For example, an interviewee might reflect on either the most recent or most salient social experiences when responding to interview prompts. These experiences may or may not best represent the respondent’s “typical” social behavior, which is what is often of most interest in social functioning assessments. Furthermore, the level of social impairment could be influenced by participant qualities (e.g., demand characteristics) or interviewer burden (e.g., time allotted to gather information), or a lack of understanding of normative social interactions. Levels of depression in the interviewee, for example, have been associated with under-reporting of interpersonal functioning (Ermel et al., 2017). Informant reports (e.g., from a caretaker) can be helpful, but they are also subject to biases or lack of detailed knowledge of the participant’s social life (Hambrecht & Häfner, 1997; Sabbag et al., 2012). For these reasons, a respondent’s level of social impairment could be either overestimated or underestimated.

To reduce the limitations inherent in standard clinician-rated instruments of social functioning, Ecological Momentary Assessment (EMA) has been used to capture momentary reports of social activity, providing a fine-grained picture of social functioning. Dozens of studies have examined momentary reports of social behavior in SZ using EMA (for a review see (Mote & Fulford, 2019)). On the whole, findings regarding frequency of interpersonal contact and quality of interactions are inconsistent. Moreover, among the few studies that have examined the relationship between EMA reports of social behavior and standardized assessments of social functioning, there is limited correspondence. We are also unaware of studies examining the association between dispositional loneliness and EMA-reported social behavior in daily life in people with SZ, despite the high prevalence of loneliness in this disorder.

One limitation of EMA is that it requires participants to report on their behavior, often at high frequency, over a long period of time (days, weeks, or months). Although EMA surveys are typically brief, some degree of burden is inherent in the requirement to interact with a mobile device in the context of daily life. Sensors commonly available on smartphones have the potential to serve as additional measures of social behavior that are completely unobtrusive (i.e., require no level of interaction from the participant) and often collect data continuously. These features can circumvent issues associated with the inconvenience and burden associated with EMA, including the inability to complete surveys during times in which it is inconvenient or otherwise infeasible (e.g., during social interactions, while at work, etc.).

Smartphone-based geo-location—the characterization of time use and movement—has been recently used to infer mobility associated with clinical outcomes (Barnett & Onnela, 2020; Cote et al., 2019). In early applications of the approach to mental health, higher self-reported depressive symptoms showed associations with less movement in both a community sample (Saeb et al., 2015) and in a study of women at risk for perinatal depression (Faherty et al., 2017). This method has recently been applied to understanding clinical outcomes in SZ. In a recent EMA study (Depp et al., 2019), people with SZ covered less distance than controls, and less distance covered was associated with higher negative symptoms. There is also some promise for the use of changes in mobility as a marker of relapse in SZ (Barnett et al., 2018). However, the use of passive smartphone data as indicators of social behavior and function in SZ is relatively unexplored.

Thus far, smartphone-based audio analysis has been used mostly in controlled settings to provide proof of concept (e.g., (Tan et al., 2020)). Researchers have recently expanded this approach to smartphone usage in daily life via recordings of phone calls to identify patterns characteristic of clinical change in bipolar disorder (Gideon et al., 2016; Khorram et al., 2018). We are unaware of any studies employing the use of audio data from smartphones collected in the ambient environment (i.e., not from phone conversations) as a marker of clinical outcomes in SZ. Such an approach allows for a broader characterization of social behavior, capturing interactions as they occur organically and spontaneously in the context of daily life; this approach is potentially more generalizable than collecting audio data only during times in which the person is engaging in phone conversations. The application of this passive sensing approach in the current study will inform future work in this burgeoning area of research.

In the current study, we examined both active (EMA self-report) and passive (sensor-based) indicators of social behavior, and their correspondence, in people with and without SZ. For passive metrics, we focused on mobility as assessed through geo-location using the global positioning system (GPS) and voice activity detection (VAD) as captured in the ambient environment via microphone. We examined associations of both sensing features with standard assessments of social impairment to understand the extent to which such metrics could be used as proxies for gold-standard measures of social functioning.

Method

Participants

Twenty people with schizophrenia or schizoaffective disorder (SZ) and 15 controls residing in the San Francisco Bay Area participated. Participants with SZ were recruited via clinician referrals and brochures/flyers posted in local clinics; none the participants in this group were hospitalized at the time of study entry. Control participants were recruited via community flyers and public advertisements on websites typically used to recruit for psychological research studies. Exclusion criteria were the following: a history of head trauma, stroke, neurological disease, or loss of consciousness; a current mood episode (depression or mania); substance dependence within the past six months; not fluent in English; below the age of 18 or above the age of 70. For controls, people with any past or current Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV-TR; American Psychiatric Association, 2000) Axis I diagnosis were excluded.

Measures

Clinical assessments.

Diagnoses were confirmed by trained research assistants using the Structured Clinical Interview for DSM-IV – Patient Version (SCID-P; First et al., 1994). We assessed social functioning using the Quality of Life Scale – Interpersonal Relations subscale (QLS-IR; Heinrichs et al., 1984) and dispositional loneliness using the 20-item UCLA Loneliness Scale (UCLA-LS; Russell et al., 1980).

Ecological Momentary Assessment (EMA).

As part of a larger study (Mote et al., 2019), participants were asked to complete brief surveys three times each day for seven consecutive days using the Ethica Data application (www.ethicadata.com) on smartphones we provided them for study purposes. Surveys were administered a minimum of 90 minutes apart throughout each day at pseudorandom time points within the windows of 10am-1pm, 2–5pm, and 5–8pm. Participants were asked a variety of questions related to their current social context. There were two items we focus on in the current study: 1) “Who are you with?” (response options were ‘Alone,’ ‘Alone at home with someone I know who is in another room,’ ‘Around other people at home,’ ‘In public around people I don’t know,’ and ‘In public around people I know’) and 2) “Since the last prompt, how many times did you talk or communicate with someone?” (response options were ‘No interactions,’ ‘1 interaction,’ ‘2 or 3 interactions,’ ‘4 or more interactions.’ For the first item, we dichotomized scores to indicate whether the participant was alone (including being alone at home with someone else in another room) or with others¹. Mode of social interaction (e.g., in person, over the phone, using social media) was not assessed.

Study procedures

Potential participants completed a brief phone screening prior to being invited to participate in the study. Once invited to participate, participants met with a trained research assistant and completed informed verbal and written consent. Participants were informed of study details, including that geo-location and microphone data would be gathered semi-continuously throughout the study. Next, participants completed a clinical interview and demographics questionnaire. The researcher then provided the participant with a smartphone and introduced them to the Ethica Data application, including the EMA and sensing portions of the study. Across seven days, participants completed prompted EMA surveys and were asked to wear a round button in a visible location on their clothing, approximately three inches in diameter, to indicate to others in their immediate environment that conversations could be recorded. One week later, participants returned the phone to the researcher (either by returning to the research laboratory where the original assessment took place, or during a home visit the researcher made as part of a larger ongoing study). Other clinical measures (i.e., QLS and UCLA-LS) were completed at this visit. Study procedures were approved by the Institutional Review Board from San Francisco State University.

Data Analyses

Data preparation is described in Supplemental Materials. We first compared groups on our active (EMA) and passive (sensing) metrics. From ambient audio (rVAD) estimations, we compared groups on: 1) instances of speech activity across all recordings, 2) instances of speech activity in the time between consecutive EMA prompts, and 3) the relationship between the average duration of instances of speech activity and the number of EMA-reported interactions between prompts. For all correlational analyses, we aggregated repeated assessments across the seven-day study period to create a mean value for each variable.

A primary question concerns the extent to which EMA reports of social activity cohere to passive metrics. As such, we examined bivariate correlations between EMA-reported number of interactions experienced and proportion of EMA signals spent alone with mobility metrics and voice activity, separately by group. Given the exploratory nature of the analyses, we considered Pearson correlation magnitudes of 0.30 or greater (moderate effect) as meaningful signals, and thus summarize these findings (Cohen, 2013). We also flag correlations with p values less than 0.05, following convention. A secondary question concerned the extent to which standard assessments of social functioning and perceived social connection (i.e., loneliness) related to the above sensing metrics. As such, we examined bivariate correlations between passive metrics (mobility, ambient audio) aggregated across the study with both the QLS-IR and the UCLA-LS, separately by group. We then compared the magnitude of correlations between groups for both the primary and secondary analyses using Fisher r to Z transformations, with a two-tailed p value of 0.05 as an indication of a statistically significant difference between correlations. Consistent with standards of adherence to EMA surveys in other published studies in SZ (e.g., (Granholm et al., 2013)), we included data from participants who responded to a minimum of 25% of EMA prompts in analyses; all participants in this study met this minimum threshold.

Results

In preliminary analyses, we examined group differences in baseline variables and usable data from sensing metrics. People with SZ were older and less likely to be employed, on average, than controls (each p < 0.05). Consistent with prior work, people with SZ also reported significantly more dispositional loneliness and had significantly lower interviewer-rated social functioning than controls (ps < 0.01). Groups did not significantly differ in the number of GPS data points or in the amount of audio data collected; however, a small effect size indicated that controls had more usable GPS data points than the SZ group (Cohen’s d = 0.38) (see Table 2).

Table 2.

Sample characteristics, social behavior measures, and sensing metrics

	SZ (n = 20)	Controls (n = 15)	t or chi-squared	p	Cohen’s d or V
	M (SD) or n (%)	M (SD) or n (%)
Demographics
Age	53.30 (7.70)	43.33 (14.15)	−2.47	0.023	0.88
Male	15 (75%)	11 (73%)	0.01	0.911	0.07
Years Education	14.06 (3.02)	15.47 (3.20)	1.26	0.219	0.45
Employed	5 (29%)	12 (80%)	8.19	0.004	0.11
Married/Cohabitating	3 (18%)	1 (7%)	0.88	0.349	0.07
White/Caucasian	6 (35%)	9 (60%)	1.95	0.162	0.72
Social Behavior
UCLA-LS	33.14 (9.59)	21.08 (8.36)	−3.39	0.002	1.34
QLS-IR	2.33 (1.52)	4.89 (1.29)	5.02	< 0.001	1.82
Mobility
GPS data points	22,302 (15,618)	27,377 (15,628)	1.16	0.255	0.38
Distance traveled (km)	33.44 (58.13)	74.99 (154.86)	0.92	0.371	0.36
Flight duration (min)	3.15 (6.46)	2.36 (3.41)	−0.40	0.689	0.15
Time at home (hrs)	15.37 (4.44)	14.21 (7.21)	−0.52	0.611	0.19
Significant locations	1.88 (0.70)	1.57 (0.53)	−1.37	0.182	0.50
Pause (probability)	0.81 (0.12)	0.85 (0.13)	0.75	0.461	0.32
Routine	0.67 (0.19)	0.64 (0.27)	−0.42	0.677	0.13
Speech
Total audio duration (hrs)	21.89 (8.19)	22.06 (9.98)	0.06	0.956	0.02
rVAD duration (hrs)	4.21 (1.91)	3.96 (1.87)	−0.39	0.702	0.13

Notes. All participants reported their age and gender. Three participants with SZ did not report on other demographic information; QLS-IR = Quality of Life Scale – Interpersonal Relations; rVAD = robust Voice Activity Detection; UCLA-LS = UCLA-Loneliness Scale

Females with SZ had lower overall social functioning (QLS-IR) than did males (t = 2.38, p < 0.05). People with SZ who were currently employed reported significantly more social interactions per EMA prompt (M = 2.23) than did those who were unemployed (M = 1.52; t = −2.30, p < 0.05). People with SZ who were married or cohabitating also had higher social functioning (QLS-IR) than did those who were unmarried or not cohabitating (t = −2.47, p < 0.05).

No demographic variables were associated with EMA-reported time spent alone or dispositional loneliness. Demographic variables were also mostly unrelated to mobility metrics or duration of speech detected using rVAD in either group, with two exceptions: people with SZ who were married or cohabitating 1) were less likely to spend the study period stationary relative to moving (i.e., paused; t = 2.29, p < 0.05) and 2) had longer speech duration detected (t = −2.22, p < 0.04) than those were unmarried/not cohabitating.

Associations between sensing metrics and EMA-reported social activity

As reported in a prior manuscript (see Mote et al., 2019), groups did not differ in average number of EMA interactions reported or proportion of EMA signals alone. Because controls had slightly more usable GPS data points than those with SZ, we examined associations between usable data and EMA-reported social activity. Across groups, usable GPS data was unrelated to either the number of interactions reported (r = 0.17, p = 0.34) or the proportion of time spent alone (r = 0.01, p = 0.94) reported via EMA.

Mobility.

In controls, a higher average number of EMA-reported interactions across the study period was significantly associated with longer daily distance traveled (r = 0.60) as captured by imputed GPS trajectories. There were also moderate associations between more EMA-reported interactions and shorter duration of movements (r = −0.47) and higher probability of being stationary relative to moving (r = 0.31). Higher proportion of time spent alone during EMA signals across the study period was associated with significantly more time spent at home (r = 0.57) and higher likelihood of sticking to a weekday routine (r = 0.70). Higher proportion of time spent alone was also moderately associated with less distance covered (r = −0.38) and fewer significant locations visited (r = −0.48).

In people with SZ, average number of EMA-reported interactions across the study was mostly unrelated to GPS-derived mobility metrics. More interactions reported across the study were moderately associated with less time spent at home (r = −0.37) and lower probability of being stationary relative to moving (r = −0.36). As in controls, higher proportion of EMA-reported time spent alone across the study was significantly associated with fewer significant locations visited (r = −0.56).

The only statistically significant difference in correlation magnitudes between the groups was in the relationship between proportion of time spent alone and likelihood of sticking to a weekday routine (Z = 2.09, p = 0.04), an association that was present among controls but not those with SZ.

Voice Activity.

For controls, number of EMA-reported interactions across the study was moderately associated with longer duration of voice activity detected (rVAD; r = 0.46). In people with SZ, voice activity was unrelated to the number of EMA-reported interactions. Interestingly, a higher proportion of EMA signals in which the participant reported being alone was related to significantly more voice activity detected across the study (r = 0.51) in people with SZ.

We also examined the duration of speech activity within each EMA block. Across both groups, longer duration of speech activity in a given block appeared to be associated with more interactions reported in that same block (see Figure 1). However, when examining this relationship separately by group, controls appeared to be more likely than people with SZ to demonstrate voice activity during both blocks in which they reported no interactions and blocks in which they reported four or more interactions.

Figure 1.

Average duration in seconds of speech detected per ambient recording as a function of the number of interactions reported in the EMA (results are reported for controls, the SZ group, and combined)

Associations between sensing metrics and social functioning

Mobility.

In controls, higher dispositional loneliness (UCLA-LS) was associated with significantly shorter mean movement duration (r = −0.58). Loneliness was also associated with moderately more significant locations visited (r = 0.57), less time spent at home (r = −0.31), and lower likelihood of following a weekday routine (r = −.46). Higher social functioning (QLS-IR) showed moderate associated with several mobility metrics, including more distance traveled (r = 0.36), shorter movement durations (r = −0.43), less time spent at home (r = −0.37), and lower likelihood of following a weekday routine (r = −0.41).

In people with SZ, higher dispositional loneliness was associated with significantly fewer significant locations visited (r = −0.58). Loneliness was also moderately associated with more distance traveled (r = 0.32), longer mean movement duration (r = 0.47), and lower likelihood of following a weekday routine (r = −0.40). Higher social functioning was moderately associated with more distance traveled (r = 0.36) and shorter mean movement duration (r = −0.40).

The association between loneliness and movement duration—a negative correlation in controls, and a positive correlation in people with SZ—was significantly different between groups (Z = −3.10, p < 0.01). The association between loneliness and number of significant locations visited—a positive correlation in controls, and a negative correlation in people with SZ—was also significantly different between groups (Z = 3.47, p < 0.01).

Voice Activity.

Voice activity (rVAD) was not associated with social functioning measures (UCLA-LS or QLS-IR) in controls. In people with SZ, more voice activity was moderately associated with lower dispositional loneliness (r = −0.45) and higher social functioning (r = 0.39).

Discussion

We explored the utility of smartphone sensor data, including mobility captured via GPS and speech captured via microphone, as indicators of social behavior and function in people with and without SZ. Findings suggested these data show promise as correlates of measures of social activity, functioning, and loneliness, with areas of divergence between people with and without SZ. In general, mobility appeared more promising as an indicator of social behavior in controls (i.e., significant associations in five of six mobility variables) than in people with SZ (i.e., significant associations in one mobility variable). On the other hand, speech activity showed more consistent associations with measures of social behavior in people with SZ than in controls.

Mobility and Social Activity

Several mobility metrics showed strong associations with EMA-reported social activity in controls, suggesting these data may provide information on momentary reports of social activity for this group. For example, more interactions reported were associated with longer distance traveled, which may indicate that this measure represents engagement in activities or role responsibilities that involve social contact (e.g., work commute, travel to social groups). Time spent at home and consistency in schedule (i.e., weekday routines), on the other hand, might serve as proxies for social isolation, as indicated by strong associations with proportion of EMA signals reported alone. In general, mobility may provide some indication of levels of social activity in controls.

In people with SZ, mobility was mostly unrelated to momentary reports of social activity. In fact, only one mobility marker—fewer significant locations visited—was associated with higher proportion of time spent alone. Significant locations visited may reflect engagement in structured activities that involve the presence of others (e.g., work/school, self-care) among people with SZ. The general lack of association between EMA reports of social activity and mobility markers in people with SZ could reflect less overall mobility in this group, consistent with prior work (Depp et al., 2019). Indeed, although not statistically significant, people with SZ covered less than half the distance per day than did controls while in the study (33 vs. 75 km). Given distance traveled was strongly associated with number of interactions reported in controls, the limited range in this mobility metric among those with SZ in this study may preclude identification of potential associations with social activity in this group.

We also found associations between social activity measures at baseline and mobility in controls—dispositional loneliness showed associations with several mobility measures, while general social functioning was less related. Lower loneliness at baseline was associated with longer flight duration, more time spent at home, and higher likelihood of following a routine, while higher loneliness was associated with more significant locations visited. While difficult to identify a clearly discernible pattern, these findings indicate that loneliness could be a function of a combination of mobility metrics that may, together, speak to the extent to which people feel connected with others in their social worlds. For example, duration of movement, time at home, and daily routine could reflect engagement in social activities with close others (e.g., romantic partners, friends), while significant locations visited could reflect more time spent engaging in superficial connections (e.g., coworkers, acquaintances), in this group. Although there were no significant associations between social functioning and mobility, the pattern of associations was consistent with findings regarding loneliness, suggesting loneliness (perceived social connection) and social functioning (quantity of social activity) were related to mobility in similar ways in this group.

Dispositional loneliness was also related to several mobility metrics in people with SZ, albeit in some opposing ways. More significant locations visited was significantly associated with lower loneliness, and lower likelihood of following a daily routine was moderately associated with lower loneliness. More distance traveled and longer movement duration, however, were moderately associated with higher loneliness. For this group, significant locations may involve more in-depth social connection, such as in settings with loved ones or treatment providers. On the other hand, general mobility, as indicated by distance traveled and duration in movement, may reflect travel to activities involving less meaningful connection (e.g., work or errands). Although social functioning was mostly unrelated to mobility in this group, more distance traveled, but less time spent moving, were related to moderately higher functioning at baseline. It could be that those with higher social functioning were more likely to engage in work or other occupations that were reflected in these mobility metrics.

Voice Activity Detection and Social Activity

We captured hundreds of hours of ambient audio and processed these data using automated voice activity detection algorithms (rVAD) to examine the utility of conversation detection as a marker of social outcomes. The is the first and only study we are aware of that has used this approach to quantify speech activity using data collected from smartphones in naturalistic environments. Performance of the rVAD was promising, especially given the limited quality of these naturalistic audio data (see Supplemental File 2).

There were no group differences in the number of ambient recordings captured, or in the duration of speech detected within these recordings. Furthermore, when parceling rVAD by EMA block, we identified an overall positive association between number of interactions reported and duration of voice detected within each block. However, interesting group differences emerged, such that controls had significantly more voice activity when they reported having no interactions than when they reported having a few (1–3) interactions, while people with SZ had significantly more voice activity when they reported having 2–3 interactions than when they had 4 or more interactions. It could be that controls were more likely to engage regularly in conversations that they did not consider as social interactions (e.g., superficial “small talk”) than those with SZ. People with SZ, on the other hand, may have had a lower threshold for defining a social interaction. In future work, more information on the content of voice activity detected could help clarify support for these assumptions.

We also examined associations between rVAD and social activity and functioning in both groups. More interactions reported via EMA were moderately associated with longer duration of voice activity in controls. This finding suggests that speech activity served as an indicator of the quantity of social interactions across the study period. For people with SZ, the number of EMA-reported interactions was unrelated to voice activity. The proportion of EMA signals in which participants with SZ were alone, however, was significantly associated with more speech across the study period. This finding could simply reflect that people with SZ who engaged in more conversations were more likely to respond to EMA signals when alone than when with others. Another possibility is that people with SZ were more likely to engage in self-talk while alone, and less likely to speak when with others. In future work, examining the amount of speech in the context of different types of social interaction (e.g., in the presence of strangers vs. close contacts) could help clarify potential ways in which groups differ in patterns of speech and social activity. We also found that more voice activity was related to moderately lower loneliness and higher social functioning in people with SZ, providing preliminary evidence that passively collected data on speech activity could serve as a proxy for social behavior. In all, these findings suggest that speech in naturalistic environments could serve as an indicator of meaningful social outcomes among people with SZ.

Summary and Conclusions

This study adds to the rapidly growing body of work capitalizing on the ubiquity of smartphones for capturing naturalistic data in people with SZ. In the current study, we did not examine associations with symptoms or other makers of psychopathology, and instead focused on social behavior. Previous studies have examined sensor-based data as predictors of relapse in SZ. Buck and colleagues (Buck et al., 2019) identified reductions in the number and duration of outgoing phone calls and number of text messages sent as significant predictors of relapses. These findings, in addition to work mentioned above linking mobility with relapse and symptom severity (Barnett et al., 2018; Depp et al., 2019), suggest smartphone sensors can supplement, and may even enhance, the assessment of meaningful outcomes in this population.

One limitation of the current study is that we did not focus on the quality of social relationships. Social relationships are not always supportive and can be sources of conflict and stress (Kavanagh, 1992; Thoits, 2011). For example, high expressed emotion (over-involvement and hostility) from significant others can lead to relapse and hospitalization in people with SZ (Barrowclough & Hooley, 2003). It will be important in future to work to examine the relevance of both the quantity and quality of social activity for understanding social functioning in SZ. Another limitation is that our sample size was relatively small—as such, tests of group differences were likely underpowered. Indeed, although there were several associations between sensor data and social functioning measures that appeared to differ between groups, only three correlations were statistically significant from each other between groups (i.e., EMA-reported time spent alone with likelihood of following a routine, and dispositional loneliness with flight duration and significant locations visited). We also could not adequately test for the potential confounds of age and race/ethnicity on the associations examined in this study. Nonetheless, we did gather up to 21 EMA reports on social activity and semi-continuous sensor data on each participant, giving us reliable estimates of these features within each group. Testing such associations in larger samples, using statistical methods that more directly capture the dynamic nature of these data (e.g., multilevel modeling), will allow for detection of more subtle group differences. Relatedly, sampling of social behavior at greater frequency using EMA would allow us to test more fine-grained associations with sensor data. Finally, because we provided all participants with study phones, the extent to which participants kept the phones ‘on hand’ could have influenced findings. For example, if controls were more likely to have a personal phone than people with SZ, it is possible that data we collected were biased toward adherence. However, we did not see group differences in adherence to EMA surveys or in passive sensor data collected, making it less likely that groups differed in amount of time they kept the study phones on them. Future work can help determine the extent to which loaner phones might influence both active and passive metrics of social behavior.

Our findings speak to the promise of digital phenotyping—the moment-by-moment quantification of behavior in situ using smartphones—as an approach to understanding objective markers of social activity in people with and without SZ. This is an important step forward given the challenges in obtaining reliable and valid data on social functioning in SZ, including the reliance on clinical interviews that introduce retrospective recall and other biases.

Table 1.

Description of smartphone sensors

Metric	Description
Mobility
Distance traveled	Mean distance traveled per day in kilometers
Flight duration	Mean of each movement duration in minutes per day
Time at home	Mean time spent at home per day in hours
Significant Locations	Mean number of significant locations visited per day
Pause	Mean probability of being stationary relative to time moving per day
Routine	Mean circadian routine (break in routine vs. following routine) per day
Speech
rVAD	Amount of time in which user speech is detected in hours

Table 3.

Bivariate correlations between sensing metrics (GPS mobility and ambient audio) and loneliness, social functioning, and EMA reports of social engagement – Controls

GPS Mobility	Number of Interactions (EMA)	Proportion of Time Alone (EMA)	UCLA-LS	QLS-IR
Distance traveled (km)	0.603*	−0.375	0.007	0.355
Mean flight duration (min)	−0.466	−0.213	−0.581*+	−0.425
Time at home (hrs)	−0.125	0.569*	−0.309	−0.373
Significant locations (number)	−0.158	−0.476	0.566+	0.220
Pause (probability)	0.307	0.242	0.153	0.157
Weekday routine	−0.167	0.699*+	−0.459	−0.412
Ambient Audio
rVAD	0.463	0.214	−0.280	0.142

Note. QLS-IR = Quality of Life Scale-Interpersonal Relations; rVAD = robust Voice Activity Detection; UCLA-LS = UCLA Loneliness Scale; for UCLA-LS and QLS-IR, ns = 12 to 13 Light orange = moderate effect; dark orange = large effect;

^*p < 0.05;

^{+ =}significantly different from SZ group, two-tailed p < 0.05.

Table 4.

Bivariate correlations between sensing metrics (GPS mobility and ambient audio) and loneliness, social functioning, and EMA reports of social engagement – Schizophrenia

GPS Mobility	Number of Interactions (EMA)	Proportion of Time Alone (EMA)	UCLA-LS	QLS-IR
Distance traveled (km)	0.065	−0.140	0.323	0.356
Mean flight duration (min)	0.154	0.173	0.466+	−0.404
Time at home (hrs)	−0.372	0.289	−0.285	−0.088
Significant Locations (number)	0.064	−0.563*	−0.583*+	−0.097
Pause (probability)	−0.360	0.219	−0.174	0.070
Weekday routine	−0.201	0.077+	−0.399	−0.136
Ambient Audio
rVAD	0.017	0.509*	−0.446	0.394

Note. QLS-IR = Quality of Life Scale-Interpersonal relations; rVAD = robust Voice Activity Detection; UCLA-LS = UCLA Loneliness Scale; for UCLA-LS and QLS-IR, ns = 12 to 13 Light green = moderate effect; dark green = large effect;

^*p < 0.05;

^{+ =}significantly different from controls, two-tailed p < 0.05.

Highlights.

• Smartphone sensors allow for continuous, unobtrusive assessment

• Mobility from GPS and speech from microphone can be used to quantify social activity

• Each sensor provides unique information for social activity in people with and without schizophrenia