Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2022 Jun 1.
Published in final edited form as: J Psychiatr Res. 2021 Apr 30;138:500–506. doi: 10.1016/j.jpsychires.2021.04.039

Social Functioning in Schizophrenia: Comparing Laboratory-based Assessment with Real-world Measures

Danielle B Abel a, Kyle S Minor a
PMCID: PMC8192459  NIHMSID: NIHMS1699614  PMID: 33971484

Abstract

Researchers have measured social functioning in schizophrenia using many different strategies. Recent technological advances have made it possible to passively measure behaviors in real-world social situations—allowing for more objective, ecologically valid assessments. Yet, research testing the convergent validity among real-world and laboratory-based social functioning assessment is sparse. The purpose of this study was to test the convergent validity among four social functioning measures: two interview-based rating scales, a self-reported ecological momentary assessment (EMA), and a passive, ambulatory ecological assessment. Data was collected from 36 people with schizophrenia and 33 control participants. Across the entire sample, relationships between interview-based ratings and real-world measures of social functioning only demonstrated small correlations (r’s=0.17–0.19), whereas real-world measures exhibited moderate correlations with one another (r=0.36). Within groups, real-world measures showed moderate, significant relationships in the control group (r=0.44) but not in the schizophrenia group (r=0.27). For those with schizophrenia, the interview-based measures of social functioning were moderately associated with ambulatory ecological assessment (r’s=0.38 and 0.47), but only small associations were observed with self-reported EMA (r’s=0.15 and 0.17). Results suggest social functioning assessments are not highly convergent and likely target different aspects of social functioning. Laboratory-based measures offer global impressions of social functioning whereas real-world measures represent a more nuanced approach. Moreover, ambulatory ecological assessment may most accurately gauge frequency of daily social interactions for those with schizophrenia as it circumvents common pitfalls of self-report and offers a less-biased, in-depth evaluation of social behavior.

Keywords: Schizophrenia, Social Functioning, Ecological Momentary Assessment, Mobile Sensing, Ambulatory Ecological Assessment

Introduction

From questionnaires and structured interviews to day reconstruction and ecological momentary assessment (EMA), researchers have measured social functioning in schizophrenia using many different strategies. Although most strategies rely on participants subjectively describing their experiences, technological advances have allowed researchers to unobtrusively assess behaviors in real-world social situations (Ben-Zeev et al., 2016; Buck et al., 2019; Mehl, 2017; Robbins et al., 2014), yielding more ecologically-valid assessments. For people with schizophrenia, these new methodologies offer potential for accurately gauging real-world social behaviors.

Forms of Social Functioning Assessment

Social functioning in schizophrenia has conventionally been measured in laboratories with clinician-rated assessments based on interviews (Bellack et al., 2007). These time- and cost-efficient scales are widely used and produce good psychometric data (Birchwood et al., 1990; Cornblatt et al., 2007; Heinrichs et al., 1984; Startup et al., 2002). However, like all laboratory measures, interview-based assessments have limited ecological validity. They often require subjects to report behaviors from days, months, or years ago and are vulnerable to retrospective error. This is especially problematic when assessing those with psychotic illnesses who often experience issues—cognitive impairment, poor memory, lack of insight—that affect subjective reporting (Bellack et al., 2007; Durand et al., 2015). Clinical ratings of laboratory interviews are also limited by potential response biases (e.g., social desirability, demand characteristics) and their inability to account for the complexity of peoples’ daily social behaviors.

Real-world assessment offers a solution to gain immediate, ecologically-valid assessments of functioning as data on participants’ feelings, behaviors, and environment are collected in the moment or shortly thereafter (Hektner et al., 2007). This immediacy protects against retrospective bias, and the real-world context of assessment increases external validity. Over the last two decades, researchers have employed ecological momentary assessment (EMA) to actively engage subjects to self-report social behaviors throughout their day via paper assessments (Myin-Germeys et al., 2001) or mobile devices (Depp et al., 2016). Using EMA with people with schizophrenia is feasible and generally well-accepted (Granholm et al., 2008; Oorschot et al., 2009). However, compliance across studies has been mixed with response rates ranging from 28%−69% (Granholm et al., 2008; Moitra et al., 2017). Compliance may depend on study length, compensation, or clinical status of participants (e.g., outpatient versus inpatient). EMA studies suggest people with schizophrenia report more social stress and preference to be alone compared to controls, but also experience more positive affect with others than when alone (Mote et al., 2019). Questions remain regarding qualitative aspects of social experiences for those with schizophrenia (Mote et al., 2019). Although EMA has greater ecological validity than laboratory-based assessments, it is still limited by its reliance on subject compliance and vulnerability to response biases. Moreover, EMA may be burdensome on subjects and more expensive than clinician-rated assessments.

Alternatively, ambulatory ecological assessment uses mobile devices to passively collect data by recording behaviors, sounds, or movements. Because it bypasses subjects’ recollection and response biases, ambulatory ecological assessment has been considered more objective (Mehl, 2007). It also allows researchers to increase measurement frequency without overburdening subjects and is less dependent upon subjects’ compliance. This advantage is relevant for people with schizophrenia who show lower response rates to EMA than healthy people (Rintala et al., 2019). Thus, ambulatory ecological assessment increases ecological validity and circumvents common pitfalls of subjective reports. However, the novelty of this method makes it challenging to compare results to past research. Ambulatory ecological assessment also relies on the development of valid and reliable coding systems and may be more time-consuming for researchers. Table 1 summarizes the strengths and weakness of ambulatory ecological assessment compared to other methods.

Table 1.

Comparing three methodologies of assessing social functioning in schizophrenia

Clinician-rated scales Ecological Momentary Assessment Ambulatory Ecological Assessment
Examples The Global Functioning Scale—Social (Cornblatt et al., 2007); Heinrich’s Quality of Life Scale: interpersonal relations scale (Heinrichs et al., 1984); Social Functioning Scale (Birchwood et al., 1990) “Since the last questionnaire, about how many times did you talk or communicate with someone else?” (Granholm et al., 2013); “Are you currently alone?” (Oorschot et al., 2012); “We are interacting” rated on 7-point Likert scale (Leendertse et al., 2018) The Electronically Activated Recorder (Mehl, 2007); Voice Activity (Fulford et al., 2020)
Strengths Widely used with good psychometric data for many measures; Time and cost efficient; Wide use makes it easy to compare with previous findings Moderately used with evidence for feasibility and validity; Momentary assessment bypasses subject’s recollection; Can account for changes in functioning across time points Momentary assessment bypasses subject’s recollection; More objective ratings since data collection circumvents subject’s response biases; Can account for changes in functioning across time points; less burdensome on participant
Weaknesses Retrospective self-report is vulnerable to response biases (e.g., incorrect recall, social desirability bias, demand characteristics); Completed in laboratory environment which limits external validity; General measure that cannot account for fluctuations in social functioning in everyday life Self-report is subjective and vulnerable to response biases (e.g., incorrect recall, social desirability bias); Relies on subject compliance, and missing data is inevitable; Responding to multiple prompts throughout the day requires significant subject time and effort; Subject may need training to operate mobile technology Novelty of method makes it more challenging to compare to previous research; Relies on development of psychometrically valid and reliable coding systems; Coding data may be more time consuming; Misses social interactions outside of recording interval; Defines social interaction by speech and may miss non-verbal communication (e.g., text messaging, online chat, etc.)
Open in a new tab

In schizophrenia, researchers have begun employing ambulatory ecological assessment by using mobile devices to collect location, movement, sleep duration, and voice activity (Ben-Zeev et al,. 2015; Ben-Zeev et al., 2016; Fulford et al., 2020). These studies demonstrated the feasibility and acceptability of ambulatory ecological assessment in those with schizophrenia (Ben-Zeev et al., 2016) and discovered behavioral markers that are associated with mental health (Ben-Zeev et al,. 2015). In healthy populations, Mehl (2017) has successfully used audio recording to assess socially relevant activities during daily life. However, only two studies (Abel et al., 2021; Fulford et al., 2020) have used ambulatory ecological assessment to measure social functioning in schizophrenia. Results of these preliminary studies suggest momentary audio data can be collected from people with schizophrenia with relatively little interference in their daily lives (Abplanalp et al., 2021). Moreover, people with schizophrenia may interact with others at the same rate as healthy controls but exhibit less substantive interactions (Abel et al., 2021). Further work is needed to replicate these findings and determine the value of ambulatory ecological assessment in measuring social functioning in schizophrenia.

Comparing Methodologies

Although different forms of real-world and laboratory assessments are intended to assess social functioning, it is unclear how these different methods compare to one another. In a review of laboratory-based assessments of social functioning in schizophrenia, Burns & Patrick (2017) found scales varied in their measurement approach, types of domains covered, and scoring systems. Furthermore, clinician-rated scales seem to tap into broad social functioning including both quality and quantity of social interactions, whereas real-world assessments simply detect quantity of daily social interactions. Studies that have begun to test the convergence between momentary social behavior and laboratory functioning scales have shown mixed results (Gard et al., 2014; Granholm et al., 2008; Vasconcelos e Sa et al., 2016). In support of the convergent validity between these methods, Granholm and colleagues (2008) and Gard and colleagues (2014) found moderate associations between laboratory assessments and momentary assessments of social behaviors. Conversely, Vasconcelos e Sa and colleagues’ (2016) investigation of the associations between EMA and clinician-rated functioning yielded null results. Schneider and colleagues (2017) compared a laboratory-based scale with momentary measures of social behavior and found significant correlations with some EMA variables (e.g., time spent alone/with others), but not others (e.g., preference for being alone/with others). Moreover, there were fewer significant associations between EMA and laboratory measures in the control group compared to the schizophrenia group, suggesting laboratory-based scales may be less sensitive to detecting differences in social functioning for higher functioning individuals.

Fulford and colleagues (2020) is the first schizophrenia study to compare ambulatory ecological assessment to other social functioning methods. This study tested associations between momentary self-reported number of interactions (i.e., EMA) with smartphone recorded mobility and voice activity (i.e., ambulatory ecological assessment). These two forms of real-world assessment were moderately correlated in healthy control participants but were unrelated in schizophrenia. Fulford and colleagues (2020) further compared real-world measures to a laboratory-based social functioning measure. Results showed increased voice activity detected by ambulatory ecological assessment was moderately associated with higher social functioning in those with schizophrenia; yet, ambulatory and laboratory-based social functioning measures were unrelated in healthy controls. This suggests that, compared to healthy controls, people with schizophrenia may exhibit different patterns of daily social behavior which are reflected in some social functioning measures (e.g., laboratory measures), but not others (self-reported EMA).

Clearly, further research is needed to test the convergent validity between EMA, ambulatory ecological assessment, and laboratory-based measures of social functioning. Identifying convergence between different approaches will provide support for their validity and allow researchers to better understand the utility of different methods across people and contexts.

The Current Study

This study’s primary goal was to test the convergent validity among four social functioning assessments: self-reported EMA, ambulatory ecological assessment (using passive audio recording), and two interview-based rating scales. We hope to replicate Fulford et al. (2020)’s findings in a larger sample and further test the utility of implementing ambulatory ecological assessments of daily social behaviors. We expected moderate associations across all four measures, with the highest associations between similar measures (i.e., the two real-world assessments, the two laboratory rating scales). A secondary goal was to test whether associations differed in schizophrenia and control groups. Based on findings by Fulford et al. (2020) and Schneider et al. (2017), we anticipated stronger associations between real-world assessments and laboratory measures of social functioning in the schizophrenia group than the control group. Regarding comparisons between EMA and passive ambulatory assessment, we expected the control group to exhibit larger associations than the schizophrenia group (see Fulford et al., 2020).

Method and Materials

Participants

This study involved secondary data analysis of two unpublished projects. Inclusion criteria for participants were: a) age 18–60; b) English fluency; c) no change in medication or outpatient status in the 30 days prior to testing; d) ability to give informed consent; e) no active substance dependence; f) no documented intellectual disability; and g) no history of neurological illness or traumatic brain injury with loss of consciousness greater than five minutes. The schizophrenia group was recruited from community mental health clinics in Indianapolis, IN and had a diagnosis of a schizophrenia-spectrum disorder according to the Mini International Neuropsychiatric Interview (M.I.N.I.; Sheehan et al., 1998). The control group was recruited via local health care centers and a volunteer registry. Control participants were excluded if they met criteria for a current psychiatric disorder or experienced past psychotic symptoms. Participants who did not complete all study procedures were excluded from analyses.

Measures

Laboratory methods.

The global functioning scale: social (GFS-S; Cornblatt et al., 2007).

The GFS-S measures one’s overall ability to get along with others and maintain age-appropriate relationships. Based on a clinical interview, it uses a single item rated on a 10-point scale to represent the gestalt of social functioning. Greater scores represent higher functioning. The GFS-S was adapted from two “gold-standard” instruments for measuring social functioning in schizophrenia (Hall, 1995; Skodol et al., 1988). It demonstrated high interrater reliability (ICC=0.85, p<0.001; Cornblatt et al., 2007) and high construct validity in schizophrenia-spectrum populations (Auther et al., 2006; Piskulic et al., 2011).

Heinrich’s quality of life scale: interpersonal relations scale (QLS-IR; Heinrichs et al., 1984).

The QLS-IR is one subscale of a larger measure of quality of life for those with schizophrenia. It is composed of 8-items rated on a 7-point scales that focus on specific domains of social functioning (amount of social contact, capacity for intimacy, active versus passive social participation, social avoidance tendencies). Higher scores indicate greater functioning. The QLS-IR is widely used and shows good inter-rater reliability (ICC=0.89, p<0.01) and high convergence with similar measures (r=0.81, p=< 0.01; Gupta et al., 2000).

Real-world Assessments.

Self-report EMA.

Social journals are paper packets where participants record their daily activities. For each hour, participants report what they were doing (free response format), if they were wearing the EAR (yes/no), and whether they interacted with others (3-point scale: 1=No interaction; 2=Around others but not interacting; 3=Interacting with others). The interaction variable was dichotomized to match the ambulatory ecological assessment (see below) whereby a 1 or 2 rating signified no interaction and a 3 signified an interaction. From this measure, we computed a self-reported frequency of social interaction score. Similar diary methods have been used for both clinical and healthy populations (Csikszentmihalyi and Larson, 1987; Myin-Germeys et al., 2003; Myin-Germeys, et al., 2001).

Ambulatory Ecological Assessment.

The electronically activated recorder (EAR; Mehl, 2007) is a smartphone application that collects real-world observations of social interactions through audio recording (Pennebaker et al., 2003). The EAR was programmed to audio record for 5 minutes every 90-minutes between 6:00am and 12:00am, yielding 24 audio recordings over two days. This time frame shows good temporal stability (Mehl and Robbins, 2012) and convergence with four-week periods (Mehl et al., 2001; Mehl et al., 2012). To address confidentiality, participants could remove the EAR at any time, and the device included a sign alerting others that it could be recording. Participants also had the opportunity to hear and delete any recordings they did not wish to share prior to the research team hearing them. Consistent with previous studies, participants deleted less than 1% of files (Minor et al., 2018; Robbins et al., 2011; Robbins et al., 2014).

Audio recordings were then coded for social interactions using the Social Environment Coding of Sound Inventory (SECSI; Mehl, 2017; Mehl et al., 2012; Mehl et al., 2007). Social interactions were operationalized as any time the subject spoke with another person during the recording. This was rated on a two-point scale (0=No interaction, 1=Interaction). Five trained raters completed the coding for this project with high inter-rater reliability, ICC = 0.950, 95% CI [0.910, 0.972], p < 0.001. From these codes, a social interaction frequency score was computed.

Procedure

After providing informed consent, participants completed clinician-rated assessments of social functioning. Next, participants were given an iPod Touch with the EAR application and told to wear it in a protective carrying case attached to the outside of their clothing (e.g., belt, shirt pocket, pants pocket) over the next two days. The iPod Touch was locked and password protected so participants could not use the device for other purposes. The EAR application passively recorded audio without participants’ input. Participants were also provided a blank social journal, instructed to keep it with them, and complete one journal entry during each waking hour over the two-day period. Participants were offered two-way transportation and received $10/hour for the research session. They were also paid $50 after completing EAR and EMA procedures. Study procedures were carried out in accordance with the latest version of the Declaration of Helsinki and approved by the Institutional Review Board at Indiana University.

Analyses

Correlations were used to determine convergence between social functioning indices across the four assessment methods (two clinician-rated scales, EMA, and EAR ambulatory assessment). These analyses were run in two parts. First, we conducted correlation analyses across the full sample to test overall convergent validity. Next, we ran correlations within the schizophrenia group and the control group in order to determine group differences in the convergent validity of these measures.

Results

Descriptive Analyses

The final sample (N=69) included 36 people with schizophrenia and 33 healthy control participants. Descriptive statistics revealed no group differences in age, sex, ethnicity, nor the day of the week of real-world assessment (weekday versus weekend); groups significantly differed in race, whereby the schizophrenia group was predominantly Black/African American, and the control group was mostly White (see Table 2). Table 2 outlines social functioning data across groups. EAR audio files were excluded from analyses if the subject was sleeping, if the file was inaudible, or if the subject was not wearing the EAR. Groups did not differ on number of excluded audio files, t(67)=1.46, p=0.149. Shapiro-Wilk tests revealed the GFS-S and QLS-IR yielded non-normally distributed data; thus, Spearman Rho correlations were used.

Table 2.

Demographic and Social Functioning Data in Schizophrenia and Control Groups

Schizophrenia Group (n = 36) Control Group (n = 33)
M (SD) M (SD) Test Statistic p
Age 46.06 (10.58) 42.97 (12.24) t = 1.12 0.27
Sex: % Female 55.6 45.5 χ2 = 0.70 0.40
Race: χ2 = 6.29 0.04
% Black/African American 63.9 36.4
% White 30.6 60.6
% Multi-Racial 5.6 3.0
Ethnicity: % Non-Hispanic 86.1 93.9 χ2 = 121 0.55
Day of Week of EMA: % Weekday 38.9 42.4 χ2 = 2.27 0.32
Social Functioning Data
GFS-S 5.94 (1.22) 8.39 (1.12) t = −8.69 0.00
QLS-IR 2.97 (1.08) 5.17 (0.84) t = −9.37 0.00
EMA 43.70 (31.26) 46.97 (21.86) t = −0.50 0.62
EAR 42.46 (27.25) 40.48 (23.59) t = 0.32 0.75
Open in a new tab

Notes: n = number; M = mean; SD = standard deviation; GFS-S = Global Functioning Scale – Social; QLS-IR = Quality of Life Scale: Interpersonal Relations Subscale; EMA = Ecological Momentary Assessment; EAR = Electronically Activated Recorder

Correlations Across Sample

First, correlations tested associations between laboratory and real-world measures of social functioning across the full sample. Results revealed a significant, medium correlation between EMA and EAR measures, r(69)=0.36, p=0.003. However, neither the EMA nor EAR measure was significantly correlated with either of the clinician-rated laboratory measures. The clinician-rated GFS-S and QLS-IR measures were highly correlated, r(69)=0.83, p<0.001 (see Table 3).

Table 3.

Spearman’s Rho Correlation Analyses Across Full Sample (N=69)

1. 2. 3. 4.
Laboratory Assessments
 1. GFS-S – 0.83** 0.19 0.19
 2. QLS-IR 0.83** – 0.17 0.17
Real-world Assessments
 3. EMA 0.19 0.17 – 0.36**
 4. EAR 0.19 0.17 0.36** –
Open in a new tab

Notes: GFS-S = Global Functioning Scale – Social; QLS-IR = Quality of Life Scale: Interpersonal Relations Subscale; EMA = Ecological momentary assessment; EAR = Electronically Activated Recorder

*

p < .05,

**

p < .01

Within-group Correlations

Next, correlations were tested within each group. In the schizophrenia group, the correlation between EMA and the EAR was not significant. The EAR exhibited moderate correlations with both laboratory assessments (GFS-S: r(36)=0.47, p=0.004, QLSIR: r(36)=0.38, p=0.021), but the EMA score did not (see Table 4). In the control group, EMA and the EAR showed moderate correlations, r(33)=0.44, p=0.011. Neither laboratory measure was associated with EMA or the EAR in the control group. Across both groups, significant correlations were observed between lab-based measures (see Table 4).

Table 4.

Spearman’s Rho Correlation Analyses Within Groups

Schizophrenia Group (N=36) Control Group (N=33)
1. 2. 3. 4. 1. 2. 3. 4.
Laboratory Assessments
1. GFS-S – 0.68** 0.17 0.47** – 0.62** 0.11 0.13
2. QLS-IR 0.68** – 0.15 0.38* 0.62** – 0.18 0.16
Real-world Assessments
3. EMA 0.17 0.15 – 0.27 0.11 0.18 – 0.44*
4. EAR 0.47** 0.38* 0.27 – 0.13 0.16 0.44* –
Open in a new tab

Notes: GFS-S = Global Functioning Scale – Social; QLS-IR = Quality of Life Scale: Interpersonal Relations Subscale; EMA = Ecological momentary assessment; EAR = Electronically Activated Recorder

*

p < .05,

**

p < .01

Discussion

This project investigated convergent validity among four social functioning measures: two clinician-rated laboratory assessments, a self-reported EMA, and a passive ambulatory ecological assessment. Two interesting findings emerged. First, in line with our hypotheses, the real-world assessments of social functioning were moderately convergent in the control group and across the full sample; yet EMA and EAR measures were not correlated in the schizophrenia group. Second, contrary to our hypothesis, laboratory measures of social functioning exhibited small associations with real-world assessments across the full sample. However, as expected, analyses in the schizophrenia group yielded moderate correlations between the lab-based measures and the EAR. Overall, convergent validity among the four social functioning assessments was lower than expected, suggesting these methods may evaluate different aspects of the construct.

EMA versus Ambulatory Ecological Assessment

This is one of the first studies to assess convergent validity between self-report EMA and ambulatory ecological assessment of social functioning in schizophrenia. Across the full sample, real-world measures converged with medium effect sizes. These results were maintained in the control group, but correlations between EMA and the EAR were not significant in those with schizophrenia. A potential explanation for this lower correlation is that social interactions in the schizophrenia group were outside of the 5-minute EAR interval and therefore not captured. However, because our results are consistent with those of Fulford et al. (2020), they may alternatively point to key differences in how social functioning is operationalized in EMA versus passive measures.

In the ambulatory ecological assessment, “social interactions” were defined as any time the participant spoke with another person; in the EMA, participants indicated if they were “interacting” with someone else. Thus, EMA requires participant interpretation. Participants may differ in what they consider an “interaction,” and some may include non-verbal exchanges that the EAR does not capture (e.g., text messaging, online chat). Differences in item interpretation is often a concern in self-report measurement (Furr, 2018). Moreover, the general validity of self-report in people with schizophrenia has been questioned (Bowie et al., 2007; Sabbag et al., 2012). Research shows those with schizophrenia over-estimate their functioning (Sabbag et al., 2012), and discrepancies between self-report and more objective forms of assessment (e.g., performance-based measures, clinician ratings) are theorized to vary as a function of cognitive impairment, symptom severity, and level of insight (Andorko et al., 2019; Ermel et al., 2017; Hasson-Ohayon et al., 2011; Patterson et al., 1997). Therefore, low associations between social functioning indices in our schizophrenia group may be due to a combination of measurement differences and lower insight and/or cognitive functioning in schizophrenia.

Laboratory Assessment versus Real-world Assessment

Across both groups, convergent validity between laboratory and real-world assessments was not supported. This may be explained by differences in the aspects of social functioning assessed within each measure: real-world measures assessed frequency of daily social interactions, whereas the laboratory measures assessed qualitative aspects of social functioning. However, in the schizophrenia group, both laboratory-based measures were significantly associated with the EAR but not EMA. Moreover, the EAR was more highly correlated with laboratory measures than it was with the EMA in the schizophrenia group. These results are counter-intuitive (i.e., real-world assessments are theoretically more similar) and may suggest the relationships between different methodologies are more complex.

Investigations of relationships between laboratory-based assessments and real-world measures of social functioning have yielded mixed results (Gard et al., 2014; Granholm et al., 2008; Schneider et al., 2017; Vasconcelos e Sa et al., 2016). Most studies have only compared laboratory assessments to EMA. Thus, these contrasting results may be due to inconsistencies between subjective and objective assessments of functioning. For instance, although Granholm et al.’s (2008) results supported the convergent validity between laboratory methods and EMA, this study used two forms of self-report (retrospective and momentary). Alternatively, Vasconcelos e Sa et al. (2016) compared momentary self-report to clinician-rated scales and found no association. Therefore, functioning as measured by EMA, while convergent with other forms of self-report, may not coincide with clinician-rated scales or more objective assessments. Our ambulatory ecological assessment may offer a less biased, momentary measure of verbal exchanges in schizophrenia that therefore corresponds more closely with objective laboratory assessments of social functioning like clinician-rated scales. This interpretation is supported by the results of Fulford et al. (2020) who found moderate associations between mobile sensing and laboratory-based measures of social functioning in those with schizophrenia.

It is noteworthy that laboratory-based assessments were associated with the ambulatory ecological assessment in the schizophrenia group, but not the control group. This is consistent with Schneider et al. (2017) who found a similar pattern when comparing momentary social functioning to a clinician-rated scale. It seems rating scales based on clinical interviews may lack sensitivity to detect minor differences in social behaviors, especially for those with better social functioning. In our study, GFS-S and QLS-IR scores in the control group were all high and showed little variability. Thus, although global rating scales may correctly capture variations in social functioning in those with schizophrenia, they yield restricted range and “ceiling effects” in healthy populations. By assessing social functioning at specific instances throughout one’s day, real-world assessments may capture nuanced differences.

Study Limitations and Future Directions

This study adds to the literature testing the convergent validity among different types of laboratory and real-world social functioning assessments. Moreover, it is one of the first studies to compare EMA and ambulatory ecological assessments of social functioning in schizophrenia. However, this research has limitations. First, our sample size makes power to detect true relationships a concern, and significant racial differences between groups limits our interpretation of group differences in convergent validity. Next, although the EAR offers an innovative method for passive data collection, participants may act differently in social situations because they are wearing a recording device. Though reactivity is a concern, research suggests participants habituate within a few hours of wearing the EAR (Mehl and Holleran, 2007) and ambient audio data can be collected from people with schizophrenia with relatively little interference in their daily lives (Abplanalp et al., 2021). We also alleviated this issue by ensuring participants were unaware of when they were recorded and allowing participants to approve recordings before our team accessed them. Lastly, the social journal is limited because it relies heavily on subject compliance. We assume participants record their behaviors throughout the day, but they may forget and complete the journal later, disqualifying it as a true in-the-moment assessment. Future studies should utilize mobile technology with time-stamped entries. However, it is still necessary to examine convergent validity with this type of EMA since paper diary methods are used in schizophrenia research (Myin-Germeys et al., 2003; Myin-Germeys et al., 2001; Thewissen et al., 2011).

Our results yield important implications regarding the clinical utility of different social functioning methods. Interview-based assessments are useful for understanding broad aspects of one’s social functioning (e.g., satisfaction with relationships, social competence). These methods may be most useful for intake interviews or in settings where clinicians need a “big-picture” view of client functioning. Real-world assessments are suitable for understanding more granular details about social functioning (e.g., how people interact, who they interact with, quality of daily interactions) and are favorable for tracking improvements in functioning over time. Specifically, EMA may be best for harnessing latent factors of social functioning (e.g., performance appraisals, emotional response), whereas EAR recordings may better track behavioral factors (e.g., time spent around others, number of arguments). In a clinical setting, real-world assessments could help patients understand their social lives and address problems in therapy.

More research comparing real-world and laboratory assessments of social functioning in this population is needed. Studies should test the convergent validity among these methods with larger samples. Because insight, symptom severity, and cognitive functioning may affect the validity of self-report in those with schizophrenia (Andorko et al., 2019; Ermel et al., 2017; Hasson-Ohayon et al., 2011; Patterson et al., 1997), researchers should investigate convergence between EMA and ambulatory ecological assessment across people of different symptom and functioning levels (e.g., inpatient, outpatient). Researchers should also examine how factors that systematically differ between schizophrenia and control groups (e.g., employment) may influence one’s types of social activities (e.g., online communication) and thus be differently reflected across real-world measures. Lastly, as ambulatory ecological assessment is less burdensome on participants than EMA, researchers should examine feasibility and compliance rates across these methods to determine relative clinical utility. Eventually, it will be vital to illustrate the incremental validity and clinical utility of ambulatory ecological assessments of social functioning. As technology advances, this work will determine if it proves useful to supplement, or even replace, traditional measures of social functioning.

Acknowledgements:

We wish to thank our research team, especially Beshaun Davis, Matthew Marggraf, and Kathryn Hardin, for their hard work completing interviews and collecting the data for this project. We also thank the participants who volunteered for this research.

Role of Funding Source

Funding:

This work was supported by the National Institutes of Health, National Center for Advancing Translational Sciences, Clinical and Translational Sciences Award (sub-award to K.S. Minor) [grant numbers KL2TR001106, UL1TR001108 PI: A. Shekhar]; and the American Psychological Foundation Pearson Early Career Grant (PI: K.S. Minor). This funding had no role in designing the study, data collection, data analysis, manuscript preparation, or the decision to submit the completed manuscript for publication.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Conflict of Interest: All authors declare no conflicts of interest in the current study.

References

  1. Abel DB, Salyers MP, Wu W, Monette MA, Minor KS, 2021. Quality versus quantity: determining real-world social functioning deficits in schizophrenia. Psychiatry Research. Advance online publication. 10.1016/j.psychres.2021.113980 [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Abplanalp SJ, Gold A, Gonzalez R, Doshi S, Campos-Mendez Y, Gard DE, Fulford D, 2021. Feasibility of using smartphones to capture speech during social interactions in schizophrenia. Schizophrenia Research, 228, 51–52. Advance online publication. 10.1016/j.schres.2020.12.007 [DOI] [PubMed] [Google Scholar]
  3. Andorko ND, Rakhshan-Rouhakhtar P, Hinkle C, Mittal VA, McAllister M, DeVylder J, Schiffman J, 2019. Assessing validity of retrospective recall of physical activity in individuals with psychosis-like experiences. Psychiatry Research, 273, 211–217. 10.1016/j.psychres.2019.01.029 [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Auther AM, Smith CW, Cornblatt BA, 2006. Global Functioning: Social Scale. Glen Oaks, NY: Zucker Hillside Hospital. [Google Scholar]
  5. Bellack AS, Green MF, Cook JA, Fenton W, Harvey PD, Heaton RK, Laughren T, Leon AC, Mayo DJ, Patrick DL, Patterson TL, Rose A, Stover E, Wykes T, 2007. Assessment of community functioning in people with schizophrenia and other severe mental illnesses: A white paper based on an NIMH-sponsored workshop. Schizophrenia Bulltein, 33, 3, 805–822. doi: 10.1093/schbul/sbl035 [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Ben-Zeev D, Scherer EA, Wang R, Xie H, Campbell AT, 2015. Next-generation paychiatric assessment: Using smartphone sensors to monitor behavior and mental health. Psychiatric Rehabilitation Journal, 38, 3, 218–226. 10.1037/prj0000130 [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Ben-Zeev D, Wang R, Abdullah S, Brian R, Scherer EA, Mistler LA, Hauser M, Kane JM, Campbell A, Choudhury T, 2016. Mobile behavioral sensing for outpatients and inpatients with schizophrenia. Psychiatric Services, 67, 558–561. doi: 10.1176/appi.ps.201500130 [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Birchwood M, Smith J, Cochrane R, Wetton S, Copestake S, 1990. The Social Functioning Scale. The development and validation of a new scale of social adjustment for use in family intervention programmes with schizophrenic patients. The British Journal of Psychiatry, 157, 853–859. doi: 10.1192/bjp.157.6.853 [DOI] [PubMed] [Google Scholar]
  9. Bowie CR, Twamley EW, Anderson H, Halpern B, Patterson TL, Harvey PD, 2007. Self-assessment of functional status in schizophrenia. Journal of Psychiatric Research, 41, 12, 1012–1018. 10.1016/j.jpsychires.2006.08.003 [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Buck B, Scherer E, Brian R, Wang R, Wang W, Campbell A, Choudhury T, Hauser M, Kane JM, Ben-Zeev D, 2019. Relationships between smartphone social behavior and relapse in schizophrenia: A preliminary report. Schizophrenia research, 208, 167–172. 10.1016/j.schres.2019.03.014 [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Burns T, Patrick D, 2007. Social functioning as an outcome measure in schizophrenia studies. Acta Psychiatr Scand, 116, 403–418. doi: 10.1111/j.1600-0447.2007.01108.x [DOI] [PubMed] [Google Scholar]
  12. Cornblatt BA, Auther AM, Niendam T, Smith CW, Zinberg J, Bearden CE, Cannon TD, 2007. Preliminary findings for two new measures of social and role functioning in the prodromal phase of schizophrenia. Schizophrenia Bulletin, 33, 688–702. doi: 10.1093/schbul/sbm029 [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Csikszentmihalyi M Larson R, 1987. Validity and reliability of the experience-sampling method. The Journal of Nervous and Mental Disease, 175, 9, 526–536. doi: 10.1097/00005053-198709000-00004 [DOI] [PubMed] [Google Scholar]
  14. Depp CA, Moore R,C, Perivoliotis D, Holden JL, Swendsen J, Granholm EL, 2016. Social behavior, interaction appraisals, and suicidal ideation in schizophrenia: The dangers of being alone. Schizophrenia Research, 172, 195–200. doi: 10.1016/j.schres.2016.02.028 [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Durand D, Strassnig M, Sabbag S, Gould F, Twamley EW, Patterson TL, Harvey PD, 2015. Factors influencing self-assessment of cognition and functioning in schizophrenia: Implications for treatment studies. European Neuropsychopharmacology, 25, 2, 185–191. 10.1016/j.euroneuro.2014.07.008 [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Ermel J, Carter CS, Gold JM, MacDonald AW 3rd, Daniel Ragland J, Silverstein SM, Strauss ME, Barch DM, 2017. Self versus informant reports on the specific levels of functioning scale: Relationships to depression and cognition in schizophrenia and schizoaffective disorder. Schizophrenia Research. Cognition, 9, 1–7. 10.1016/j.scog.2017.04.001 [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Fulford D, Mote J, Gonzalez R, Abplanalp S, Zhang Y, Luckenbaugh J, Onnela J, Busso C, Gard DE, 2020. Smartphone sensing of social interactions in people with and without schizophrenia. Journal of Psychiatric Research. Advance online publication. 10.1016/j.jpsychires.2020.11.002 [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Furr RM, 2018. Psychometrics: An introduction (3rd ed.). Sage, Thousand Oaks, CA. [Google Scholar]
  19. Gard DE, Sanchez AH, Cooper K, Fisher M, Garrett C, Vinogradov S, 2014. Do people with schizophrenia have difficulty anticipating pleasure, engaging in effortful behavior, or both? Journal of Abnormal Psychology, 123, 4, 771–782. 10.1037/abn0000005 [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Granholm E, Loh C, Swendsen J, 2008. Feasibility and validity of computerized ecological momentary assessment in schizophrenia. Schizophrenia Bulletin, 34, 3, 507–514. 10.1093/schbul/sbm113 [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Granholm E, Ben-Zeev, Fulford D, & Joel S (2013). Ecological momentary assessment of social functioning in schizophrenia: Impact of performance appraisals and affect on social interactions. Schizophrenia Research, 145, 120–124. doi: 10.1016/j.schres.2013.01.005 [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Gupta N, Mattoo SK, Basu D, Lobana A, 2000. Psychometric properties of quality of life (QLS) scale : a brief report. Indian Journal of Psychiatry, 42, 4, 415–420. [PMC free article] [PubMed] [Google Scholar]
  23. Hall RC, 1995. Global assessment of functioning. A modified scale. Psychosomatics, 36, 267–275. doi: 10.1016/S0033-3182(95)71666-8 [DOI] [PubMed] [Google Scholar]
  24. Hasson-Ohayon I, Roe D, Kravetz S, Levy-Frank I, Meir T, 2011. The relationship between consumer insight and provider-consumer agreement regarding consumer’s quality of life. Community Mental Health Journal, 47, 5, 607–612. 10.1007/s10597-011-9380-2 [DOI] [PubMed] [Google Scholar]
  25. Heinrichs DW, Hanlon TE, Carpenter WT Jr, 1984. The quality of life scale: an instrument for rating the schizophrenic deficit syndrome. Schizophrenia Bulletin, 10, 3, 388–398. 10.1093/schbul/10.3.388 [DOI] [PubMed] [Google Scholar]
  26. Hektner JM, Schmidt JA, Csikszentmihalyi M, 2007. Experience sampling method: Measuring the quality of everyday life. Thousand Oaks, CA: Sage Publications, Inc. [Google Scholar]
  27. Leendertse P, Myin-Germeys I, Lataster T, Simons CJP, Oorschot M, Lardinois M, Schneider M, van Os J, Reininghaus U, & For Genetic Risk and Outcome of Psychosis investigators. (2018). Subjective quality of life in psychosis: evidence for an association with real world functioning? Psychiatry Research, 26, 116–123. [DOI] [PubMed] [Google Scholar]
  28. Mehl MR, 2007. Eavesdropping on health: a naturalistic observation approach for social health research. Social and Personality Psychology Compass, 1, 359–380. doi: 10.1111/j.1751-9004.2007.00034.x [DOI] [Google Scholar]
  29. Mehl MR, 2017. The electronically activated recorder or EAR: a method for the naturalistic observation of daily social behavior. Current Directions in Psychological Science, 26, 184–190. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Mehl MR, Holleran SE, 2007. An empirical analysis of the obtrusiveness of and participants’ compliance with the electronically activated recorder (EAR). European Journal of Psychological Assessment, 23, 248–257. doi: 10.1027/1015-5759.23.4.248. [DOI] [Google Scholar]
  31. Mehl MR, Pennebaker JW, Crow DM, Dabbs J, Price JH, 2001. The electronically activated recorder (EAR): a device for sampling naturalistic daily activities and conversations. Behavior Research Methods, Instruments, & Computers, 33, 517–523. [DOI] [PubMed] [Google Scholar]
  32. Mehl MR, Robbins ML, 2012. Naturalistic observation sampling: The Electronically Activated Recorder (EAR), in: Mehl MR, Conner TS (Eds.), Handbook of research methods for studying daily life. Guilford Press, New York, pp. 176–192. [Google Scholar]
  33. Mehl MR, Robbins ML, Deters FG, 2012. Naturalistic observation of health-relevant social processes: the electronically activated recorder methodology in psychosomatics. Psychosomatic Medicine, 74, 410–417. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Mehl MR, Vazire S, Ramirez-Esparza N, Slatcher RB, Pennebaker JW, 2007. Are women really more talkative than men? Science, 317, 82. [DOI] [PubMed] [Google Scholar]
  35. Minor KS, Davis BJ, Marggraf MP, Luther L, Robbins ML, 2018. Words matter: Implementing the electronically activated recorder in schizotypy. Personality Disorders: Theory, Research, and Treatment, 9, 2, 133–143. 10.1037/per0000266 [DOI] [PubMed] [Google Scholar]
  36. Moitra E, Gaudiano BA, Davis CH, Ben-Zeev D, 2017. Feasibility and acceptability of post-hospitalization ecological momentary assessment in patients with psychotic-spectrum disorders. Comprehensive Psychiatry, 74, 204–213. 10.1016/j.comppsych.2017.01.018 [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Myin-Germeys I, Delespaul P, van Os J, 2003. The experience sampling method in psychosis research. Current Opinion in Psychiatry, 15, S33–S38. doi: 10.1097/00001504-200304002-00006 [DOI] [Google Scholar]
  38. Myin-Germeys I, van Os J, Schwartz JE, Stone AA, Delespaul PA, 2001. Emotional reactivity to daily life stress in psychosis. Arch Gen Psychiatry, 58, 1137–1144. doi: 10.1001/archpsyc.58.12.1137 [DOI] [PubMed] [Google Scholar]
  39. Oorschot M, Kwapil T, Delespaul P, Myin-Germeys I, 2009. Momentary assessment research in psychosis. Psychological Assessment, 21, 4, 498–505. 10.1037/a0017077 [DOI] [PubMed] [Google Scholar]
  40. Oorschot M, Lataster T, Thewissen V, Lardinois M, van Os J, Delespaul PA,E,G, & Myin-Germeys I (2012). Symptomatic remission in psychosis and real-life functioning. The British Journal of Psychiatry, 201, 3, 215–220. doi: 10.1192/bjp.bp.111.104414 [DOI] [PubMed] [Google Scholar]
  41. Patterson TL, Semple SJ, Shaw WS, Halpain M, Moscona S, Grant I, Jeste DV, 1997. Self-reported social functioning among older patients with schizophrenia. Schizophrenia Research, 27, 2–3, 199–210. 10.1016/S0920-9964(97)00078-9 [DOI] [PubMed] [Google Scholar]
  42. Pennebaker JW, Mehl MR, Niederhoffer KG, 2003. Psychological aspects of natural language use: Our words, our selves. Annual Review of Psychology, 54, 547–577. doi: 10.1146/annurev.psych.54.101601.145041 [DOI] [PubMed] [Google Scholar]
  43. Piskulic D, Addington J, Auther A, Cornblatt BA, 2011. Using the global functioning social and role scales in a first-episode sample. Early Intervention in Psychiatry, 5, 3, 219–223. 10.1111/j.1751-7893.2011.00263.x [DOI] [PubMed] [Google Scholar]
  44. Rintala A, Wampers M, Myin-Germeys I, Viechtbauer W, 2019. Response compliance and predictors thereof in studies using the experience sampling method. Psychological Assessment, 31, 2, 226–235. 10.1037/pas0000662 [DOI] [PubMed] [Google Scholar]
  45. Robbins ML, Focella ES, Kasle S, Lopez AM, Weihs KL, Mehl MR, 2011. Naturalistically observed swearing, emotional support, and depressive symptoms in women coping with illness. Health Psychology, 30, 789–792. doi: 10.1037/a0023431 [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Robbins ML, López AM, Weihs KL, Mehl MR, 2014. Cancer conversations in context: naturalistic observation of couples coping with breast cancer. Journal of Family Psychology, 28, 380–390. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Sabbag S, Twamley EW, Vella L, Heaton RK, Patterson TL, Harvey PD, 2012. Predictors of the accuracy of self assessment of everyday functioning in people with schizophrenia. Schizophrenia Research, 137, 1–3, 190–195. 10.1016/j.schres.2012.02.002 [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Schneider M, Reininghaus U, van Nierop M, Janssens M, Myin-Germeys I, GROUP Investigators, 2017. Does the Social Functioning Scale reflect real-life social functioning? An experience sampling study in patients with a non-affective psychotic disorder and healthy control individuals. Psychological Medicine, 47, 2777–2786. doi: 10.1017/S0033291717001295 [DOI] [PubMed] [Google Scholar]
  49. Sheehan DV, Lecrubier Y, Sheehan KH, Amorim P, Janavs J, Weiller E, Hergueta T, Baker R, Dunbar GC, 1998. The Mini-international neuropsychiatric interview (M.I.N.I.): The development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. J Clin Psychiatry. 59(Suppl 20):22–33. [PubMed] [Google Scholar]
  50. Skodol AE, Link BG, Shrout PE, Horwarth E, 1988. Toward construct validity for DSM-III Axis V. Psychiatry Research, 24, 13–23. doi: 10.1016/0165-1781(88)90135-7 [DOI] [PubMed] [Google Scholar]
  51. Startup M, Jackson MC, Bendix S, 2002. The concurrent validity of the Global Assessment of Functioning (GAF). The British Journal of Clinical Psychology, 41, 417–422. doi: 10.1348/014466502760387533 [DOI] [PubMed] [Google Scholar]
  52. Thewissen V, Bentall RP, Oorschot M,A Campo J, van Lierop T, van Os J, Myin Germeys I, 2011. Emotions, self-esteem, and paranoid episodes: an experience sampling study. The British Journal of Clinical Psychology, 50, 2, 178–195. 10.1348/014466510X508677 [DOI] [PubMed] [Google Scholar]
  53. Vasconcelos E Sa D, Wearden A, Hartley S, Emsley R, Barrowclough C, 2016. Expressed Emotion and behaviourally controlling interactions in the daily life of dyads experiencing psychosis. Psychiatry Research, 245, 406–413. 10.1016/j.psychres.2016.08.060 [DOI] [PubMed] [Google Scholar]

RESOURCES