Sentiment in Speech is Associated with Symptoms Severity in Psychosis

Abstract

Background:

Sentiment in the speech of people with schizophrenia spectrum disorder (SSD) may reflect psychosis severity. Previous research examines speech from semi-structured interviews or self-narrative prompts, where differences in measured sentiment may be driven by differences in life experiences. We measured sentiment in speech evoked from standardised stimuli among participants with a psychotic disorder.

Methods:

Two cohorts (N=97) participated in this study. Symptom domains were assessed using the Brief Psychiatric Rating Scale and were represented as Anxious Depression, Hostile Suspiciousness, Thought Disturbance, and Withdrawal Retardation. Participant speech during picture description tasks was quantified for sentiments: Valence, Arousal, Dominance, Happiness, Sadness, Anger, Fear, Disgust, and Surprise. Correlations between clinical and sentiment measures were conducted separately for the two cohorts and two timepoints in Cohort 1. Within-participant longitudinal relationships were examined with linear mixed models.

Results:

Several replicable relationships between sentiment and symptom severity were found: two replicable findings among Cohorts 1 and 2 and three replicable findings across Cohort 1 timepoints. Five findings were also generalised to within-participant longitudinal relationships as indicated by linear mixed models.

Conclusions:

Sentiment measures were related to the four symptom domains in the context of standardised stimuli, suggesting a disruption in emotion processing among people with a psychotic disorder.

Introduction

Previous studies have shown that symptoms severity in psychosis may be related to the sentiment of speech content in individuals with schizophrenia spectrum disorders (SSD). For example, increased anger word use during the Indiana Psychiatric Illness Interview and in speech collected using experience sampling method (ESM) has been correlated with lower functioning, depressed mood, and more profound positive symptoms (Minor et al., 2015; Vakhrusheva et al., 2020). Sentiment has also been related to measures of social cognition in psychosis, with arousal-evoking word usage during open-ended and picture-description tasks having shown to be negatively correlated with emotion processing and mentalizing ability (Tang et al., 2023a). However, in a group comparison between individuals with SSD and non-psychiatric control participants during a semi-structured interview, St-Hilaire et al. (2008) found that participant groups were similar in the amount and type of emotion words spoken. This suggests the need for replicating positive findings across different cohorts.

Thus far, automated detection of sentiment in the speech content of individuals with psychosis have primarily examined speech from semi-structured clinical interviews or self-narrative prompts (Cohen et al., 2009; Minor et al., 2015; St-Hilaire et al., 2008; Vakhrusheva et al., 2020). These open-ended and semi-structured speech tasks are a suitable method for evoking sufficient quantities of self-descriptive speech. However, different participants have different life experiences, so the content of the speech is not controlled for. Since the experience of psychosis is also associated with meeting greater adversity in day-to-day life, both past and present, this opens the possibility that the observed relationships between sentiment and psychosis symptoms are driven by differences in the life experiences which individuals are relating, rather than a specific difference in how emotion is being processed or expressed. To our knowledge, few studies have measured emotion expression among individuals with psychosis in the setting of structured stimuli where the content is standardised.

Objectives.

In this study, we explored relationships between sentiment and symptom dimensions associated with psychotic disorders in the context of standardised picture description stimuli. We use the term “emotion” to refer to one’s internal experience and “sentiment” to refer to linguistic expression of emotions in both speech and writing. We hypothesised that more severe symptomatology would be associated with decreased positive sentiment (positive Valence, Happiness) and increased negative sentiment (Sadness, Anger, Fear, Disgust). Our goal was to discover and report reproducible findings by comparing results across samples, across timepoints (test-retest), and across cross-sectional to longitudinal frameworks.

Methods

Participant Statistics.

Participants were recruited from inpatient and outpatient facilities at Zucker Hillside Hospital in Queens, New York (Table 1). Participants were evaluated with the Structured Interview for DSM-IV (American Psychiatric Association, 1994), and diagnoses were assigned by expert review (SXT) or consensus case conference based on criteria from the DSM-5 (American Psychiatric Association, 2013). All procedures were approved by the Institutional Review Board at the Feinstein Institutes for Medical Research at the hospital, and all participants provided informed consent. Diagnoses included schizophrenia, schizoaffective disorder, schizophreniform disorder, delusional disorder, bipolar with psychosis, and unspecified psychotic disorder. The proportions of diagnoses in the different cohorts are listed in Table 1. Two cohorts were used in this study. Cohort 1 consists of longitudinal data from two timepoints (Timepoint 1, at admission: n=68; Timepoint 2, at discharge: n=53). Cohort 2 includes an independent group of participants with psychosis. They participated at a convenient point in their treatment and were not selectively recruited at a particular point in their care (n=29). These cohorts or subsamples from these cohorts have been described using different natural language processing analyses previously (Nikzad et al., 2022; Tang et al., 2021; Tang et al., 2023a; Tang et al., 2023b).

Table 1.

Participant Characteristics. T1—Timepoint 1. T2—Timepoint 2. BPRS—Brief Psychiatric Rating Scale. Comorbid Diagnoses include Anxiety Disorders, Bipolar Disorders, Depressive Disorders, and Substance Use Disorders. Other Comorbid Diagnoses include Attention Deficit Hyperactivity Disorder, Avoidant/Restrictive Food Intake Disorder, Antisocial Personality Disorder, and Catatonia.

	Cohort 1 T1	Cohort 1 T2	Cohort 2
Sample
N	68	53	29
Age (mean years ± SD)	26.5 ± 5.2	26.0 ± 4.7	24.5 ± 4.2
Gender n (%)
Man	47 (69%)	37 (70%)	17 (59%)
Non-Binary	3 (4%)	3 (6%)	2 (7%)
Woman	15 (22%)	11 (21%)	10 (34%)
Unknown	3 (4%)	2 (4%)	0 (0%)
Race n (%)
Asian	12 (18%)	10 (19%)	3 (10%)
Black	30 (44%)	24 (45%)	11 (38%)
White	15 (22%)	10 (19%)	7 (24%)
Multiple	4 (6%)	3 (6%)	3 (10%)
Other	6 (9%)	5 (9%)	4 (14%)
Unknown	1 (1%)	1 (2%)	1 (3%)
Hispanic Ethnicity n (%)	10 (15%)	6 (11%)	3 (10%)
Education Level (mean years ± SD)	14.1 ± 1.9	13.9 ± 1.8	13.8 ± 2.1
Clinical Characteristics
Psychotic Disorder
Schizophrenia	34 (50%)	29 (55%)	15 (52%)
Schizoaffective	15 (22%)	13 (24.5%)	3 (10%)
Schizophreniform	4 (6%)	2 (4%)	2 (7%)
Bipolar I with Psychosis	4 (6%)	3 (5.5%)	1 (3%)
Delusional Disorder	0 (0%)	0 (0%)	1 (3%)
Unspecified Psychotic Disorder	11 (16%)	6 (11%)	7 (24%)
Comorbid Diagnoses (%)
Anxiety Disorders	5 (7%)	5 (9%)	4 (14%)
Bipolar Disorders	9 (13%)	7 (13%)	7 (24%)
Depressive Disorders	3 (4%)	3 (6%)	4 (14%)
Substance Use Disorders	31 (46%)	15 (28%)	12 (41%)
Other	8 (12%)	7 (13%)	1 (7%)
BPRS Anxious Depression (Mean ± SD)	8.7 ± 4.4	7.1 ± 4.0	6.2 ± 3.3
BPRS Hostile Suspiciousness (Mean ± SD)	8.8 ± 3.3	7.2 ± 3.4	6.3 ± 3.1
BPRS Thought Disturbance (Mean ± SD)	12.1 ± 3.5	9.4 ± 3.9	6.4 ± 3.9
BPRS Withdrawal Retardation (Mean ± SD)	5.6 ± 3.0	6.3 ± 3.4	6.2 ± 3.6
BPRS Total Score (Mean ± SD)	48.6 ± 11.0	42.5 ± 12.9	35.6 ± 11.1
WRAT Standard Score (Mean ± SD)	96.6 ± 13.9	99.9 ± 13.0	99.4 ± 11.8

Clinical and Language Assessments.

Symptom severity was rated by trained clinical assessors using the Brief Psychiatric Rating Scale (BPRS) (Overall & Gorham, 1962). It is a well-validated scale (Hofmann et al., 2022) covering psychosis-related domains that could contribute to emotion processing and expression. It provided an efficient approach to look at several important domains without creating many multiple comparisons. Symptom scores were calculated according to Shafer (2005). The four symptom domains include Anxious Depression, Hostile Suspiciousness, Thought Disturbance, and Withdrawal Retardation. Participant speech was recorded in response to picture-description tasks where participants were shown an image and asked to describe it in as much detail as possible. Within each cohort and timepoint, all participants received the same three stimuli, but the stimuli varied across cohorts and timepoints. Participants in Cohort 1 at both timepoints were asked to describe two scenes with multiple actions and characters. Additionally, they were asked to describe either an abstract Rorschach image (Timepoint 1) (Rorschach, 1921) or a social interaction depiction from a Thematic Apperception Test image (Timepoint 2) (Murray, 1943). Participants in Cohort 2 were asked to describe a scene with multiple actions and characters (Goodglass et al., 2001), an abstract Rorschach image, and a social interaction depiction from a TAT image. These assessments were administered by trained research coordinators, audio-recorded, and transcribed verbatim. The Wide Range Achievement Test Reading subtest was administered to rule out significant language impairment that may interfere with the study (Wilkinson & Robertson, 2006). All three cohorts scored within one standard deviation under the mean on average which does not indicate significant language impairment (Cohort 1 Timepoint 1 = 96.6 ± 13.9, Cohort 1 Timepoint 2 = 99.9 ± 13.0, Cohort 2 = 99.4 ± 11.8) (Mean=100, Standard Deviation=15) (Casaletto et al., 2014). Most participants scored greater than or equal to 70 besides two participants from Cohort 1 Timepoint 1 who scored 64 and 69.

Quantification of Sentiment.

Participants’ de-identified verbatim transcripts of the picture description tasks were processed using the library Natural Language Toolkit (NLTK) version 3.9.1. in Python (Bird et al., 2009). NLTK’s word tokenizer was used to tokenize transcripts into individual words, then each word was assigned sentiment values based on sentiment dictionaries. . No words were filtered out a priori, but the dictionaries generally did not return standardised ratings for function words (e.g., pronouns, conjunctions, determiners, auxiliaries, etc.). Valence (i.e., positive vs. negative), arousal (i.e., excited vs. calm), and dominance (i.e., in control vs controlled) were defined as continuous measures ranging from 1 to 9 based on published norms (Bradley & Lang, 1999; Warriner et al., 2013). Sentiment values in this dictionary were converted to z-scores prior to analysis. Participant scores were calculated by summing the sentiment score for each word spoken per timepoint and normalising it by the total number of words produced by the participant during that timepoint. Another set of sentiment features was calculated from dichotomous variables indicating whether each word was related to each of six sentiments: happiness, sadness, anger, fear, disgust, and surprise (Mohammad & Turney, 2013). Participant scores were normalised in the same manner as the previous dictionary. On average across the three groups, the dictionary for valence, arousal, and dominance returned standard ratings for 18.8% of the words spoken by participants when excluding articles and filler words., and the dictionary for happiness, sadness, anger, fear, disgust, and surprise returned 18.4% of the words spoken by participants when excluding articles and filler words.

Statistics.

All statistical analyses were completed in R version 4.3.1 (R Core Team, 2016). First, we explored correlations between sentiment and the four symptom domains at baseline for Cohort 1 to identify candidates for potential relationships. Cohort 1 was used to find candidate relations because it has more participants than Cohort 2.

Additionally, they were used instead of a multivariate correlation method because of the preliminary nature of this work. Here, one-dimensional correlations were computed between sentiment and psychopathology symptoms without accounting for interactions between sentiment dimensions. Correlation p-values were generated using the Hmisc package version 5.1.0 in R, and the correlation plot was generated using the corrplot package version 0.92 (Harrell, 2023; Wei & Simko, 2021). Correlations with trend-level p<0.10 were considered candidates for further validation. Post-hoc power analysis was conducted for Spearman’s correlations from Cohort 1 Timepoint 1 to assess the adequacy of our sample size. Second, to establish reproducibility, we attempted to replicate candidate relationships in Cohort 1 at Timepoint 2 and in Cohort 2.

Finally, we examined whether candidate findings for between-person relationships between sentiment and the five symptom domains would also be reflected in within-person longitudinal relationships between sentiment and these domains using random-intercept linear mixed models (LMM). This LMM type was chosen to determine one-dimensional relations between sentiment and clinical symptoms as opposed to accounting for multivariate effects. Additionally, this type matches our implicit assumption that the outcome variable changes over time, and individuals within the cohorts may have different baseline symptoms severity. The models were generated using the nlme package version 3.1.162 in R and were fitted using restricted maximum likelihood (REML) (Pinheiro et al., 2023). While maximum likelihood estimation (MLE) is available as an option, REML is generally preferred because it produces less biased estimates of variance components (Pal & Chakravarty, 2019). Data used in each model was normalised using Yeo-Johnson transformation to meet the model’s normality assumptions (Yeo & Johnson, 2000). This was done using the bestNormalize package version 1.9.1 (Peterson, 2021). We performed visual inspection of the residuals after Yeo-Johnson transformation. Based on the sample sizes of the analysis, the degree of normality was determined to be acceptable for the LMM approach. The models included data from Cohort 1 at Timepoint 1 and Timepoint 2, where each timepoint was a separate observation. The fixed effect for the linear mixed model was the sentiment feature, the dependent variable was the symptom domain from the BPRS, and the random effect was the participant ID. A significant finding implies that, accounting for different starting points for any given participant, there is a consistent relationship between how symptoms and sentiment change over time – e.g., when there is a change in Withdrawal Retardation symptoms, there is also a change in the amount of Anger expressed in the transcript, in the same direction as the change in symptoms.

Results

Relationships Between Sentiment and Four Symptom Domains in Cohort 1.

There were five statistically significant associations identified as candidates for the relationship between sentiment expressed in the picture description tasks and the symptom domains in Cohort 1 at Timepoint 1 (Figure 1, Table 2). They include correlations between Anxious Depression and Anger (𝜌=0.30; p=0.01); Hostile Suspiciousness and Anger (𝜌=0.29; p=0.02) as well as Surprise (𝜌=0.26; p=0.03); and Thought Disturbance and Happiness (𝜌=−0.43; p<0.001) as well as Disgust (𝜌=−0.28; p=0.02). In addition, 8 relationships were trend-level (p<0.10) and were included along with the significant relationships as candidates for replication. Post-hoc power analysis resulted in two correlations above the conventional threshold of 0.80, Happiness and Thought Disturbance (1-β=0.86) as well as Anger and Withdrawal Retardation (1-β=0.93).

Figure 1.

Spearman’s Correlations of Clinical and Sentiment Measures Among Participants with Schizophrenia Spectrum Disorder in Cohort 1 at Timepoint 1. Heatmap representations of Spearman’s correlation coefficients for clinical and sentiment measures. Blue indicates a positive correlation, and orange indicates a negative correlation. ⁺p<0.10; *p<0.05; **p<0.01; ***p<0.001.

Table 2.

Replicating Spearman’s correlations of clinical and sentiment measures from Cohort 1 Timepoint 1 in Cohort 1 Timepoint 2 and Cohort 2.

BPRS Symptom Domain	Sentiment	Cohort 1 T1 (Candidate Relationships)	Cohort 1 T2 (Replication across Timepoints)	Cohort 2 (Replication across samples)
Anxious Depression	Valence	𝜌=−0.23+	𝜌=−0.24+	𝜌=0.05
	Arousal	𝜌=0.23+	𝜌=0.13	𝜌=0.13
	Dominance	𝜌=−0.22+	𝛒=−0.30 *	𝜌=0.06
	Happiness	𝜌=−0.21+	𝜌=0.14	𝜌=0.21
	Anger	𝛒=0.30 *	𝜌=0.10	𝜌=0.07
Hostile Suspiciousness	Dominance	𝜌=−0.20+	𝜌=−0.16	𝜌=0.04
	Anger	𝛒=0.29 *	𝜌=0.07	𝜌=0.31+
	Surprise	𝛒=0.26 *	𝛒=0.45 ***	𝜌=0.14
Thought Disturbance	Dominance	𝜌=−0.22+	𝜌=−0.04	𝜌=−0.06
	Happiness	𝛒=−0.43 ***	𝜌=0.17	𝛒=−0.38*
	Disgust	𝛒=−0.28 *	𝜌=−0.18	𝜌=0.20
Withdrawal Retardation	Happiness	𝜌=−0.24+	𝜌=0.07	𝜌=−0.19
	Anger	𝜌=0.23+	𝜌=0.02	𝜌=−0.08

Correlations with p≥0.10 are shaded. Significant findings are bolded. T1—Timepoint 1. T2—Timepoint 2.

⁺p<0.10;

^*p<0.05;

^**p<0.01;

^***p<0.001.

Replication in an Independent Sample.

Of the candidate relationships, the finding that participants with high Thought Disturbance produced fewer words denoting Happiness was replicated in Cohort 2 (𝜌=−0.38; p=0.04). Additionally, Hostile Suspiciousness and Anger sentiment were significantly positively correlated in Cohort 1 and had a trend-level correlation in Cohort 2 (𝜌=0.31; p=0.10). The remaining candidate correlations in Cohort 1 were not replicated in Cohort 2 (Table 2).

Replication across Timepoints.

Among the candidate relationships from Cohort 1 at Timepoint 1, two were replicated as significant relationships (p<0.05) and one was partially replicated as a trend-level relationship (p<0.10) in Timepoint 2. Effect sizes were of the same direction and similar degree, including Anxious Depression and Dominance (𝜌=−0.30; p=0.03), Hostile Suspiciousness and Surprise (𝜌=0.45; p<0.001), as well as Anxious Depression and Valence (𝜌=−0.24; p=0.08). Multiple linear regressions for the replicated relationships demonstrated limited effects of verbal ability on the interaction of sentiment and symptom domains. Verbal ability had a significant effect on the interaction of Anxious Depression and Dominance in Cohort 1 Timepoint 2 (Est.=0.81, p=0.04). It had no significant effect on the other interactions of sentiment and symptom domains.

Within-Participant Longitudinal Relationships.

The significant between-participant relationships between sentiment and the symptom domains in Cohort 1 were evaluated longitudinally as within-individual relationships (Table 3). Here, we looked for fixed effects of sentiment predicting psychiatric symptoms across 2 timepoints in a random intercept linear mixed model. Five of the candidate relationships were reflected in within-individual longitudinal relationships. Anxious Depression with (positive) Valence (Coeff.=−1.1, p=0.02), Dominance (Coeff.=−0.6, p=0.01), and Happiness (Coeff.=−1.5, p=0.03); Thought Disturbance and Happiness (Coeff.=−16.1, p<0.01); and Withdrawal Retardation and Anger (Coeff.=3.7, p=0.04).

Table 3.

Linear Mixed Models: The Effects of Sentiment Scores on BPRS Symptom Domain Scores. Only significant or trend-level combinations of sentiment features and BPRS domain scores from Spearman’s correlation analyses in Cohort 1 at Timepoint 1 were tested. Data used in each model was normalised using Yeo-Johnson transformation. Significant findings are bolded. BPRS—Brief Psychiatric Rating Scale.

	BPRS Anxious Depression			BPRS Hostile Suspiciousness			BPRS Thought Disturbance			BPRS Withdrawal Retardation
	Estimate	SE	p	Estimate	SE	p	Estimate	SE	p	Estimate	SE	p
Valence	−1.1	4.2	0.02
Arousal	0.1	0.1	0.45
Dominance	−0.6	0.1	0.01	−0.8	0.5	0.10	−0.9	0.65	0.15
Happiness	−1.5	0.6	0.03				−16.1	4.3	<0.01	0.1	0.9	0.93
Sadness
Anger	1.9	1.4	0.20	7.3	7.1	0.30				3.7	1.7	0.04
Fear
Disgust							−19.5	11.1	0.08
Surprise				0.2	4.7	0.96

Discussion

We explored relationships between sentiment expressed in a picture description task and symptoms severity in multiple domains. There are some consistent, replicable findings. Specifically, Thought Disturbance and Happiness sentiment were significantly negatively correlated among both cohorts and within-person longitudinal analysis, suggesting an effect between sentiment and symptom severity that could be replicated over time and across samples. Also, Hostile Suspiciousness and Anger sentiment were significantly positively correlated in Cohort 1 at Timepoint 1 and had a trend-level positive correlation in Cohort 2. Together, these results support the hypothesis that relatively positive emotion dimensions would be negatively correlated with symptom severity, and relatively negative dimensions would be positively correlated with symptom severity. Post-hoc power analysis conducted on Spearman’s correlations of data from Cohort 1 Timepoint 1 resulted in two correlations above the conventional threshold for noteworthy associations. More meaningful relationships may have been found with a greater sample size. None of the relationships were significant across samples and timepoints in the Spearman’s correlation analyses. However, some of the effect sizes were similar in direction and degree for significant relationships. The lack of significance could be attributed to the smaller sample sizes; the relationships would likely be more stably represented in larger samples. Additionally, participants of Cohort 1 at Timepoint 2 were stabilised for discharge, so their symptom severity may have been reduced, which may have affected the correlation strength and significance due to decreased variance in clinical symptoms severity. Another reason for the lack of replication could be that participants in different cohorts and timepoints responded to different (although similar) stimuli, possibly causing inconsistencies when comparing groups; there are few studies which systematically compare the same features derived from different tasks.

The fact that we found some replicable relationships between sentiment and the four symptom domains even in the context where all participants are responding to the same stimulus implies that observed differences in sentiment may reflect fundamental changes in emotion processing in SSD. Sentiment expressed in speech production may be influenced by multiple levels of emotional processing: perception, experience, and expression. Accurate emotion perception seems to be impaired in SSD (Ito et al., 2013; Van den Stock et al., 2011) and correlated with SSD-related symptoms as well as a lower quality of life (Poole et al., 2000). Although there is evidence that the experience of emotions is not impacted in schizophrenia (Kring & Elis, 2013; Mote et al., 2014), impairments in emotion expression have been documented (Aghevli et al., 2003; Kring & Moran, 2008; Mote et al., 2014). These differences may be more disrupted when psychosis psychopathology is more severe. Emotion processing may also be impacted by premorbid functioning, premorbid IQ, and duration of untreated psychosis, so further studies should account for these variables when determining correlations between speech sentiment and psychiatric symptoms. The Wide Range Achievement Test Reading subtest was administered to screen for significant language impairment, and significant language impairment was not found: only two participants scored below 70, specifically from Cohort 1 Timepoint 1 with scores of 64 and 69.

A limitation of this study is that the findings are unable to distinguish among changes in emotion perception, experience, or expression. That is, the participants may describe the same pictures with different sentiments because they perceive the emotional content of the stimuli differently, the feelings evoked by the stimuli are experienced differently, the same internal experience of emotion is expressed differently in speech, or some combination of these factors. Additionally, this study had relatively small sample sizes (Cohort 1 Timepoint 1, n=68; Cohort 1 Timepoint 2, n=53; Cohort 2, n=29). We attempted to isolate reproducible findings by replicating across different cohorts, but this approach may miss smaller effect sizes.

We also assumed that speech from semi-structured interviews and self-narrative prompts is guided by life experiences, and therefore it does not provide a standardised approach to obtaining sentiment features or reliably give insight into one’s speech encoding and processing. Further studies can evaluate these claims by comparing the speech outputs of these stimuli to those of picture-description stimuli.

Moreover, our method of extracting sentiment features does not account for negations or exclamations, so it might miss relevant context when obtaining sentiment scores. Although it may miss context, the method was chosen over other approaches. VADER (Hutto & Gilbert, 2014) stratifies sentiments into only three categories: positive, negative, and neutral sentiment, and LIWC (Boyd et al., 2022) lacks categories such as surprise or disgust. These more advanced methods of calculating sentiment may be used in future studies alongside the method used in this study.

In summary, we found evidence that there are relationships between psychiatric symptom dimensions and automated, objective measures of sentiment even in the context of fixed stimuli. This could be related to differences in multiple levels of emotion processing. Results obtained here contribute to the ongoing search for linguistic biomarkers of psychosis with the goal of developing automated detection methods. Computational analysis may provide one scalable method to detect these important areas of impairment in SSD.

Data availability statement

The data that support the findings of this study are available from the corresponding author, SXT, upon reasonable request.