The Language of Paranoia: Linguistic Analysis of SMI Speech with Considerations of Race and Sex

Abstract

Background:

Linguistic analysis, notably using conceptually derived linguistic categories, has been used to quantify various aspects of serious mental illness. It has the potential for understanding paranoia, defined in terms of perceived and intentional threats from others. However, paranoia and the language expressing it potentially varies due to demographic factors, notably race and sex.

Aims:

This study aims to expand upon prior findings linking linguistic expression and serious mental illness symptoms by focusing on paranoia and evaluating potential moderating roles of race and sex in two archived studies using two separate speaking tasks.

Methods:

We hypothesized that a limited feature set of linguistic categories derived from these speaking tasks would accurately classify clinical ratings of paranoia using regularized regression. It was further hypothesized that these relationships would vary as a function of Black versus White and male versus female identities.

Results:

Unexpectedly, there were no differences in model accuracy as a function of race and sex, suggesting no overt bias or differential functioning from demographics in our models.

Conclusions:

Results highlight the strengths and limitations of using linguistic analysis to understand paranoia. Exploring variation amongst paranoia scoring could improve model accuracy across different demographic groups.

1.0. Introduction

Paranoia, involving self-relevant persecution, threat, or conspiracy beliefs that affect functioning, is a trans-diagnostic and pernicious symptom of serious mental illnesses (SMI; Oltmanns & Okada, 2006; Bebbington & Freeman, 2017). Paranoia negatively impacts the social welfare, social support systems, and financial stability of those diagnosed (Marson et al., 2006). “Gold standard” measures of paranoia rely on clinical ratings based on interviews and self-report scales (Aboraya et al., 2013; Statham et al., 2019). While beneficial, there are potential sex and racial biases associated with these measures. Of note, Black participants have shown more severe clinical ratings of paranoia and have been found 2.4 times more likely to be diagnosed with schizophrenia in comparison to White participants, regardless of the race of the clinician interviewing them (Olbert et al., 2018; Schwartz &Blankenship, 2014). There is some evidence of sex differences in positive symptoms, though the results are inconsistent across studies (Riecher-Rössler et al., 2018). While there are reasons minoritized individuals might experience paranoia, particularly during a clinical interview, it is unlikely that these differences in symptoms and diagnoses reflect true pathophysiological differences. Pinkham et al. (2016) found that groups varied on bias assessment tasks and not in their functional capacity, for example. Prior studies also call for consideration of broader contextual factors that would influence minoritized individuals to exhibit paranoid symptoms, notably environmental and historical factors (Vargas et al., 2020 & Wolny et al., 2023). For these reasons, there is a pressing need to develop culturally sensitive measures of paranoia. This present study uses linguistic analysis approaches to understand potential biases in clinical ratings of paranoia while moderating for race and sex.

Linguistic analysis, which involves inferring meaning from the systematic objectifying of various language properties, can provide a relatively standardized and automated approach complementary to clinical interviews (Tausczik et al., 2010). Because of its algorithmic nature, it is potentially more transparent than human judgment (Hitczenko et al., 2022). Linguistic analysis approaches to understand psychosis more generally have been well developed (e.g., successfully using language markers to detect psychosis; Liddy, 2001; Corcoran et al., 2020) and have focused on a variety of aspects of language, including semantics and syntactical analysis. Clinical interviews, for example, are a rich source of information about patient health and are typically qualitatively assessed on a case-by-case basis by a clinician. Linguistic analysis provides the ability to detect irregularities that span time (e.g., over multiple sessions) that a clinician may not be sensitive to. Linguistic analysis can also help explicate and overcome biases that busy clinicians may explicitly or implicitly apply while executing their duties. Emerging evidence supports the notion that linguistic analysis can be used to understand paranoia using linguistic (a.k.a, 1-gram, or “word soup”) analysis. Missing from these studies is a consideration of race and sex, an issue in that linguistic expression varies as a function of race and sex, more broadly speaking (Hitczenko et al., 2022).

This paper evaluates the speech of Black and White Americans diagnosed with SMI across two archived language studies. Our aim in this study is to see if a) clinical ratings of paranoia can be predicted by linguistic analysis software and b) if sex and race biases are evident in these models.

2.0. Methods

2.1. Participants.

Data were analyzed from two archived language studies, procured from (a) 88 individuals with SMI engaged in clinical interviews (k = 168 samples) and (b) 117 individuals with SMI engaged in 90-second autobiographical monologues (k = 517 samples). The studies received ethical approval from the Louisiana State University Institutional Review Board (IRB #2679 and IRB #3618). All participants involved in these studies give their consent to be included. The data has been explored elsewhere (Cohen et al., 2019; Cohen et al., 2014; Cowan et al., 2022), though this is the first study examining linguistic analysis and paranoia from these data collections. Approximately two-thirds of the samples were from men (k = 432) and one-third from female participants (k = 253) and were roughly even in their split between White (k = 343) and Black (k = 327) racial groups. Participants not fitting male/female or Black/White demographics were excluded from the analyses of sex or race as there was insufficient representation for statistical analyses (n = 3). Participants met U.S. federal definitions of SMI (ADAMHA, 1992). Lifetime diagnoses, determined by clinical interviews (see below), included schizophrenia (n = 122), major depressive disorder (n = 40), bipolar one disorder (n = 31), psychosis not otherwise specified (n = 2), and other SMIs (n = 10). Participants were free from major medical or other neurological disorders that would be expected to impair compliance with the research protocol. Though the participants endorsed substance use, no participant endorsed clinically relevant substance use per the Alcohol Use Disorders Identification Test /Drug Use Disorders Identification Test (AUDIT/DUDIT) scores (Berman et al., 2005; Bush, 1998).

2.2. Clinical measures.

Diagnoses were determined using structured clinical interviews (e.g., Structured Clinical Interview for DSM–IV–TR; First & Pincus, 2002) conducted by doctoral students under the supervision of a licensed psychologist (A. S. Cohen). Interviewers were evenly split between male and female and were blind to the purpose of this study. Most interviewers were White, and none were Black. Clinically-rated paranoia, our primary dependent variable, was measured using the “Suspiciousness” item from the Brief Psychiatric Rating Scale (BPRS; Overall & Gorham, (1988). See supplemental Figure 2 for a histogram of scores. The overall level of paranoia in the studies was mild, with 15.55 % of participants in studies 1 and 2 showing moderate or greater clinically-rated paranoia. Diagnoses and symptom ratings reflected consensus from the respective research teams.

2.3. Speaking Tasks

The language was evaluated from two separate speaking tasks. The first involved analyzing the first and last five minutes of the semi-structured clinical interviews with participants (i.e., two samples per participant), while the second involved 90-second “autobiographical monologues” on positive, negative, and neutral-valanced memories (i.e., three samples per participant). The first task was conducted in a “conversational” format, while the second was conducted with the participant watching a computer screen with the interviewer silent. We examined the first five minutes of speech from the clinical interviews to help standardize the questions people were asked (given that questions vary considerably across participants) and the last five minutes to control for changes that might occur throughout the interview. Moreover, the SCID interview comprises yes-no questions administered in a checklist manner with limited linguistic information being conveyed. The beginning of the SCID features more open-ended questions. All language was hand-transcribed by trained research assistants. The interviewer’s language was omitted from the analyses.

2.4. Language analysis

Linguistic analysis was performed using the Linguistic Inquiry and Word Count software (LIWC: Boyd et al., 2022). The LIWC program performs keyword searches based on predefined “dictionaries” of psychologically relevant word stems. These word stems are organized into categories. LIWC analysis yields a frequency count of the total instances of target words from each category. These numbers are then divided by the total number of words in the text to control for verbosity. The LIWC software has previously been used in SMI studies (Buck & Penn, 2015; Fineberg, 2018). For the present study, we focused on categories with potential semantic relationships with paranoia. LIWC categories were selected based on conceptual overlap with paranoia, including those from the pronoun, emotion, social, and drive categories (k = 36; see supplemental features). Previous research has found significance in evaluating self-referential and negatively valanced speech when categorizing paranoia, with participants often speaking definitively about their interpretations of ambiguous scenarios (Finigstein et al., 1992; Finn et al., 2018; Cohen et al., 2022). This broad categorization of paranoia encouraged the current study to various categories to encompass paranoia while focusing on self-referential and other categories that may be relevant (i.e., certainty vs. tentative).

2.5. Analyses

Analyses were conducted in three steps. First, we evaluated the participants’ descriptive, demographic, and clinical characteristics. Second, we conducted regularized regression (see next paragraph for description) using linguistic features to predict paranoia ratings. These regressions were conducted with paranoia binarized to address the ordinal nature of the scaling and non-normal distribution of paranoia ratings (see section 3.1). Binarization reflected low and high severity based on scores of moderate or greater severity (i.e., four or greater; see Table 1). Third, we evaluated whether models differed as a function of race and sex and whether they differed in relation to clinically-rated paranoia as a function of race and sex. Analyses were conducted using R Statistical Software (version 4.2.2; R Core Team, 2022) using base, psych (version 2.2.9; Revelle, 2022), and lme4 (version 1.1.31; Bates et al., 2015), and stabs (version 0.6.4; Hofner & Hothorn, 2021; Hofner et al., 2015) packages. All linguistic features scores were standardized and trimmed (i.e., values exceeding 3.50 standard deviations were replaced with a value of 3.50). If consequent skew scores were below 2, they were normalized using the R bestNormalize package (version 1.9.1; Peterson, 2021; Peterson & Cavanaugh, 2020), and binarized (zero/nonzero) the skew value remained above 2.

Table 1. Top 20 features from overall models of linguistic features predicting paranoia.

1^st person combined (singular and plural) was created to reduced feature number

Clinical Interview		Autobiographical Monologue
Feature	Weight	Feature	Weight
Personal pronouns	−5.32	Cognitive processes	−1.2
1st person singular	4.66	Negative emotion	0.77
3rd person singular	2.38	1st person plural	−0.67
Positive emotion	−2	3rd person singular	0.59
3rd person plural	1.97	Affiliation	0.51
Emotional tone	1.69	1st person combined	0.46
Cognitive processes	−1.44	Insight	0.43
Present focus	1.44	Past focus	−0.37
Sadness	1.31	Drives	0.36
2nd person	1.23	Social processes	−0.36
Future focus	−1.02	Emotional tone	0.35
1st person combined	0.9	Differentiation	0.34
Risk	−0.8	2nd person	0.3
Past Focus	−0.61	Informal language	0.24
Social processes	−0.59	Power	−0.22
1st person plural	−0.58	Tentative	0.22
Certitude	0.56	Anxiety	−0.19
Informal language	0.54	Anger	−0.14
Ingestion	0.51	1st person singular	0.13
Family	0.48	Interrogatives	0.12

2.6. Regularized Regression

We used the GLMNet (version 4.1–8; Friedman et al., 2010) implementations of Least Absolute Shrinkage and Selection Operator (LASSO) regularized regression, which allows for feature selection and evaluation using a large, potentially inter-correlated set of features. LASSO encourages sparse models (i.e., models with many feature weights set to zeros) by penalizing the model proportional to the sum of the magnitudes of the weights (Tibshirani, 1996). LASSO is well-suited for modeling datasets with potential multi-collinearity. LASSO performs feature selection as part of the model optimization procedure so the identified features will be maximized. This improves the identification of linguistic features most relevant to paranoia. We employed 10-fold cross-validation (with an 80% training and 20% test case split). A model was fit to the training set and evaluated on the test set. This was repeated so that each of the ten subsamples was used as the test set. Statistics are reported on an average of 10-fold, as well as the full model (without cross-validation). Statistics include hit rate, correct rejection, and accuracy, as the sum of the hit rate and correct rejection rate is divided by 2 so that 0.5 corresponds to random performance. Models were built with all participants’ data and then compared as a function of sex and race.

Stability selection (using stabs; version 0.6.4; Hofner & Hothorn, 2021; Hofner et al., 2015) was used to identify linguistic features significant in predicting paranoia. Stability selection is a subsampling procedure that resembles bootstrapping (Meinshausen & Bühlmann, 2010; Shah & Samworth, 2013). By training on thousands of random subsets of cases, we can estimate feature probabilities given different regularization amounts. Sets of features identified through stability selection are reliably important over many sets. The inclusion threshold is set to control the familywise error rate.

2.7. Patient and Public Involvement

Given that this study relied on archived data, patients or members of the public did not participate in the design or analysis of the current research. However, their lived experiences, as captured in the original interviews and monologues, provided the foundation for this analysis.

3.0. Results

3.1. Data Inspection

Participants, on average, produced 292 words in study 1 and 531 words in study 2. There were few differences in linguistic features as a function of demographic groups (see supplemental materials). Only a single feature in a single task (of 144 comparisons) exceeded a medium effect size (i.e., Cohen’s D value > 0.50). For the monologue task, men (M = 2.00, SD = 1.28) and women (M = 2.52, SD = 1.53) showed small differences in scores (t = 1.87, p < 0.10, d = 0.37) that were nonsignificant. For the clinical interview task, men (M = 1.98, SD = 1.17) and women (M = 2.58, SD = 1.54) showed similarly small differences in paranoia scores (t = 1.88, p < 0.10, d = 0.44). For the monologue task, White (M = 2.14, SD = 1.38) and Black participants (M = 2.25, SD = 1.42) showed no significant difference in scores (t = 0.45, p > 0.10, d = 0.08) which was paralleled in the clinical interview task where White (M = 2.02, SD = 1.21) and Black (M = 2.34, SD = 1.43) participants showed no significant difference (t = 1.12, p > 0.10, d = 0.24). When comparing the paranoia scores between the two groups, men and women showed no significant difference (17% versus 23%, χ² (1, n = 114) = 0.61, p = 0.43, w = 0.07) during the monologue task. This similarity was also found during the clinical interview task (22.6% versus 14%, χ² (1, n = 88) = 0.52, p = 0.47, w = 0.08). For the monologue task, White and Black participants were not statistically different in paranoia scores (18.2% versus 15.3 %, χ² (1, n = 114) = 0.03, p = 0.87, w = 0.02), and for the clinical interview, Black and White participants had similar paranoia scores (21.3% versus 12.2%, χ² (1, n = 114) = 0.72, p = 0.40, w = 0.08).

Relatively accurate models could be built from training data for the monologue and clinical interview speech (non-paranoid/paranoid cases were 439/57 and 141/27, respectively). Training set AUC values were .75 and .89 for the monologues and clinical interview samples (averaged across the 10 folds), but there were signs that the model did not work similarly well for all participants. First, accuracy in the test sets (i.e., comprising 20% of the data) was much lower (e.g., AUCs of .57 and .56 for the monologues and interview samples). Second, model accuracy was highly variable across the 10 training folds. For example, the range of AUC values across the folds for the monologue task ranged from .36 to .86 and .49 to .69 for the clinical interview tasks. Though not ideal, AUC values for the training sets were deemed sufficiently high to allow the evaluation of features and our hypotheses.

The top 20 linguistic features and their respective weights for each model are included in Table 1. The features were very different across the two models and predicted scores from one model and those applied to the data of the other task did not correlate with clinical paranoia ratings (p’s > 0.50). Stability selection found that a single linguistic feature related to the “cognitive process” was replicable across iterations of the monologue-based model. No features were identified from the stability selection of the interview data.

3.3. Demographic differences (Figure 1)

Figure 1. Model performance as a function of race and sex.

Predicted scores (i.e., “logit” scale scores from the combined training/test set) were plotted (see Figure 1), suggesting that the models were independent of each other. Four separate regressions examined whether clinical ratings and demographics interacted to predict relevant machine learning scores. For each regression, a) paranoia ratings were not significantly associated with machine learning scores, and b) neither demographics nor demographic-by-paranoia interactions were significantly associated with machine learning scores.

4.0. DISCUSSION

The present study examined relationships between natural language, clinically-rated paranoia, race, and sex. There were three notable findings. First, in contrast to at least some reports in the literature (Strakowsk et al., 1993; Whaley, 2004), Black and White participants were similar to each other in clinically rated paranoia. Second, high accuracy was obtained when using machine learning to model clinically rated paranoia from linguistic features, replicating prior research using linguistic analysis to capture paranoia (e.g., Cohen et al., 2022). Third, our linguistic models performed similarly as a function of race and sex, suggesting there was no overt bias or differential functioning from demographics in our models. These results were demonstrated across two corpora and several different speaking tasks.

Our study’s general lack of overt racial differences is encouraging and highlights the utility of linguistic-based features predicting paranoia. Our raw features (i.e., LIWC features) were generally similar as a function of race, and there were no detectable racial differences in how our models performed. It is important to note that we limited our linguistic features to those conceptually tied to paranoia, and we excluded features that we thought would overtly reflect education or socio-economic status (e.g., word length). It has long been known that linguistic-based measures of paranoia will need to be vetted for use across culturally different people, and the present study suggests that it can be used without overt biases commonly reported in the extant literature. There was evidence that our model did not perform similarly well for all people in our studies (as evidenced by the variable accuracy across folds). There was no evidence that the accuracy scores reflected race.

At the same time, women showed slightly higher (i.e., at a small effect size level, but non-significant) clinically rated paranoia than men in one study. Despite this, there were no major differences in raw features, and there were no detectable racial differences in how our models performed. This is difficult to interpret, as there is inconsistency across the literature in terms of paranoia and sex. Some have raised relationships between hormonal fluctuations and psychosocial factors that influence paranoia (Riecher-Rössler et al., 2018; Freeman et al., 2011), though it is far from clear that this affected the present results. Regardless, demographics did not influence linguistic expression or model performance.

Our use of a clinical rating of paranoia for modeling purposes assumes that the rating applies similarly (or not demonstrably different) across demographic groups, and this may not be the case. As discussed in the introduction, paranoia may be systematically rated for different reasons as a function of demographics. To create culturally sensitive linguistic models of paranoia, demographic differences could reflect at least four potential sources. First, it could be that the clinical raters were inaccurate in their ratings, perhaps because they were insensitive to cultural/racial/sex differences in how paranoia potentially presents itself during a clinical interview. It has been found that self-reported paranoia scores are potentially inflated amongst black study participants due to social and historical contextual factors that raters may not account for (Wolny et al., 2023). This issue may have been compounded by racial mismatches between the interviewer and the participant (Atkinson, 1985). Second, it could be that the clinical interviews and ratings used were not appropriately calibrated for the Black and/or female participants interviewed in this study. For example, the clinical interview may not have asked questions appropriate for the cultural groups recruited in this study. Third, it could be that paranoia is truly elevated in certain demographic groups; thus, the clinicians in this study accurately detected true group differences. Disparities in socioeconomic status, stress, and trauma reflect another potential cause of paranoia, and Race-based theories have linked paranoia specifically to Black men (Mosley et al., 2018). Recent findings have also discussed these concerns, highlighting how environmental stressors may lead to the over-representation of paranoid symptoms in high-risk groups, potentially resulting in misdiagnosis (Vargas et al., 2020). Considering sex differences, it has been proposed that estrogen impacts the expression of positive symptoms (Gogos et al., 2015). Fourth and somewhat related, it could be that paranoia severity is exaggerated by sociocultural disparities in the assessment and treatment of paranoia symptoms more generally. Studies show that Black individuals are less likely to receive quality care regarding therapy, diagnosis, and treatment for their symptoms (McGuire & Miranda, 2008; Cook et al., 2014; Swartz & Blankenship, 2014). Some studies discuss differences in diagnosis about sex as well (Thara & Kamath, 2015; Li et al., 2016). If true group racial differences in paranoia are captured in clinical rating scales, our models may also be missing these differences and, hence, may suffer in accuracy.

Some limitations warrant mention. A significant limitation of this study is that the data was collected primarily from the southern area of the United States, and the cultural impacts of that region could alter the representation of the language data. Second, our definition of paranoia was an industry “gold standard” but may not be culturally sensitive (Dowbiggin, 2000; Whaley, 1997). This is particularly important considering that our linguistic measures relied on predefined dictionaries (largely developed on racially and geographically homogenous groups) based on single-word stems (thus missing important between-word contexts). Future research should employ more sophisticated linguistic methods. Third and most importantly, our study does not account for the various social contexts that may influence clinical ratings of paranoia scores. As previously mentioned, various social and environmental factors may influence how paranoia is presented during the assessment process. Cognitive-based models of paranoia propose that it is a multidimensional construct that is continuous in the general population (Freeman et al., 2005). The severity of paranoia scores exists on a spectrum, with, for example, paranoia related to social anxiety being a common subvariant in the population. The present study used BPRS ratings to quantify people’s paranoia scores, which may have missed important context for their behaviors. Finally, our language samples did not explore cultural mistrust and how it is represented amongst the studied populations (Benkert et al., 2006; Whaley, 2001; Terrell et al., 2009).

In future studies, SMI research would benefit from a closer analysis of intra-racial groups. Studies often rely on between-group comparisons even when data has found that some (but not all) variance stems from a lack of cultural awareness rather than true racial differences (Trierweiler et al., 2006; Jones & Gray, 1986). The focus on race as an explanation for group variance promotes an aspect of racial essentialism within psychology (Haslam & Whelan, 2008; Markus, 2008; Causadias et al., 2018). Looking within groups would offer more meaningful data on symptom diagnostics within SMI patients. It would also be essential to analyze cultural mistrust amongst our research population. Future research would benefit from collecting a more symptomatic sample to train language models to evaluate potential group differences reliably and properly. The present study was limited by the limited number of participants with paranoia in either speaking task, which may have impacted model training/accuracy. Furthermore, asking more targeted questions about the participant’s environment and other social factors may improve how future studies classify paranoia, potentially resulting in improved diagnostic accuracy for model training purposes. Acknowledgment that participants and clinicians perceive each other is salient to explaining how language will be represented in linguistic studies.