Natural Language Processing and Psychosis: On the Need for Comprehensive Psychometric Evaluation

Abstract

Background and Hypothesis

Despite decades of “proof of concept” findings supporting the use of Natural Language Processing (NLP) in psychosis research, clinical implementation has been slow. One obstacle reflects the lack of comprehensive psychometric evaluation of these measures. There is overwhelming evidence that criterion and content validity can be achieved for many purposes, particularly using machine learning procedures. However, there has been very little evaluation of test-retest reliability, divergent validity (sufficient to address concerns of a “generalized deficit”), and potential biases from demographics and other individual differences.

Study Design

This article highlights these concerns in development of an NLP measure for tracking clinically rated paranoia from video “selfies” recorded from smartphone devices. Patients with schizophrenia or bipolar disorder were recruited and tracked over a week-long epoch. A small NLP-based feature set from 499 language samples were modeled on clinically rated paranoia using regularized regression.

Study Results

While test–retest reliability was high, criterion, and convergent/divergent validity were only achieved when considering moderating variables, notably whether a patient was away from home, around strangers, or alone at the time of the recording. Moreover, there were systematic racial and sex biases in the model, in part, reflecting whether patients submitted videos when they were away from home, around strangers, or alone.

Conclusions

Advancing NLP measures for psychosis will require deliberate consideration of test-retest reliability, divergent validity, systematic biases and the potential role of moderators. In our example, a comprehensive psychometric evaluation revealed clear strengths and weaknesses that can be systematically addressed in future research.

Natural Language Processing (NLP) and Proof of Concept

Natural Language Processing (NLP) is a multidisciplinary pursuit that involves computational processing, analyzing, and quantifying various aspects of language. It has become indispensable to modern society. NLP has also been critical for understanding language, and how it relates to cognition, affect, and social functions. It is increasingly being used to “digitally phenotype” aspects of psychosis-spectrum disorders^1–3; an endeavor that could reshape diagnosis, assessment, and treatment. From a pragmatic perspective, NLP allows for automation that can enhance traditional assessment by improving the efficiency, ecological validity, and accuracy of data collection, for example, by using data collected using mobile recording devices and social media platforms as individuals navigate their daily routines.^4,5 The use of “big” and high-dimensional data, collected on large, and demographically heterogeneous samples can also help address interpretive and practical constraints on traditional assessments.⁶ NLP can facilitate systematic and repeated assessment (e.g., hourly, daily, weekly); important because “naturalistic” measurements can be less affected by learning and practice effects than traditional clinical measures.^7,8 In all, NLP can improve accuracy for objectifying relatively specific aspects of psychopathology.^9–11

Clinicians have long used language to understand psychosis, and at least 5 decades ago began to use NLP to objectify and automate this process.^12,13 Since that time, NLP has been used in a large and complicated literature spanning psychology, psychiatry, engineering, computational, and other clinical, and basic sciences.^1–3,14,15 NLP has been used to both measure and understand various aspects of psychosis, including present, and future DSM IV/5 diagnosis,^16,17 illness onset,¹⁸ negative symptoms,^19–21 thought/language disorder,^22–24 social functioning,²⁵ hallucinations,²⁶ cognition,²⁷ paranoia,^13,25 substance use,²⁸ hope,²⁹ obsessive-compulsive symptoms,³⁰ and anhedonia.³¹ It has been applied to a variety of media as well, such as online social media,²⁵ medical records,³² standardized cognitive test responses,^33,34 clinical interviews,¹⁸ autobiographical monologs,³¹ and ambulatory audio/video recordings.^35,36

Despite decades of “proof of concept” data supporting the use of NLP in psychosis research, there has been no implementation for psychiatric or psychological clinical care to our knowledge, nor has there been approval by governmental regulatory agencies for clinical trials, forensic use, or for service determination. While there is aggregate support for NLP to understand psychosis, there is limited support/replication for any specific NLP solution. In part, this reflects the reality that NLP involves a broad set of analytic methodologies focusing on potentially disparate aspects of language (e.g., syntactic, lexical, semantics) and reflecting different training corpuses³ and different patient needs.³⁷ Moreover, strategies for evaluating NLP measures of psychosis have been relatively constrained, focusing primarily on a few aspects of validity with little attention to reliability, specificity, and bias. This article will a) discuss critical, but overlooked, aspects of psychometrics for evaluating NLP-based measures of psychosis, and b) highlight them by evaluating an NLP measure meant for tracking paranoia. In doing so, we hope to demonstrate a general psychometric approach for overcoming a major obstacle in implementing these measures.

Reliability, Validity and Bias

Traditionally, measures of psychiatric/psychological phenomenon are evaluated using reliability, and validity, the former being considered a prerequisite for the latter. Reliability concerns the consistency of a measure: across time (test–retest reliability), individual items of a measure (e.g., internal consistency), informants (e.g., inter-rater reliability), and situations (e.g., situational reliability). Validity concerns the measurement accuracy of its intended constructs; evaluated based on putative structure (e.g., structural validity) and potential convergence with conceptually related (e.g., convergent measure) and unrelated (e.g., divergent validity) constructs, and clinically-relevant criterion (e.g., concurrent, and predictive criterion validity). The precise aspects of reliability and validity, and their benchmarks, vary as a function of the nature of the measure and its application.³⁸ An NLP measure for monitoring change in disorganization using social media posts in European teenagers will differ in psychometric evaluation from a measure measuring alogia during clinical interviews in Asian adults.

There are several broad concerns with how NLP-based measures of psychosis have been evaluated thus far. First, reliability is rarely reported. Of the 206 peer-reviewed articles identified from an EBSCO search conducted on 10/24/2021 using terms “natural language processing” and “psychosis or schizo*” revealed 206 peer-reviewed entries, none presented reliability data. This is by no means a comprehensive literature search, but highlights the focus on validity over reliability. Insufficient attention to reliability, notably test-retest reliability has been identified as a major concern with network analysis,³⁹ taxometrics,⁴⁰ machine learning,⁴¹ neuroscience⁴² and mobile assessment^43,44 more generally. Elsewhere, we have evaluated test-retest reliability of computerized vocal and facial features and found it was often unacceptable without considering the influence of “moderating” variables, such as time of day, scope of assessment, and social factors .^45,46

Second, validity is often evaluated with respect to sensitivity (e.g., criterion validity: convergence with a “gold standard” measure) with only superficial evaluation of specificity. The “generalized deficit” issue, concerning the false appearance of specificity due to more global group differences, has long been recognized as a potential confound in psychopathology assessment.^46–48 Language abnormalities can reflect a variety of cognitive, emotional, and psychological factors, many of which are nonspecific to schizophrenia (e.g., social impoverishment, chronic stress). NLP solutions often show impressive accuracy in differentiating patients from nonpatients, or symptomatic from asymptomatic patients. However, it is often not clear the degree to which this specificity is masked by nonspecific factors. Confounding this process is the loss of interpretability of original features when using machine learning solutions. In most NLP studies, features are engineered/optimized beyond recognition because knowing what was predicted when performance was good is sufficient. However, interpretability and “explainability”, are increasingly being recognized as critical for the implementation of NLP technologies .⁴⁹

Third, systematic influences on reliability and validity, such as from demographic, cultural, linguistic, and other individual differences, are often not considered. Systematic “biases” have long been a focus of psychometric evaluation,⁵⁰ and of NLP more generally.^51,52 Efforts to address them have been increasing within the last few years,¹ though to our knowledge, the potential effects on NLP measures of psychosis are poorly understood and rarely examined.^6,53 This is important for NLP to help address issues of systemic racism and inequality in psychosis assessment/treatment .^54,55

There are challenges in applying traditional psychometrics to NLP. First, some aspects of reliability and validity don’t readily apply to NLP. NLP solutions are generally not meant to comprehensively capture constructs (i.e., latent phenomena), so evaluating the conceptual inter-relations of individual items/features through internal consistency, and construct and structural validity is often of limited use and considered unnecessary .⁵⁶ Moreover, the automated and objective nature of NLP obviates traditional inter-rater reliability, or at least, limits it to using different computers, datasets, software packages, and referential corpora if relevant. A greater challenge in evaluating NLP solutions involves the reality that language is highly dynamic within people, over time (e.g., min, hr, and days), and as a function of speaking task, environment, and other psychologically-relevant “moderators” (e.g., stress levels).^57,58 This can pose a challenge for establishing test-retest and situational reliability.

NLP Measure of Paranoia: An Example

Paranoia, defined in terms of self-relevant persecution, threat, or conspiracy from external agencies,⁵⁹ is a transdiagnostic and pernicious symptom of serious mental illnesses (SMI) that is typically measured using clinical ratings and self-report. Efforts to measure paranoia using NLP have been undertaken for over five decades, typically using lexical “text search” approaches using predefined dictionaries of word tokens (e.g., “good”, “bad”, “afraid”). Language samples typically involve autobiographical monologues and medical records from clinical settings.^13,32,60,61 Not surprisingly, symptoms of paranoia have been associated with relatively high negative affective and low affiliative word use. While promising in criterion and convergent validity, there has been limited evaluation of test–retest reliability, divergent validity/specificity, and systematic biases associated with these measures. Lexical expression, particularly negative affect, is likely a nonspecific measure of paranoia in that abnormalities are central to many types of psychopathology and symptoms (e.g., depression, anxiety). Moreover, they may differ in use by demographic factors. Clinically rated paranoia has been higher in Black than White samples⁶² and likely reflects the interplay of a complex set of cultural, interpersonal, and professional influences.⁶³

NLP Measure of Paranoia: NLP Feature Development/Selection

NLP feature development and selection is a critical component of model development, and there are a variety of data and conceptually driven approaches used. Within paranoia research, studies have tended to use “out of box” solutions that may be nonspecific to paranoia; tapping negative affect more generally. Developing a small-feature set that aligns with social cognitive definitions of paranoia ⁶⁴ may improve the psychometric characteristics, in particular, specificity to paranoia. In the next section, we evaluated several NLP measures for tracking relatively subtle fluctuations in paranoia from brief, topically-flexible speech samples procured from a smartphone while individuals navigated their daily routines. Patients with serious mental illness (SMI) provided “video selfies” over a week-long period. We employed an NLP procedure called sentiment analysis,⁶⁵ which is similar to lexical analysis approaches but takes into account valence shifters (e.g., negators, amplifiers) at a relatively fine-scale (phrases and complex word structures). Given that the operational definition of paranoia involves threat directed from the outside to the self, we focused on “self-other” references, defined as language that contained reference to both self and “non-self” using an NLP procedure that identifies subject-object dependencies within language. Finally, we examined the role of three moderators that putatively exaggerate, or at least concomitantly occur, with paranoia. Based on evidence that paranoia is dynamic as a function of being alone, around strangers, and experiencing threat,⁶⁶ we considered whether language was produced whether the individual was away from home, whether they were around strangers, and whether they were alone. Our focus on “self-other” speech and potential moderators was intended to improve specificity to paranoia beyond more global aspects of psychopathology.

We employed regularized regression to develop a model predicting clinically rated paranoia based on three NLP features, with consideration of three moderators (i.e., away from home, among strangers, and alone; see figure 1). Our NLP measure was evaluated in terms of a) test-retest reliability, b) convergence with clinical ratings of paranoia and self-report ratings of fear in the moment, divergence with measures of anxiety/depression, and c) bias in demographic factors.

Fig. 1.

Study rational and methods: description of the three key features and moderators developed and evaluated in this study, and the psychometric evaluation plan.

NLP Measure of Paranoia: Methods

Participants:

Data was collected from 35 individuals with DSM-5⁵⁹ diagnoses of schizophrenia (SZ; n = 31) or bipolar disorder (BP; n = 4). Patients were primarily female (n = 10 men, 25 women) and White (23 White, 10 Black, 2 missing data) with an average age of 40.7 ± 11.5 and education of 13.7 ± 2.6 yr. Individuals with SZ were recruited from local community outpatient mental health centers. Clinical diagnosis was determined via the Structured Clinical Interview for DSM-5.⁶⁷ Participants provided written informed consent and received monetary compensation for their participation. This project was approved by the University of Georgia and Louisiana State University Institutional Review Boards and executed in accordance with internationally-recognized standards for the ethical conduct of human research .^45,68

Clinical Measures

The Positive and Negative Symptom Scale (PANSS)⁶⁹ was used to measure psychiatric symptoms. The “Suspiciousness/Persecution” scale was used as our criterion, and the “Depression/Anxiety” factor score was used for divergent validity. Clinical ratings were made by staff trained to reliability standards (alpha >.8) on gold standard training tapes.

Mobile assessment

Participants were provided mobile phones preloaded with data collection software.⁷⁰ Surveys recorded current information such as whether the participant was “alone” at the time of the recording, and whether they were around “strangers”. This information was coded as binary (i.e., yes, no) and not mutually exclusive. Self-reported in-the-moment “suspiciousness” and “fear” were also obtained (scale = 1–100), as was geolocation. Values greater than 50 meters of their home “pin” were deemed “away” from home.

Videos were recorded at the end of the surveys but were optional. Participants were instructed to record themselves while giving a step-by-step description of their past hour for 60 seconds. Participants were compensated $1 for each survey completed with no additional incentive for completing videos. Patients completed videos for an average of 23% (standard deviations = 20%) of the surveys. 499 videos were available for analysis.⁴⁵ Geolocation data were available for 368 videos, and self-report ratings were available for 412 videos. Approximately 45% of recordings were conducted away from home (164 of 368), 17% conducted with strangers (46 of 412), and 43% conducted alone (176 of 412). Missing data and distributions of surveys are in supplemental table 1.

Natural Language Processing

Speech was transcribed into text documents by trained research assistants; approximately 10% of which were reviewed by a supervisor. Using Python (v3.8) and Natural Language Toolkit library (NLTK),⁷¹ documents were filtered by length (character length > 50), contractions were expanded, stop-words (e.g., the, an, a) were removed, and words not in NLTK English dictionary were removed. For each document, Stanford CoreNLP⁷² annotators were used to identify parts of speech, lemmatize words, parse sentence-level dependencies, and extract open-domain relation triples (OpenIE). This yielded “dependencies”, representing a subject, a relation, and the object of the relation. Sentiment analysis, conducted using the SentimentR package,⁶⁵ provides summary scores of the emotional valence of text using a bipolar scale from ‐1 (extremely negative) to 0 (neutral) to 1 (extremely positive). For each document, we extracted three NLP features to employ in our models: 1) The sentiment from the entire language sample (i.e., “Sentiment”), 2) the number of self-other dependencies in the entire language sample (i.e., “Self-Other Dependencies”), and 3) the minimum value from sentiment analysis of each individual self-other dependency (i.e., “Self-Other Sentiment”). The minimum value reflects the most negative of the Self-Other dependencies in that sample.

Patients produced, on average, 18.9 ± 14.3 dependencies, of which 2.7 ± 2.0 contained self and non-self in the subject and object. The average sentiment of these within-speech dependencies was slightly positive (0.2 ± 0.2; range = 10), and the minimum (0.0 ± 0.3; range = ‐1.1 to 0.7) and maximum (0.3 ± 0.3; range = ‐1.0 to 1.2) sentiment values from these dependencies (evaluated within an individual language sample) suggested there was notable range within a sample.

Analyses

First, we extracted and evaluated our three NLP features. Evaluation focused on convergence with our criterion (i.e., Clinically Rated Paranoia), test–retest reliability (using Intra-class Correlation Coefficients; ICC), and relationship to demographic variables. Second, we modeled Clinically Rated Paranoia from our three NLP features using ridge regression, which employed a five-fold cross-validation process (using an 80–20% training-test split to fit the original). We computed three models, each with seven terms: the three NLP features, one moderator (e.g., being home versus away during the recording), and three NLP by moderator interactions. Models varied based on the moderator used (i.e., three moderators, three models). An “Integrated NLP Measure” was computed using the average of the three fitted final model scores. We were unable to fit them together using additional regressions because of missing moderator data (e.g., missing GPS data). Third, the Integrated NLP measure was evaluated in a) test-retest reliability, b) convergence with mobile measures of suspiciousness and fear, c) divergence with clinical ratings of depression/anxiety, and d) potential bias in demographic variables. Analyses were conducted in R (R Core Team, 2017). We were unable to nest data within each patient because we were modeling “2nd order” variables (e.g., PANSS scores). Given that the number of videos varied by participant, reliability analyses (i.e., involving ICCs) were conducted on data averaged by day. This helped standardize data as a function of time—since recording times potentially varied across participants. All Variance Inflation Factor scores were below 2.50 suggesting multi-collinearity was not an issue. NLP data were standardized and trimmed (i.e., at 3.50 SDs).

NLP Measure of Paranoia: Results

Step 1: Feature Evaluation

Preliminary Statistics

(Supplemental Table 2). A good range of clinically rated paranoia was observed in the sample, with 17 patients showing no/questionable clinically rated paranoia (k = 283 videos) and 18 patients showing mild or greater clinically rated paranoia (k = 216 videos). A correlation matrix of dependent variables, features, and machine learning based scores is provided in supplemental table 2. Paranoia was inversely associated with the number of videos submitted at a trend level (r[33] = ‐0.30, P = .08). Patients with bipolar disorder versus schizophrenia were lower in clinically rated paranoia at a trend level (t[34] = 2.19, P = .07),

Test–Retest Reliability.

Average ICC values of the three features were good to excellent across days (range of ICC’s = 0.63–0.83). Single ICCs were lower (range of ICC’s = 0.21–0.45). This suggests single time-point scores were a reliable measure of average scores, but were not themselves stable over time.

Criterion Validity

(Supplemental Table 2). The NLP features were quite modest in their relations to Clinically Rated Paranoia (range of r’s = ‐0.06 to 0.05).

Step 2: Modeling Clinically Rated Paranoia from NLP measures

Model Development (

Table 1, Supplemental Table 5). NLP features showed relatively similar RMSE and MAE values across training and test cases, suggesting that overfitting was not a major concern. Final models explained 10%, 5%, and 9% of the variance for the “Away from Home”, “Around Strangers” and “Alone” models respectively. An “Integrated NLP Measure” was computed as the average of available model scores.

Table 1.

Model statistics forregularized regressions

	Training case (80%)				Testing case (20%)
	K	RMSE	MAE	R 2	K	RMSE	MAE	R 2
	Model: away from home
5-Fold average	276	1.32	1.09	0.10	69	1.34	1.09	0.10
Final model	345	1.32	1.09	0.10	–	–	–	–
	Model: being with strangers
5-Fold average	330	1.39	1.21	0.05	82	1.39	1.22	0.04
Final model	412	1.398	1.20	0.05	–	–	–	–
	Model: being alone
5-Foldaverage	330	1.36	1.15	0.11	82	1.35	1.15	0.08
Final model	412	1.35	1.14	0.09	–	–	–	–

Notes: K, number of samples; RMSE, root mean square error; MAE, mean absolute error. Final model reflects all cases.

Step 3: Integrated NLP Measures of Paranoia

Test–Retest Reliability

(Table 2). The Integrated Measure showed average and single ICC values of 0.91 and 0.62, suggesting good to excellent test-retest reliability for both average and single time-point applications. Average ICC values for the independent models were generally good to excellent for all data, and for White, Black, and Male, and Female participants. Single ICC values varied from poor to good.

Table 2.

Intra-class correlation coefficient (ICC) values for the three NLP-based models of paranoia as a function of all data and of race and sex

		Model 1: away from home	Model 2: being with strangers	Model 3: being alone	Integrated NLP measure
All data1		0.77 (0.40)	0.72 (0.34)	0.81 (0.46)	0.91 (0.62)
Race2	White	0.93 (0.71)	0.71 (0.33)	0.71 (0.33)	0.74 (0.37)
	Black	0.86 (0.56)	0.88 (0.59)	0.91 (0.68)	0.89 (0.68)
Sex2	Male	0.45 (0.17)	0.62 (0.29)	0.80 (0.43)	0.73 (0.10)
	Female	0.87 (0.62)	0.58 (0.26)	0.61 (0.28)	0.74 (0.42)

Average and single (in parentheses) ICC values are presented.

¹Data averaged by day for 7 days; ICC values reflect stability across days,

² Data averaged over 4 days, due to missing data.

Convergent/Divergent Validity

(Supplemental Table 1). As expected, the Integrated NLP Measure was associated with significantly increased Clinical Ratings of Paranoia (r[460] = 0.42, P <.001), Momentary Ratings of Suspiciousness (r[410] = 0.20, P < .001), and Momentary Ratings of Fear (r[410] = 0.21, P < .001). Also as expected, the Integrated NLP Model was not significantly correlated with clinical ratings of depression/anxiety (r[460] = 0.00, P = 1.00). Clinically Rated Paranoia was similarly associated with both Momentary Ratings of Suspiciousness (r[410] = 0.31, P < .001) and Fear (r[410] = 0.30, P < .001), and was not related to Clinically Rated Depression/Anxiety (r[497] = 0.00, P = 1.00).

Demographic Biases

(Figure 2; Supplemental Table 6). The Integrated NLP Measure showed differential prediction of Clinically Rated Paranoia for race and gender. Correlations for White (r[371] = 0.46, P < .001) and for female (r[316] = 0.40, P < .001) participants were much higher than for Black (r[63] = 0.00, P = 1.00) and male (r[142] = 0.18, P = .03) participants. Neither Age nor Education were significantly associated with the Integrated Model (r’s = 0.01 and ‐0.07, P’s > .11).

Fig. 2.

Integrated NLP Solution as a function of race (top panel) and sex (bottom panel), with relative frequency of moderating variables (i.e., when videos were recorded). Solid line reflects white/male. Dotted line reflects black/female.

Clinically Rated Paranoia was significantly higher in Black compared to White participants (t[2, 472] = 9.11, P < .001, d = 1.49). The Integrated NLP Measure was significantly different in Black and White participants (t[2, 472] = 5.02, P < .001, d = 0.65), though the magnitude as approximately half as large. Both Clinically Rated Paranoia and the Integrated NLP Measure were significantly different between Men and Women (P’s < .001, d’s = 0.84 and 0.82 respectively). Both Age and Education were significantly associated with Clinically Rated Paranoia (r[472]’s = 0.19 and ‐0.29, P’s < .001). The integrated NLP measure was significantly associated with age and education (r’s = 0.19 and ‐0.10 respectively).

Moderators differed as a function of demographics. Black and White participants were notably different in whether they shared videos while Away from Home (14% versus 49% of total videos; Chi-square = 5.30, P = .02) or Around Strangers (0% versus 14% respectively; Chi-square = 11.38, P < .001). Men and Women were different at a significant/trend level in whether they shared videos while Away from Home (58% versus 38% of total videos; Chi-square = 9.01, P= .003) or Around Strangers (22% versus 6% respectively; Chi-square = 26, P = .02) or Alone (54% versus 37% respectively; Chi-square = 3.56, P = .06). Increasing age was associated with being alone (r[410] = 0.15, P < .001) and being home (r[366] = ‐0.26, P < .001). Education was not significantly associated with any of the moderators. There were no dramatic differences (i.e., > 10%) for participants as a group, but Black participants produced many more surveys when away from home (35%) compared to video “selfies” when away from home (14%) (supplemental table 6).

Conclusions and Future Directions

While there is no “one size fits all” psychometric evaluation strategy for NLP measures of psychosis, there are common psychometric components appropriate for most. Test–retest reliability, divergent validity (i.e., addressing specificity), and demographic bias are three metrics that have received limited attention to date and are likely relevant regardless of the media evaluated (e.g., social media, traditional neuropsychology test, clinical interview), temporal scope of the assessment (e.g., single clinical interaction, repeated measurement), or clinical scope/purpose. Evaluating them will be critical for realizing NLP’s potential for measuring psychosis and for adopting them for institutional use [reference to Marder, Hauglid, and Palaniyappan commentaries].

The Integrated NLP-based measure of paranoia examined in this study was reliable over a week-long epoch, showed good convergence with our criterion and conceptually related measures, and showed specificity/divergence with global measures of negative affect and psychopathology. It also showed higher criterion validity for White and Female participants than for Black and Male participants. This latter point underscores the need for comprehensive psychometric evaluations for NLP measures of psychosis since an evaluation strategy failing to consider demographic biases would have missed this critical issue. This study was limited in that the sample was modest in size, was demographically constrained, and video data were incomplete for many participants (due, in part, to a lack of compensation for data completion). The sample exclusively comprised people with SMI, and a community sample with a broader continuum of symptom severity would be important for future research. Moreover, reliability evaluation was limited to a single week in temporal scope; which is a limited epoch for understanding paranoia as a dynamic construct. Future research should include demographically diverse samples, in part, to power understanding of how demographic intersectionality (e.g., race, sex) affects measure psychometrics. Larger and more diverse samples are also critical for generalization.

While the present article was focused on general psychometric issues concerning NLP measures of psychosis, it is worth considering how one might address the sorts of biases found in our results. For Black participants, the bias appeared to reflect a lack of data across moderating conditions; participants were unlikely to provide videos when not at home or when around strangers. The reasons for this are not fully clear, though participants may have been reticent to provide data in these conditions due to concerns about personal safety, or stigma or negative evaluation from others. Paranoia, whether cultural, and/or mental illness in origin, likely did not attenuate these concerns. Outreach to potential participants as stakeholders will be critical in addressing these concerns, which could involve adapting data collection (e.g., using more subtle, less publicly noticeable procurement methods), providing additional participant support, training, compensation, or expanding the scope of assessment (e.g., to other situations where individuals experience paranoia but are comfortable providing videos). Men in this study showed lower criterion validity than women, and this did not seem to reflect a lack of data provision away from home or near strangers. The present sample size was not adequate for resolving demographic biases, inter-sectionality between demographic variables, and for generalizing our findings across diverse people and communities. While including a larger and more diverse sample than in this study is important, understanding the cultural differences in how paranoia manifests through language is also a critical concern highlighted by our data.

In closing, it is worth highlighting that language aspects were fairly dynamic over time. NLP data can be aggregated across language samples and across time and space/environment in various ways (e.g., both within and between testing sessions). This is important to consider for optimizing reliability, sensitivity, and specificity of a construct of interest^19,44,73 and is akin to “situational reliability”, an aspect of psychometrics rarely examined in NLP-psychosis research. In the present data, NLP features varied as a function of moderating variables conceptually tied to paranoia. Our integrated model including these moderating variables showed improved reliability over most models/features not including them.

Realizing NLP’s potential for measuring aspects of psychosis will require large amounts of complex data collected from geographically and culturally diverse groups,⁷⁴ and coordination between multidisciplinary and international groups.^{75, 76} A comprehensive psychometrics strategy will be a critical part of this endeavor.

Funding

This work was supported by NIH (National Institute of Health) (grant number R21 MH112925) to G.P.S. The authors have declared that there are no conflicts of interest in relation to the subject of this study.

Competing interests: The authors declare that there are no competing interests.