Computational linguistic analysis applied to a semantic fluency task: A replication among first-episode psychosis patients with and without derailment and tangentiality

Abstract

Automated tools do not yet exist to measure formal thought disorder, including derailment and tangentiality, both of which can be subjectively rated using the Scale for the Assessment of Positive Symptoms after a clinical research interview. CoVec, a new automated tool, measures the semantic similarity among words averaged in a five- and ten-word window (Coherence-5 and Coherence-10, respectively). One prior report demonstrated that this tool was able to differentiate between patients with those types of thought disorder and patients without them (and controls). Here, we attempted a replication of the initial findings using data from a different sample of patients hospitalized for initial evaluation of first-episode psychosis. Participants were administered a semantic fluency task and the animal lists were analyzed with CoVec. In this study, we partially replicated the prior findings, showing that first-episode patients with derailment had significantly lower Coherence-5 and Coherence-10 compared with patients without derailment. Further research is warranted on this and other highly reliable and objective methods of detecting formal thought disorder through simple assessments such as semantic fluency tasks.

1. Introduction

Formal thought disorder (FTD) is a long-recognized impairment of thought processes characterized by disorganization and difficult-to-follow speech (Bleuler, 1950). It is a hallmark feature of schizophrenia, and clinical rating scales (e.g., the Scale for the Assessment of Positive Symptoms, SAPS) have been used to categorize it into aspects such as derailment and tangentiality, and to assess severity (Andreasen, 1984). Derailment is a pattern of spontaneous speech in which spoken ideas may have unclear or no connection between them. Tangentiality is another type of FTD in which the speaker replies to a question in an oblique, tangential, or even irrelevant manner. Objective measures for clinicians to record or track severity of these manifestations of schizophrenia in a standardized manner currently do not exist.

A growing body of research investigates various computational methods to discriminate between individuals with schizophrenia and unaffected controls based on semantic coherence and speech connectedness. Elvevåg and colleagues (2010) showed that compared to other statistical measures, semantic analysis contributed the most to differentiating those with schizophrenia from healthy controls. Maher and coworkers (2005) found that patients with schizophrenia presented with a higher frequency of normative associations, suggesting that language disturbance in schizophrenia may be driven by a hyperactivity of associational networks. Mota and colleagues (2017) analyzed dream reports and showed that speech connectedness was impaired in schizophrenia. From these findings, it seems plausible that the organization of words could be informative in assessing FTD. However, clinically applicable computational methods for assessing the severity of FTD in schizophrenia are lacking.

Semantic fluency tasks are widely used in neurocognitive testing (Shao et al., 2014). Participants are asked to produce as many words from a category as possible in 60 seconds. Such tasks are usually scored based on the number of words produced. Previous studies show that patients with schizophrenia generate fewer words, but the reason for this is not clear (Bokat and Goldberg, 2003). Other researchers have used this task to manually measure semantic memory organization such as semantic clustering and switching, which refers to the organization of semantically related words into groups and the frequency of transitions between these groups, respectively (Troyer et al., 1997). Although this manual approach to measuring semantic organization has great potential, it is subjective, unstandardized, and labor intensive, limiting its clinical utility. More recent studies have developed computational methods to analyze semantic fluency tasks (Holmlund et al., 2019; Kim et al., 2019; Pauselli et al., 2018; Pietrowicz et al., 2019; Sumiyoshi et al., 2018). In fact, a meta-analysis suggested that semantic memory deficits (as measured by fluency tasks) are present among individuals with schizophrenia and their first-degree relatives (Tan et al., 2020). However, the range of fluency task metrics included in this study was limited.

CoVec is a new automated linguistic software that uses vector semantics to measure Coherence-5 and Coherence-10, which is the semantic similarity between words averaged in a five- and ten-word window, respectively (Covington, 2016). A higher value represents greater similarity. In our initial report (Pauselli et al., 2018), we demonstrated that patients with schizophrenia with derailment had a significantly lower Coherence-5 than controls and patients without derailment, and patients with tangentiality had significantly lower Coherence-5 and Coherence-10 than controls and patients without tangentiality. Here, we attempted a replication of the initial findings using data from a different sample of patients hospitalized for initial evaluation of first-episode psychosis. We hypothesized that among these patients, Coherence-5 and Coherence-10 would be lower among those with derailment and tangentiality compared with those without FTD.

2. Methods

Data were obtained as part of a study investigating the associations between premorbid cannabis use and age at onset of psychosis (Kelley et al., 2016). That study included patients admitted to the hospital for a first episode of a schizophrenia-spectrum disorder; they were referred by clinicians in three inpatient psychiatric units in Atlanta, Georgia, and three in Washington, D.C. The eligible age range for the study was 18 to 30 years. The Structured Clinical Interview for DSM-IV Axis I Disorders (First et al., 1998) was used to make research diagnoses, using all available information, including in-depth interviews with participants. Among 713 subjects referred as potentially eligible or approached due to likely being eligible, a total of 247 were enrolled from August 2008 to June 2013. The study was approved by all relevant Institutional Review Boards.

As in our prior study of outpatients with schizophrenia and controls (Pauselli et al., 2018), all participants were administered a semantic fluency task (naming as many animals as possible in 60 seconds) as part of the MATRICS Consensus Cognitive Battery (Kern et al., 2008; Nuechterlein et al., 2008). Since it was not known that the animal list would later be used as primary data after CoVec was developed, reliable transcripts of the animal list were available for only 197 of 247 participants from the larger project. Psychotic symptoms and FTD were assessed by trained raters using the SAPS (Andreasen, 1984; Andreasen et al., 1995). Derailment and tangentiality are assessed in the SAPS with a 6-point rating scale (0=None, 1=Questionable, 2=Mild, 3=Moderate, 4=Marked, and 5=Severe), which is used to evaluate all of the positive symptoms. As reported elsewhere (Birnbaum et al., 2017), inter-rater reliability was assessed using intraclass correlation coefficients (ICCs) computed using two-way random effects ANOVA models. The ICC for the FTD subscale of the SAPS was 0.89 (95% CI: 0.80, 0.94).

As in our prior report (Pauselli et al., 2018), for this analysis, we classified participants into two groups: (1) patients with a score of “None” for the SAPS derailment item, and (2) those with a score of “Moderate,” “Marked,” or “Severe” on that item. We did the same for the SAPS tangentiality item. The patients who were rated as “Questionable” or “Mild” were not included to ensure that the analyses took into consideration only patients with and without clear manifestations of derailment or tangentiality. Regarding derailment, 157 patients did not have this thought disorder, 39 did, and 51 were excluded due to “Questionable” and “Mild” ratings. In terms of tangentiality, 112 patients did not have this form of thought disorder, 73 did, and 62 were excluded due to “Questionable” and “Mild” ratings. As such, among the 79 patients with FTD, six had derailment but not tangentiality, 40 had tangentiality but not derailment, and 33 had both derailment and tangentiality.

The transcripts of the animal list were converted to plain ASCII text and hand-edited (by a researcher blinded to the subject’s FTD status) to enforce standard spelling, including combining two words into one where appropriate (e.g., red bird to redbird). It was observed that the samples were largely free of repetitions and of words not denoting animals. However, unlike our prior report, we removed any repeated words, as repetitions could affect CoVec-derived measures.

Analysis was performed with CoVec version 1.0.5912 (Covington, 2016). CoVec measures the semantic similarity of words using the vector methodology of the Stanford GloVe project (Pennington et al., 2014). Words are considered similar if they occur in similar contexts in a large set of English texts. The GloVe project’s data files, trained on 840 billion words of English text with 300-element vectors, were used as norms. The output of CoVec effectively picks out synonyms and words that are commonly used together for any reason. Mean Similarity is the average similarity of each word to the immediately preceding word. Coherence is the average similarity of each word to each of the other words in the list, regardless of order or proximity. This tends to be lower with longer samples because longer lists are inherently more diverse; thus, Coherence-5 and Coherence-10 are computed by moving a 5-word or 10-word window through the text and computing Coherence of the window as if it were the whole text, then averaging the values thus computed for all positions of the window. This produces a measure of local coherence not affected by the length of the transcript.

Distributional properties of variables and descriptive statistics were examined. Differences in the means between the two groups of participants (those with and without SAPS-rated derailment and tangentiality) were assessed using independent samples Student’s t tests. To aid in interpreting differences in means between groups, Cohen’s d was calculated, with 0.20 deemed to be a small effect, 0.50 a medium effect, and 0.80 a large effect.

3. Results

Participants included 197 first-episode psychosis patients who had animal lists that could be transcribed. As given in Table 1, the average age of participants was 24.0±4.9 years. The majority were male (147, 74.6%), African American (171, 86.8%), and single and never married (166, 84.3%). The average years of education attained was 11.8±2.3.

Table 1.

Sociodemographic Characteristics of the Study Sample, n=197

Age, mean±SD	24.0±4.9
Gender, n (%)
Male	147 (74.6%)
Race, n (%)
African American	171 (86.8%)
Caucasian	14 (7.1%)
Other	12 (6.1%)
Marital status, n (%)
Single and never married	166 (84.3%)
Years of education, mean±SD	11.8±2.3

In terms of correlations among our variables of interest, the number of animals produced was modestly negatively correlated with Coherence-5 (r=−0.236) and Coherence-10 (r=−0.322), but not with Mean Similarity (r=−0.086). Mean Similarity correlated with Coherence-5 at 0.860 and with Coherence-10 at 0.780 (and the correlation between Coherence-5 and Coherence-10 was 0.942).

Spearman correlation coefficients for the three CoVec measures and derailment ranged −.117 to −.157; for tangentiality, −.058 to −.075. Mean and standard deviation (SD) values for the four CoVec variables of interest—in both patients with and without derailment and patients with and without tangentiality—are given in Table 2. The only significant difference observed was a lower number of animals produced among patients with derailment (16.23±6.52, compared to 19.31±5.61 among those without derailment; t=2.66, df=158, p=0.009, d=0.51). The difference in Coherence-10 between patients with and without derailment approached significance (0.480±0.053 compared to 0.495±0.035; t=1.91, df=153, p=0.06, d=0.33).

Table 2.

Group Values (mean±SD) for Number of Words and the Three CoVec Variables, among Patients with and without Derailment and Tangentiality

	Number of Words	Mean Similarity	Coherence-5	Coherence-10
Derailment
Patients with moderate to severe derailment, n=28–31	16.23±6.52	0.496±0.063	0.559±0.043	0.480±0.053
Patients without derailment, n=127–129	19.31±5.61	0.507±0.037	0.569±0.030	0.495±0.035
	p=0.009 d=0.51	p=0.200 d=0.23	p=0.100 d=0.27	p=0.060 d=0.33
Tangentiality
Patients with moderate to severe tangentiality, n=54–57	18.67±6.30	0.503±0.050	0.565±0.036	0.487±0.043
Patients without tangentiality, n=89–91	19.00±5.50	0.508±0.030	0.569±0.032	0.493±0.039
	p=0.740 d=0.05	p=0.550 d=0.12	p=0.560 d=0.12	p=0.430 d=0.15

p= significance of between-group comparisons when controlling for number of words

d=Cohen’s d effect size computed using raw data rather than estimated marginal means

Note: The sample size varied among number of words and three CoVec variables since CoVec excluded animal lists with less than five or ten words generated for Coherence-5 and 10, respectively.

Even though both measures partially control for coherence being lower with longer, inherently more diverse samples, the number of words produced is likely a confounder as it is modestly negatively correlated with both Coherence-5 and Coherence-10. Therefore, we repeated the comparisons of Mean Similarity, Coherence-5, and Coherence-10 while controlling for number of words, using analyses of covariance (Table 3). Here, patients with derailment had a lower Coherence-5 (0.554±0.006 compared to 0.570±0.003; t=2.64, df=158, p=0.009, d=0.27) and Coherence-10 (0.475±0.007 compared to 0.496±0.003; t=2.92, df=154, p=0.004, d=0.33).

Table 3.

Group Values (Estimated Marginal Means ± SE) for the Three CoVec Variables, among Patients with and without Derailment and Tangentiality, Controlling for Number of Words (Analyses of Covariance)

	Mean Similarity	Coherence-5	Coherence-10
Derailment
Patients with moderate to severe derailment, n=28–31	0.493±0.008	0.554±0.006	0.475±0.007
Patients without derailment, n=127–129	0.508±0.004	0.570±0.003	0.496±0.003
	p=0.085 d=0.21	p=0.009 d=0.27	p=0.004 d=0.33
Tangentiality
Patients with moderate to severe tangentiality, n=54–57	0.503±0.006	0.565±0.004	0.487±0.005
Patients without tangentiality, n=89–91	0.508±0.005	0.569±0.003	0.493±0.004
	p=0.473 d=0.11	p=0.427 d=0.12	p=0.403 d=0.15

p= significance of between-group comparisons when controlling for number of words

d=Cohen’s d effect size computed using raw data rather than estimated marginal means

Note: The sample size varied among number of words and three CoVec variables since CoVec excluded animal lists with less than five or ten words generated for Coherence-5 and 10, respectively.

Table 4 shows the actual list of animals for a patient without derailment (and without tangentiality) and a patient with derailment (but without tangentiality), each list selected by the individual-level Coherence-5 value that most closely approximated the mean of the group.

Table 4.

Actual Lists of Animals from Two Participants, One Without Derailment (and Without Tangentiality), and the Other With Derailment (but not Tangentiality)

	Patient without Derailment	Patient with Derailment
Group Mean	0.570	0.554
Illustrative Individual Subject’s Score	0.569	0.528
Animal List	dog cat horse cow tiger llama emu goose chicken duck whale shark octopus squid squirrel rat snake spider cockroach rabbit turtle lizard gecko lion	lion tiger bear zebra alligator blowfish dolphin cat dog giraffe elephant snake spider grasshopper cricket cockroach ant monkey ape bird parakeet penguin

4. Discussion

In this replication analysis, we aimed to show that a very widely used one-minute cognitive test of verbal fluency may contain information beyond the simple number of words produced. Several recent studies have used a variety of ‘objective’ approaches to fluency data that extract more information than just mere word count (Elvevåg et al., 2007; Holmlund et al., 2019; Kim et al., 2019; Voorspoels et al., 2014). Similarly, CoVec may detect signals capable of differentiating patients with and without derailment, in subtle manifestations that may be very difficult to detect using standard clinical research tools such as the SAPS.

In this study, we replicated the finding from our prior report (Pauselli et al., 2018) that patients with derailment have a significantly lower number of words generated than those without derailment (in the prior study, about 12 words compared to about 17 words, with a Cohen’s d of 0.84, and in the current study, about 16 words compared to about 19 words, with a Cohen’s d of 0.51). It is possible that such derailments interfere with the train of thinking and thus the generation of words within the 60-second task.

In our prior report (Pauselli et al., 2018), we demonstrated that outpatients with schizophrenia with moderate to severe derailment had a significantly lower Coherence-5 (0.514) than patients without derailment (0.552) with a large effect size (d=0.97). Here, we also replicated that finding, showing that, when controlling for the number of words produced, first-episode psychosis patients with moderate to severe derailment had a significantly lower Coherence-5 (0.554) than patients without derailment (0.570) with a small to medium effect size (d=0.27). It should be noted that the 5-word and 10-word window lengths are purely arbitrary; there is no reason to think either of the lengths has any particular significance. In fact, the best window size for making these measurements may be something else, or even a “soft window” where words farther from the center are given less weight.

An example of what might be derailment among a sequence of words includes (from the prior report): “…elephant, lion, bear, tiger [all of which are zoo animals], dog, rat, bat, squirrel, mosquito, orangutan, monkey…” Here, bat rhymes with rat, and as such might represent a “loosening of associations,” and mosquito seems to be illustrative of derailment. From the current study, an example of derailment might be represented by the word “blowfish” in the example provided in Table 4; however, in general, it is very difficult to “observe” the derailment within the lists of animals from one patient to the next. As such, CoVec would seem to be detecting derailment that is very difficult for the human listener/reader to observe.

The replicated finding that those with derailment produced words with lower semantic coherence is consistent with prior literature. Just et al. (2019) tested whether computational linguistic approaches applied to speech samples in German could differentiate healthy controls, patients with FTD, and those without FTD, including derailment. They found that only the Incoherence model, which measured the cosine similarity between pairs of adjacent sentences’ embeddings using GloVe word embeddings (trained on German texts), significantly differentiated patients with and without derailment and that coherence scores were negatively correlated with SAPS derailment ratings (r=−0.5; p< 0.05) (Just et al., 2019). However, the Tangentiality model, which measured the cosine similarity between question and response, did not significantly differentiate between patients with and without FTD. In fact, in our study, CoVec also could not significantly differentiate tangentiality ratings even though coherence scores were lower among patients with compared to those without tangentiality. Perhaps CoVec is more sensitive in detecting the within-utterance phenomenon of derailment rather than how a patient replies to questions (as measured by tangentiality), which may be on different temporal and spatial scales. In addition, our studies controlled for length and removed repeated words. Therefore, the results of this study build upon the findings of prior research showing that automated tools measuring semantic similarities between words could differentiate those with and without derailment.

Several methodological limitations should be noted. First, as noted in our prior report (Pauselli et al., 2018), even though we used the SAPS—a widely recognized and utilized instrument to measure positive symptoms—there is no reason to think that it is a completely accurate or “gold standard” way of evaluating FTD because the scoring is based on a clinical interview and subjective rating. In fact, the replicated findings in this study of the inverse correlation between derailment and Coherence-5 occurred only after controlling for number of words. Nevertheless, future CoVec-type measures will probably be the “gold standard” as they are completely objective and perfectly reliable. Second, the three different CoVec output measures are strongly inter-correlated, meaning that they measure the same phenomenon using slightly different approaches.

Further research on the application of semantic vector analysis tools to speech production in schizophrenia are warranted. While we relied on data from a commonly used test of verbal fluency, such methods could be applied to a range of speaking (or writing) tasks. If further replications confirm our results, tools such as CoVec could represent an efficient, objective, and highly accurate tool for measuring and monitoring derailment and potentially other aspects of FTD.

Highlights.

• Automated tools could be useful in measuring aspects of formal thought disorder in schizophrenia and related disorders, including derailment and tangentiality.

• This study, among first-episode psychosis patients, analyzed animal lists from a 60-second semantic fluency task using CoVec.

• First-episode psychosis patients with derailment had significantly lower CoVec-derived Coherence-5 and Coherence-10 values compared with patients without derailment.