Abstract
Introduction
Formal thought disorder is a hallmark feature of schizophrenia in which disorganized thoughts manifest as disordered speech. A dysfunctional semantic system and a disruption in executive functioning have been proposed as possible mechanisms for formal thought disorder and verbal fluency impairment. Traditional rating scales and neuropsychological test scores might not be sensitive enough to distinguish among types of semantic impairments. This has lead to the proposed used of a natural language processing technique, Latent Semantic Analysis (LSA), which offers improved semantic sensitivity.
Method
In this study, LSA, a computational, vector-based text analysis technique to examine the contribution of vector length, an LSA measure related to word unusualness and cosines between word vectors, an LSA measure of semantic coherence to semantic and phonological fluency, disconnectedness of speech, and adaptive functioning in 165 older inpatients with schizophrenia.
Results
In stepwise regressions word unusualness was significantly associated with semantic fluency and phonological fluency, disconnectedness in speech, and impaired functioning, even after considering the contribution of pre-morbid cognition, positive and negative symptoms, and demographic variables.
Conclusions
These findings support the utility of LSA in examining the contribution of coherence to thought disorder and the its relationship with daily functioning. Deficits in verbal fluency may be an expression of underlying disorganization in thought processes.
Keywords: schizophrenia, verbal fluency, thought disorder, cognitive functioning, adaptive functioning
1. Introduction
Formal thought disorder refers to a disorganization of thought that manifests in communication abnormalities. Disordered speech has long been recognized as one of the core features of schizophrenia (Bleuler, 1911; Kraepelin, 1919) occurring in 20-50% of patients with schizophrenia (Andreasen and Black, 2005; Breier and Berg, 2003). These symptoms are stable across the course of the disorder, are not simply a measurement artifact arising from psychotic symptoms (Harvey, Docherty, Serper, Rasmussen, 1990), and have been identified as predictors of poor functional outcome (Bowie and Harvey, 2008; Keefe et al., 1987).
Disordered speech may manifest in various forms, often classified as either “positive” or “negative” thought disorder. Positive thought disorder is characterized by disorganized or disconnected speech, such as loose associations between concepts evident in the switching of topics disjointedly, providing tangential responses, using nonsense words, or circumstantial speech patterns. Negative thought disorder is characterized by a reduction in the amount of elaboration, or overall verbal output. Typically, these subtypes are referred to as disconnected speech and verbal underproductivity, respectively. These subtypes are distinct from one another (Andreasen, 1979a; Harvey et al., 1992), from positive and negative symptoms in recent-onset, neuroleptic-naïve (John et al., 2003) and chronically ill (White et al., 1997) samples, and are differentially associated with social functioning competencies and behaviors (Bowie et al., 2011). Disordered speech is typically rated using instruments such as the Scale for the Assessment of Thought, Language, and Communication (TLC; Andreasen, 1979a, 1979b), allowing for a clinical rating of a disordered thought phenomenon that manifests in conversational speech.
Various pathogenic neurocognitive mechanisms have been proposed to underlie disconnected speech wherein the two most prominent approaches are variants of executive dysfunction and impaired processing of semantic information and memory (Kerns and Berenbaum, 2002; Barrera et al., 2005; Stirling et al., 2006). Executive functioning is proposed to play a role in maintaining a topic of conversation, planning upcoming speech, and inhibition of inappropriate or unrelated discourse. Alternatively, disruption of semantic processing and memory may be due to a loss of stored information or impaired retrieval (Rapp and Caramazza, 1993), manifesting in difficulties retrieving correct words resulting in diffluent and nonsensical speech. There is currently no clear cognitive mechanism, though it stands to reason that it may involve a combination of semantic information processing and executive functioning (Kerns and Berenbaum, 2002; Doughty & Done, 2009).
One promising area utilizing disordered speech in understanding the role of thought disorder and cognition in schizophrenia is that of verbal fluency, specifically measures of semantic and phonological fluency. During these tasks participants are given a specific production rule, such as generating words that belong to a specific category for semantic fluency or words that begin with a specific letter for phonological fluency in a predetermined amount of time (Benton, 1968; Lezak, 1995). Verbal fluency tasks have been shown to be a sensitive indicator of general brain dysfunction (Lezak, 1995) and are thought to include semantic processing skills including controlled retrieval, semantic memory, and context processing (Docherty et al., 2011). Further, they are considered to place comparable demands on executive processing because of the requirements of efficient organization, self-monitoring of previous responses, and inhibition of inappropriate responses (Crawford and Henry, 2005; Ruff et al., 1997).
Previous research demonstrates that symptoms, and demographic characteristics such as level of education, premorbid ability, and chronological age (Crawford et al., 1992; Benton et al., 1994; Spreen and Strauss, 1998) play a role in the severity of thought disorder and verbal fluency deficits. Similar to other tests of cognitive functioning, performance on verbal fluency tasks is related to the presence of negative symptoms, but not positive symptoms (Howanitz et al., 2000; Kerns et al., 1999). Indeed, patients with more severe negative symptoms generate fewer words overall (Allen et al., 1993). In older samples, patients who have a more chronic course of illness demonstrate more impairments than younger patients (Harvey et al., 1997) and there is a pattern of longitudinal worsening in verbal productivity that is predicted by concurrent cognitive worsening (Bowie et al., 2005).
Disconnected speech has been found to be associated with production of fewer contextually-accurate semantic word clusters in a semantic fluency task (Kerns et al., 1999) and worse performance on a phonological fluency task (Aloia, 1998). Recently, Docherty et al (2011) found that that there may be differential patterns of performance on verbal fluency tasks for individuals with alogia (i.e., underproductive speech) and formal thought disorder (i.e., disconnected speech). Specifically, disconnected speech was related to the proportion of semantically related words uttered in a phonological fluency task (Docherty et al., 2011). In studies comparing healthy older adults to younger adults, elderly participants generate fewer total words and fewer category switches on a semantic fluency task, and larger cluster sizes (i.e., greater number of semantically related words in a semantic cluster prior to shifting to a new semantic cluster) on a phonological fluency task (Troyer et al., 1997).
Examinations of factors associated with social and adaptive functioning in schizophrenia repeatedly reveal cognition as the most robust predictor (Green, 1996; Harvey et al., 1998). Specifically concerning verbal fluency, previous studies have found performance on verbal fluency tasks (measured by total number of words) to be correlated with impairments in functioning in individuals with schizophrenia (Green et al., 2000), a finding which holds even after accounting for negative symptoms (Jaeger et al., 2003), and is evident in geriatric patients (Bowie et al., 2004). Moreover, verbal fluency has been shown to be an important predictor of daily problem solving skills both in a testing setting (Keefe et al., 2006; Revheim et al., 2006) and in a real-world context (Rempfer et al., 2003).
While it is evident that performance on verbal fluency tasks is associated with severity of disconnected speech, it remains unclear how unique components of incoherent discourse specifically result in poor performance in these tasks, and the functional impact of underlying dysfunctional semantic and executive systems. A promising recent development for the evaluation of discourse in schizophrenia utilizes the ability of Latent Semantic Analysis (LSA; Landauer et al., 1998) to provide automated measures of semantic relatedness. LSA is a statistical, cognitive modeling and computational tool that quantifies the semantic relatedness of words and groupings of words in a high-dimensional semantic space. In this semantic space, individual words are represented as vectors and the cosine between word vectors provides a measure of the semantic similarity between words, providing a measure of coherence (Elvevåg et al., 2007; Foltz et al., 1998). Average coherence refers to the average of all cosine values between each word and the word that immediately follows that word (e.g., the average of the cosine between words A and B, B and C, C and D etc, where A is the first and D is the last word uttered in a sequence). The vector length of any individual word provides a measure of the information value of a word (i.e., a word's unusualness). Vector length is an index of unusualness in the sense that low frequency words convey more information value (and hence have a higher vector length). LSA offers an automated, finer grained method to evaluate discourse organization than is often available from other more subjective or observational measures of thought disorder (McKenna and Oh, 2005). Applying LSA to the set of words spoken during a verbal fluency task allows for a more nuanced approach to evaluating fluency that is sensitive to the specific relationships between words uttered rather than simply being a metric that is contingent upon the total number of words produced as is the case with typical tests of verbal fluency. Further, given that speech disconnectedness is a clinical measure of an inaccessible phenomenon (i.e., the content and organization of thoughts), LSA affords an opportunity to explicitly evaluate underlying cohesiveness of thoughts.
In light of the exploratory nature of LSA, this study serves to provide an overview of the psychometric properties of latent semantic variables and the nature of their relationships to thought disorder and functioning. As such, we examined the contribution of two LSA derived constructs, namely word unusualness and coherence to semantic fluency, as well as phonological fluency, clinically rated disconnected speech, and real-world adaptive functioning. We hypothesized that average word unusualness and average coherence would be associated with semantic and phonological fluency, disconnected speech during interpersonal conversation, and adaptive functioning on an inpatient ward, even when accounting for the contribution of premorbid cognition, positive and negative symptoms, and demographic variables (level of education and age).
2. Methods
2.1. Participants
Participants in this study were part of a longitudinal study of chronic schizophrenia described in detail elsewhere (Bowie, et al., 2004; Davidson, et al., 1995). All participants were residing in a state psychiatric inpatient facility or in a restricted nursing home setting. Institutional Review Boards at Pilgrim Psychiatric Center and the Mount Sinai School of Medicine approved a waiver of signed informed consent in lieu of the participants’ verbal assent witnessed by a staff member. A structured consensus procedure was employed to establish a diagnosis of schizophrenia (see Harvey et al., 1998 for additional details).
2.2. Measures
2.2.1. Positive and Negative Symptoms
Severity of clinical symptoms was assessed with the Positive and Negative Syndrome Scale (PANSS; Kay, 1991). Following a structured interview, seven positive symptoms, seven negative symptoms, and sixteen general aspects of psychopathology were rated on a 7-point Likert scale. We used the total scores on the positive and negative domains from the empirically derived 5-factor model of the PANSS (White et al., 1997). Inter-rater reliability of these ratings in the present sample was previously found to be acceptably high (Davidson et al., 1995), with intra-class correlations (ICC; n = 30) attained during training procedures of ranging from .86 to 1.00 (all p's < .001).
2.2.2. Premorbid cognition
The Wide-Range Achievement Test-Revised (WRAT-R; Jastak, 1984) was administered in order to obtain an estimate of premorbid functioning. The WRAT word recognition reading subtest was administered as a measure of putative premorbid cognition. This component requires participants to read a standardized list of words out loud to the tester. The dependent measure is the total score (i.e., total number of words correctly pronounced).
2.2.3. Formal Thought Disorder
Thought disorder was rated with the Scale for Assessment of Thought, Language, and Communication (TLC; Andreasen, 1979a, 1979b). TLC ratings were based on 5 to 10 minutes of unscripted conversations conducted prior to other assessments. The TLC provides definitions and rating criteria for 18 subtypes of communication abnormalities. Ratings are made on a Likert scales ranging from 0 (absent) to 3 (severe) or 0 (absent) to 4 (extreme). For the purposes of these analyses, we examined the disconnected speech subtype, previously identified with confirmatory factor analysis (Harvey et al., 1992) as there is a relative paucity of speech production in negative thought disorder (i.e., underproductive factor) whereas individuals with disconnected speech would demonstrate an inability to stay on topic. Disconnected speech is the mean of the following TLC items: Derailment, Tangentiality, Loss of Goal, Circumstantiality, and Incoherence. This scale has been found to have acceptable reliability (ICC=.86).
2.2.4. Semantic (category) and phonological (letter) fluency tasks
In the semantic fluency task participants were required to name as many different animals as possible in the span of 90 secs. For the phonological fluency task participants reported as many words starting with a F, A, and S for one minute each (Spreen and Strauss, 1998). The dependent variable for both tasks was the total number of correct responses. Latent semantic analyses were applied to only the semantic fluency task.
2.2.5. Adaptive functioning
The Clinical Dementia Rating Scale (CDRS; Berg, 1988) is devised to evaluate the severity of dementia. It was used in this project due to the severity of cognitive and functional impairments in older, chronically institutionalized schizophrenia patients. The CDRS includes domains of functioning including judgment and problem solving, community affairs, ward life, and personal care. Each domain is rated on a 5, 6, or 7-point scale (0 = absent; 0.5 = questionable; 1 = mild; 2 = moderate; 3 = severe; 4 = profound; 5 = terminal), wherein scoring is completed by a clinician following a chart review and a semi-structured interview with the patient. The CDRS also includes an overall rating of dementia severity by the clinician. Previous reports suggest that the interrater reliability of the CDRS is acceptable (Berg, 1988; Berg et al., 1988) and that scores are consistently associated with performance-based assessments of cognitive functioning in this population (Harvey et al., 1999).
2.3. Data Analysis
2.3.1. Latent Semantic Analysis (LSA)
LSA is a statistical, cognitive modeling tool and computational technique for matching discourse content (Elvevåg et al., 2007; Landauer, et al., 1998). It is based on the premise that words used in similar contexts tend to have more similar meaning than words usedin different contexts, and derives a measure of the similarity of words to each other based on an analysis of their contextual usage across large text corpora. This approach has been successfully used to quantify incoherent or aberrant speech in schizophrenia (Elvevåg et al., 2007). Elvevåg et al. demonstrated the applicability of this methodology to measure incoherent discourse in a manner that bypasses problems associated with subjective ratings of frequency and severity of communication abnormalities, a common limitation among research on formal thought disorder. Their findings demonstrate that LSA is a sensitive measure capable of reliably analyzing communication abnormalities in schizophrenia.
The initial step before applying LSA is to transcribe the audio recordings into a text format, such that each response serves as its own “passage”. Next, LSA acquires a representation of semantic relationships among words (or any passage of text) by an automated analysis of millions of words of natural discourse. This analysis involves deriving the relationships among word and passage meanings using singular value decomposition (a matrix algebra technique similar to factor analysis). In this view of the text, the meaning of a word is contingent upon all the contexts in which it appears. LSA is highly sensitive to underlying relationships among words such that text vectors that share semantic content by containing terms with similar meanings may be rated highly similar (cosine between the text vectors near 1.0) regardless of the differences in the literal words used. For instance the vector representing: “Doctors operate on patients” has a cosine of 0.81 with the vector representing “Physicians do surgery”. Conceptually, both sentences express similar meaning despite using different words to convey the message. Thus, LSA is not simply a measure of semantic-relatedness based on contiguity frequencies and co-occurrence contingencies, but rather a measure capable of inferring latent relationships between words in a given text passage (Elvevåg et al., 2007; Landauer, et al., 1998). The present research used an LSA-semantic space from the TASA corpus containing 44,486 unique documents by 98,646 unique terms with 300 dimensions (see http://lsa.colorado.edu) in order to calculate the average vector length (word unusualness) of each response and the average cosine coherence between each response and the next.
To the extent that fluency tasks are a proxy measure of fluent discourse (and underlying organization), the LSA theory-driven coherence measures are sensitive assays of subtle differences in words produced in verbal fluency tasks. Participants’ responses from the semantic fluency task were transcribed and LSA was subsequently applied to each participant's answers. For each word uttered in the semantic fluency task its vector length was computed and for every pair of sequential words the cosine between the vectors for those words was computed. The average for the first set generated the average vector length (word unusualness measure) and the average of the cosines generated the average cosine (coherence measure) for each participant. In order to adjust for the relationship between coherence and total words uttered in the semantic fluency task, total number of words for the semantic fluency task was regressed onto the average cosine value. Applying LSA to verbal fluency tasks proffer the following advantages: (1) LSA is not specifically contingent upon the number of words uttered, thus there is little impact on the reliability of these data in light of the limited words that can be produced in the span of 90 seconds, (2) LSA is sensitive to underlying semantic relatedness of words spoken during a task in which the goal is to generate words belonging to a specific category (i.e., animals). Thus, the application of LSA to a verbal fluency task with constricted rules for output may especially sensitive to underlying suboptimal semantic organization structures in individuals with schizophrenia as the task dictates the degree to which participants can deviate from semantically related verbal output.
3. Results
3.1 Descriptive Characteristics and Correlations
165 native English speaking geriatric schizophrenia patients (29.6% males; 80% Caucasian, 16% Black, 4% not reported) residing in long-stay institutional settings (restricted living adult homes; n=27) or state psychiatric facilities (n=138) were included in these analyses. They were recruited as part of a longitudinal study of cognition and functioning in geriatric schizophrenia. See Table 1 for descriptive statistics, Table 2 for test performance scores, and Table 3 for percentage of sample expressing TLC subtypes subsumed under disconnected speech. The correlation between LSA variables, average cosine and average vector length, was r (165) = .32, p < .001. Further, average vector length, negative symptoms, and age share the strongest relationships with performance on the verbal fluency tasks and with adaptive functioning as measured by the CDRS. Please see Tables 4 and 5 depicting correlation coefficients between symptom, demographic, speech, and adaptive functioning variables.
Table 1.
Mean | SD | Range | |
---|---|---|---|
Age (years) | 66.03 | 12.27 | 31.41-93.18 |
Highest education (n=153) | 11.59 | 2.36 | 4-18 |
Age at first hospitalization (n=118) | 26.81 | 10.97 | 12-66 |
Table 2.
Mean | SD | Range (maximum) | |
---|---|---|---|
Estimate of premorbid cognition (n=153) | |||
WRAT Total Raw score | 51.55 | 17.94 | 2-95 (100) |
Verbal Fluency | |||
Semantic fluency (1.5 minutes; n=162) | 10.17 | 4.04 | 2-24 (N/A) |
Phonological fluency (3 minutes; n=153) | 19.75 | 11.19 | 2-70 (N/A) |
Symptoms (n=162) | |||
PANSS Positive factor | 19.01 | 6.81 | 7-39 (49) |
PANSS Negative factor | 20.33 | 6.80 | 7-39 (49) |
Adaptive Functioning (n=162) | |||
CDRS Judgment & problem solving | 1.15 | .70 | 0-3 (3) |
CDRS Community affairs | .96 | .74 | 0-3 (3) |
CDRS Ward life & hobbies | 1.06 | .76 | 0-3 (3) |
CDRS Personal care | 1.09 | .95 | 0-4 (5) |
Table 3.
Absent (%) | Mild (%) | Moderate (%) | Severe (%) | Extreme (%) | |
---|---|---|---|---|---|
Tangentiality | 46.70 | 16.40 | 20.60 | 12.70 | 3.60 |
Derailment | 54.50 | 15.20 | 17.60 | 9.70 | 3.00 |
Incoherence | 80.00 | 8.50 | 8.50 | 1.20 | 1.80 |
Circumstantiality | 66.70 | 20.00 | 11.50 | 1.80 | N/A |
Loss of Goal | 58.20 | 20.60 | 17.00 | 4.20 | N/A |
Note: Absent = No instances over whole interview; Mild = occurs once or occasionally; Moderate = two to four times; Severe = 5 to 10 times or in the case of circumstantiality, many such that this pattern of responding continues despite being interrupted by examiner; Extreme = more than 10 times, interview is largely incomprehensible).
Table 4.
PANSS positive | PANSS negative | Highest education | Chronological age | WRAT Total | Average vector length | Cosine | |
---|---|---|---|---|---|---|---|
Semantic Fluency | .17* | -.14 | .06 | -.38** | .21* | -.26** | -.06 |
Phonological fluency | .12 | -.18* | .07 | -.37** | .25** | -.19** | -.01 |
Disconnected speech | .37** | -.18* | .04 | -.14 | .05 | -.20** | -.13 |
significant at the .05 level
significant at the .01 level
Table 5.
Average vector length | Cosine | PANSS positive | PANSS negative | Highest education | Age at evaluation | WRAT Total | |
---|---|---|---|---|---|---|---|
CDRS | |||||||
Judgment & problem solving | .09 | .03 | -.06 | .40** | .02 | .24** | -.10 |
Community affairs | .22** | .12 | .02 | .44** | .05 | .23** | -.07 |
Ward life & hobbies | .16* | .04 | -.01 | .55** | .03 | .09 | -.07 |
Personal care | .19* | .10 | -.14 | .33** | .09 | .31** | -.04 |
Note: Higher scores on the CDRS are indicative of worse functioning.
significant at the .05 level
significant at the .01 level
3.2. Regressions
Hierarchical stepwise regressions were conducted to examine predictors or factors associated with dependent variables. We were interested in examining the contribution of LSA derived variables in explaining variance in dependent variables after accounting for typical related factors. Thus, premorbid cognition, demographic variables (age and education), and symptom variables (positive and negative symptoms) were forced into Block 1, followed by average vector length (word unusualness) and cosine (coherence) in Block 2.
Higher number of correct responses on the semantic fluency task was significantly predicted associated with by older age at evaluation, F(1, 131) = 21.88, p < .001, R2Δ = .14, negative symptoms, F(2, 130) = 13.99, p < .001, R2Δ = .03, premorbid cognition, F(3,129) = 10.87, p < .001, R2Δ = 03, and average vector length, F(4, 128) = 11.27, p < .001, R2Δ = 06. Age at evaluation, education, positive symptoms, negative symptoms, and cosine did not enter the equation (all ps >.05).
Better performance on the phonological fluency task was significantly associated with older age at evaluation, F(1,138) = 23.88 p < .001, R2Δ = .15, premorbid cognition, F(2,137) = 19.36, p < .001, R2Δ = .07., and positive symptoms, F(3,136) = 14.98 p < .001, R2Δ = .03. Education, negative symptoms, average vector length, and cosine did not enter the equation (all ps >.05).
Higher disconnectedness during natural speech (i.e., more severe clinically rated positive thought disorder on the TLC) was significantly associated with positive symptoms, F(1,141) = 19.72, p < .001, R2Δ = .12, negative symptoms, F(2, 140) = 12.58, p < .001, R2Δ = .03, and average vector length, F(3,139) = 11.23, p < .001, R2Δ = .04. Premorbid cognition, age at evaluation, education, and cosine did not enter the equation (all ps >.05).
Greater impairment in judgment and problem solving on the CDRS were associated with negative symptoms, F(1,141) = 29.57, p < .001, R2Δ = .17, and older age at evaluation, F(2,140) = 22.06, p < .001, R2Δ = .07. Premorbid cognition, education, positive symptoms, average vector length, and cosine did not enter the equation (all ps >.05).
Severity of deficits in community affairs on the CDRS were associated with negative symptoms, F(1,141) = 29.42, p < .001, R2Δ = .17, older age at evaluation, F(2,140) = 20.14, p < .001, R2Δ = .05, average vector length, F(3,139) = 16.27, p < .001, R2Δ = .04, and cosine (coherence), F(4,138) = 14.16, p < .001, R2Δ = 03. Premorbid cognition, education, and positive symptoms did not enter the equation (all ps >.05).
More severe impairment in ward life and hobbies on the CDRS was associated with negative symptoms, F(1,141) = 58.03, p < .001, R2Δ = .29, and average vector length, F(2,140) = 32.08, p < .001, R2Δ = .02. Premorbid cognition, age at evaluation, education, positive symptoms, and cosine did not enter the equation (all ps >.05).
Higher impairment in personal care on the CDRS was associated with negative symptoms, F(1,141) = 19.60, p < .001, R2Δ = .12, older age at evaluation, F(2,140) = 14.99, p < .001, R2Δ = .09, and average vector length, F(3,139) = 14.03, p < .001, R2Δ = .03. Premorbid cognition, education, positive symptoms, and cosine did not enter the equation (all ps >.05).
4. Discussion
Formal thought disorder is a pervasive symptom of schizophrenia (Bleuler, 1911; Kraepelin, 1919); it is associated with impairment in functional outcome (Bowie and Harvey, 2008) and is experienced by a large majority of individuals with schizophrenia (Andreasen and Black, 2005; Brier and Berg, 1999). Yet, there is little understanding of the discrete components of disorganized thought and how these are specifically related to functional outcome. The goal of the present study was to evaluate the relationship between discrete components of discourse to disconnected speech, semantic fluency, phonological fluency, and adaptive functioning in geriatric schizophrenia. The larger implications speak to the degree to which computational techniques may prove to be effective tools in the study of thought disorder and how variables derived from LSA coalesce with evidence to date on the role of disordered prose as it relates to adaptive functioning. Findings suggest that measures of latent semantic structure of speech differentially associated with disorganized speech, performance on verbal fluency tasks, and adaptive functioning over and above demographic, clinical, and premorbid cognition variables.
Coherence was not related to performance on either verbal fluency tasks, but semantic fluency was associated with average vector length. Furthermore, both verbal fluency tasks shared a significant negative correlation with average vector length. This may suggest that individuals who demonstrate difficulties in retrieval of words related to a specific demand (e.g., verbal fluency task) fail to generate “usual” verbal responses thereby hindering their performance in quickly generating many “correct” or “high frequency” words on verbal fluency tasks. Semantic fluency was also associated with older age at evaluation, negative symptoms, premorbid cognition, and average vector length, whereas phonological fluency was only related to age at testing, premorbid cognition, and positive symptoms. Previous research has found that negative symptoms are related to verbal fluency (Kerns et al., 1999; O'Leary et al., 2000), while a recent meta-analysis suggests that negative symptoms are more strongly correlated with phonological fluency than semantic fluency (Henry and Crawford, 2005). Given the similar magnitude of the correlations between verbal fluency tasks and negative symptoms, it is likely this discrepancy emerged as a function of sample size rather than differential relationships among the verbal fluency tasks and negative symptoms.
Only word unusualness emerged as a significant factor related to disconnected speech, after considering premorbid cognition and demographic variables. Positive symptoms, negative symptoms, and word unusualness were associated with disconnected speech. Positive symptoms included items that overlap with thought disorder, and previous reports support this relationship (Bowie et al., 2004). The finding of vector length (word unusualness) as related factor is in line with a study by Baskak et al (2008) that found unusual word use to be associated with the severity of formal thought disorder. A previous study by Kuperberg et al (1998) found that individuals with positive thought disorder experienced more intrusions, as it may be the case that individuals with more severe positive thought disorder may generate more semantically unrelated words (Docherty et al., 2011). Thus, we had predicted that coherence (specifically, lower coherence) would be associated with of disconnected speech. While coherence did not emerge as a significant factor related to disconnected speech, the negative correlation between the two suggests that individuals who produce more disconnected speech demonstrated lower coherence on the verbal fluency task. Furthermore, positive thought disorder is inversely correlated with activity in the Wernicke area, a region implicated in the production of coherent speech (Kircher et al., 2001). These findings are consistent with previous evidence suggesting that there is a relationship between aspects of verbal fluency performance and thought disorder (Kerns et al., 1999). A possible explanation for this relationship is the overlapping neural areas responsible for performance on verbal fluency tasks and thought disorder, wherein both are strongly associated with temporal and frontal lobe regions (Artiges et al., 2000; Curtis et al., 1998; Petty et al., 1995).
It has been proposed that verbal fluency tests provide a reliable means of finding out whether and how well people organize their thinking (Estes, 1974), thus, it stands to reason that specific features of latent disorganization may be expressed in disorganized behavior and inability to execute functional behavior. In the present study, adaptive functioning behaviors were associated with negative symptoms, age, average word unusualness, and coherence. Negative symptoms frequently play a large role in functional disability and poor overall outcome (Perlick et al., 1992; Velligan et al., 1997) and adaptive functioning impairments in older people with schizophrenia in both inpatients (McGurk et al., 2000) and outpatients (Leifker et al., 2009). Furthermore, these findings coalesce with previous reports that performance deficits in verbal fluency have functional importance as they are related to negative symptoms, lower number of social contacts (Hanuszkiewicz et al., 2009), worse performance on daily problem solving tasks (Rempfer et al., 2003), and poorer social-adaptive functioning in hospitalized geriatric schizophrenia patients (Bowie et al., 2004). Concerning the LSA variables, cosine was only associated with community behaviors, yet coherence was related to worse functioning. We speculate that this is because the LSA measure of coherence in this study was confounded by the fact that as individuals generated more words on the semantic fluency tasks (from which the LSA variables were derived) they produced more clusters (i.e., categories) of words, which inadvertently reduced their average coherence. Thus, we suggest that word unusualness (i.e., average vector length) emerges as having a stronger association with functioning because it is related to a generation of less frequently used verbal responses under certain conditions. In this regard, we suggest that uttering infrequently utilized words (which may be due to difficulties retrieving “usual words” or simply being “creative” in words generated during the task) may actually detract from overall task performance. One interpretation is that these results may speak to a possible general disorganization of semantic networks in retrieval of high frequency words and appears to be related to difficulties in execution of functional behavior.
This study has several limitations that should be taken into consideration, some of which may hinder these results from being generalized to other samples. Firstly, this study documents the performance of older, chronically ill patients, thus these findings may not readily translate to younger or more acutely ill individuals. It should be noted that cognition and functioning were reported to have the same global correlations in acutely ill and institutionalized older patients (Harvey et al., 1998). However, this age-range is also a unique strength of the current study as previous literature demonstrates that there are age-based differences in performance on verbal fluency tasks (Bowie et al., 2005; Harvey, et al., 1997; Troyer et al., 1997) highlighting the importance of considering differential predictors of performance across the lifespan. A comprehensive assessment of cognitive functioning was not employed in this study, thereby precluding the possibility of comprehensively evaluating the contribution of underlying neurocognitive mechanisms that contribute to the pathogenesis of disorganized discourse. We have published extensively on this sample in the past. Future studies may consider employing a neurocognitive battery, with a specific focus on domains related to verbal fluency performance and thought disorder including working memory (Ojeda et al., 2010), processing speed (Ojeda et al., 2008), and executive functioning (Kerns and Berenbaum, 2002). Finally, we applied Latent Semantic Analysis to a semantic fluency task that was only 90 secs in duration as opposed to natural speech. As such, the potential for unveiling more detailed underlying disorganization is limited by the constriction of specific rules governing the semantic fluency task and the total number of words one can produce in a 90 second timeframe. Future studies may benefit from applying computational techniques such as LSA to natural discourse to determine whether the latent disorganization differs in less constricted speech and how this relates to functional outcome. Furthermore, previous research has demonstrated that there is much variability, both inter- and intra-individually, on verbal fluency tasks (Elvevåg and Storms, 2003), suggesting that conclusions drawn from these data may shift depending on individual difference factors and the stability of thought disorder's impact on discourse across multiple test points.
This study demonstrates the association between disorganized discourse and adaptive functioning, and provided an overview of the psychometric properties of a computational language approach to studying disordered speech. Given the pervasive course of thought disorder in schizophrenia, these findings underscore the importance of developing efficacious treatment for thought disorder and using Latent Semantic Analysis to index change in communication patterns as a function of treatment.
Acknowledgements
Thomas Coleman for help with data collection. We also thank Mark Rosenstein for useful discussions concerning semantic coherence. This research was supported by NIMH Grant Number MH 63116 to Dr. Harvey, the Mount Sinai Silvio Conte Neuroscience Center (NIMH MH 36692; KL Davis PI). Support to Dr. Bowie is provided by the Canadian Institutes for Health Research and Ontario Ministry for Research. Dr. Elvevåg was supported financially by the Northern Norwegian Regional Health Authority (Helse Nord RHF). These sponsors had no role in study design, collection of data, data analysis, interpretation of results, or writing of the manuscript.
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
References
- Allen HA, Liddle PF, Frith CD. Negative features, retrieval processes and verbal fluency in schizophrenia. British Journal of Psychiatry. 1993;163:769–775. doi: 10.1192/bjp.163.6.769. [DOI] [PubMed] [Google Scholar]
- Aloia MS, Gourovitch ML, Missar D, Pickar D, Weinberger DR, Goldberg TE. Cognitive substrates of thought disorder, II: specifying a candidate cognitive mechanism. American Journal of Psychiatry. 1998;155(12):1677–1684. doi: 10.1176/ajp.155.12.1677. [DOI] [PubMed] [Google Scholar]
- Andreasen NC. Thought, language, and communication disorders. I. Clinical assessment, definition of terms, and evaluation of their reliability. Archives of General Psychiatry. 1979a;36(12):1315–1321. doi: 10.1001/archpsyc.1979.01780120045006. [DOI] [PubMed] [Google Scholar]
- Andreasen NC. Thought, language, and communication disorders. II. Diagnostic significance. Archives of General Psychiatry. 1979b;36(12):1325–1330. doi: 10.1001/archpsyc.1979.01780120055007. [DOI] [PubMed] [Google Scholar]
- Andreasen NC, Black DW. Introductory Textbook of Psychiatry. American Psychiatric Association; Washington, DC: 2005. [Google Scholar]
- Artiges E, Martinot JL, Verdys M, Attar-Levy D, Mazoyer B, Tzourio N, Giraud MJ, Paillere-Martinot ML. Altered hemispheric functional dominance during word generation in negative schizophrenia. Schizophrenia Bulletin. 2000;26(3):709–721. doi: 10.1093/oxfordjournals.schbul.a033488. [DOI] [PubMed] [Google Scholar]
- Barrera A, McKenna P, Berrios G. Formal thought disorder in schizophrenia: An executive or a semantic deficit? Psychological Medicine. 2005;35(1):121–132. doi: 10.1017/s003329170400279x. [DOI] [PubMed] [Google Scholar]
- Baskak B, Ozel ET, Atbasoglu EC, Baskak SC. Peculiar word use as a possible trait marker in schizophrenia. Schizophrenia Research. 2008;103(1-3):311–317. doi: 10.1016/j.schres.2008.04.025. [DOI] [PubMed] [Google Scholar]
- Benton AL. Differential behavioural effects in frontal lobe disease. Neuropsychologia. 1968;6(1):53–60. [Google Scholar]
- Benton AL, Sivan AB, Hamsher K deS, Varney NR, Spreen O. Contributions to neuropsychologial assessment. A clinical manual. 2nd ed. Oxford University Press; New York: 1994. [Google Scholar]
- Berg L. Clinical dementia rating (CDR). Psychopharmacological Bulletin. 1988;24(4):637–639. [PubMed] [Google Scholar]
- Berg L, Miller JP, Storandt M, Duchek J, Morris JC, Rubin EH, Burke WJ, Coben LA. Mild senile dementia of the Alzheimer Type II. Longitudinal assessment. Annals of Neurology. 1988;23(5):477–484. doi: 10.1002/ana.410230509. [DOI] [PubMed] [Google Scholar]
- Bleuler E. Dementia Praecox; or the Group of Schizophrenias. International Universities Press; New York: 1911. [Google Scholar]
- Bowie CR, Gupta M, Holshausen K. Disconnected and underproductive speech in schizophrenia: Unique relationships across multiple indicators of social functioning. Schizophrenia Research. 2011;131(1-3):152–156. doi: 10.1016/j.schres.2011.04.014. [DOI] [PubMed] [Google Scholar]
- Bowie CR, Harvey PD. Communication abnormalities predict functional outcomes in chronic schizophrenia: differential associations with social and adaptive functions. Schizophrenia Research. 2008;103(1-3):240–247. doi: 10.1016/j.schres.2008.05.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Bowie CR, Harvey PD, Moriarty PJ, Parrella M, White L, Davis KL. A comprehensive analysis of verbal fluency deficit in geriatric schizoophrenia. Archives of Clinical Neuropsychology. 2004;19(2):289–303. doi: 10.1016/S0887-6177(03)00041-6. [DOI] [PubMed] [Google Scholar]
- Bowie CR, Tsapelas I, Friedman J, Parrella M, White L, Harvey PD. The longitudinal course of thought disorder in geriatric patients with chronic schizophrenia. American Journal of Psychiatry. 2005;162(4):793–795. doi: 10.1176/appi.ajp.162.4.793. [DOI] [PubMed] [Google Scholar]
- Breier A, Berg PH. The psychosis of schizophrenia: Prevalence, response to atypical antipsychotics, and prediction of outcome. Biological Psychiatry. 2003;46(3):361–364. doi: 10.1016/s0006-3223(99)00040-2. [DOI] [PubMed] [Google Scholar]
- Crawford JR, Moore JW, Cameron IM. Verbal fluency: A NART-based equation for the estimation of premorbid performance. British Journal of Clinical Psychology. 1992;31(3):327–329. doi: 10.1111/j.2044-8260.1992.tb00999.x. [DOI] [PubMed] [Google Scholar]
- Crawford JR, Henry JD. Assessment of executive deficits. In: Halligan PW, Wade N, editors. The effectiveness of rehabilitation for cognitive deficits. Oxford University Press; London: 2005. [Google Scholar]
- Curtis VA, Bullmore ET, Brammer MJ, Wright IC, Wiliams SC, Morris RG, Sharma TS, Murray RM, McGuire PK. Attenuated frontal activation during a verbal fluency task in patients with schizophrenia. American Journal of Psychiatry. 1998;155(8):1056–1063. doi: 10.1176/ajp.155.8.1056. [DOI] [PubMed] [Google Scholar]
- Davidson M, Harvey PD, Powchik P, Parrella M, White L, Knobler H, Losonczy MF, Keefe RSE, Katz S, Frecska E. Severity of symptoms in chronically institutionalized geriatric schizophrenic patients. American Journal of Psychiatry. 1995;152(2):197–207. doi: 10.1176/ajp.152.2.197. [DOI] [PubMed] [Google Scholar]
- Docherty AR, Berenbaum H, Kerns JG. Alogia and formal thought disorder: Differential patterns of verbal fluency task performance. Journal of Psychiatric Research. 2011;45(10):1352–1357. doi: 10.1016/j.jpsychires.2011.04.004. [DOI] [PubMed] [Google Scholar]
- Doughty OJ, Done DJ. Is semantic memory impaired in schizophrenia? A systematic review and meta-analysis of 91 studies. Cognitive Neuropsychiatry. 2009;14(6):473–509. doi: 10.1080/13546800903073291. [DOI] [PubMed] [Google Scholar]
- Elvevåg B, Foltz PW, Weinberger DR, Goldberg TE. Quantifying incoherence in speech: An automated methodology and novel application to schizophrenia. Schizophrenia Research. 2007;93(1-3):304–316. doi: 10.1016/j.schres.2007.03.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Elvevåg B, Storms G. Scaling and clustering in the study of semantic disruptions in patients with schizophrenia: a re-evaluation. Schizophrenia Research. 2003;63(3):237–246. doi: 10.1016/s0920-9964(02)00331-6. [DOI] [PubMed] [Google Scholar]
- Estes WK. Learning theory and intelligence. American Psychologist. 1974;29(10):740–749. [Google Scholar]
- Foltz PW, Kintsch W, Landauer TK. The measurement of textual coherence with Latent Semantic Analysis. Discourse Processes. 1998;25(2&3):285–307. [Google Scholar]
- Green MF. What are the functional consequences of neurocognitive deficits in schizophrenia? American Journal of Psychiatry. 1996;153(3):321–330. doi: 10.1176/ajp.153.3.321. [DOI] [PubMed] [Google Scholar]
- Green MF, Kern RS, Braff DL, Mintz J. Neurocognitive deficits and functional outcome in schizophrenia: Are we measuring the “right stuff”? Schizophrenia Bulletin. 2000;26(1):119–136. doi: 10.1093/oxfordjournals.schbul.a033430. [DOI] [PubMed] [Google Scholar]
- Hanuszkiewicz I, Cechnicki A, Kalisz A. The relationship between cognitive deficits and the course of schizophrenia. Preliminary research on participants of a rehabilitation programme. Archives of Psychiatry and Psychotherapy. 2009;3:27–34. [Google Scholar]
- Harvey PD, Davidson M, Powchik P, Parrella M, White L, Mohs RC. Assessment of dementia in elderly schizophrenics with structured rating scales. Schizophrenia Research. 1992;7(1):85–90. doi: 10.1016/0920-9964(92)90078-j. [DOI] [PubMed] [Google Scholar]
- Harvey PD, Docherty NM, Serper MR, Rasmussen M. Cognitive deficits and thought disorder: II. An 8-month followup study. Schizophrenia Bulletin. 1990;16(1):147–156. doi: 10.1093/schbul/16.1.147. [DOI] [PubMed] [Google Scholar]
- Harvey PD, Howanitz E, Parrella M, White L, Davidson M, Mohs RC, Hoblyn J, Davis KL. Cognitive, adaptive, and clinical symptoms in geriatric patients with lifelong schizophrenia: A comparative study across treatment sites. American Journal of Psychiatry. 1998;155(8):1080–1086. doi: 10.1176/ajp.155.8.1080. [DOI] [PubMed] [Google Scholar]
- Harvey PD, Lenzenweger MF, Keefe RSE, Pogge DL, Serper MR, Mohs RC. Empirical assessment of the factorial structure of clinical symptoms in schizophrenic patients: Formal thought disorder. Psychiatry Research. 1992;44(2):141–151. doi: 10.1016/0165-1781(92)90048-8. [DOI] [PubMed] [Google Scholar]
- Harvey PD, Lombardi J, Leibman M, Parrella M, White L, Powchick P, Mohs RC, Davidson M, Davis KL. Age-related differences in formal thought disorder in chronically hospitalized schizophrenic patients: a cross-sectional study across nine decades. American Journal of Psychiatry. 1997;154(2):205–210. doi: 10.1176/ajp.154.2.205. [DOI] [PubMed] [Google Scholar]
- Harvey PD, Parrella M, White L, Mohs RC, Davidson M, Davis KL. Convergence of cognitive and adaptive decline in late-life schizophrenia. Schizophrenia Research. 1999;35(1):77–84. doi: 10.1016/s0920-9964(98)00109-1. [DOI] [PubMed] [Google Scholar]
- Henry JD, Crawford JR. A meta-analytic review of verbal fluency deficits in schizophrenia relative to other neurocognitive deficits. Cognitive Neuropsychiatry. 2005;10(1):1–33. doi: 10.1080/13546800344000309. [DOI] [PubMed] [Google Scholar]
- Howanitz E, Cicalese C, Harvey PD. Verbal fluency and psychiatric symptoms in geriatric schizophrenia. Schizophrenia Research. 2000;42(3):167–169. doi: 10.1016/s0920-9964(99)00137-1. [DOI] [PubMed] [Google Scholar]
- Jaeger J, Czobor P, Berns SM. Basic neuropsychological dimensions in schizophrenia. Schizophrenia Research. 2003;65(2-3):105–116. doi: 10.1016/s0920-9964(03)00052-5. [DOI] [PubMed] [Google Scholar]
- Jastak S. The Wide-Range Achievement Test- Revised. Jastak Associates; Wilmington, DE: 1984. [Google Scholar]
- John JP, Khanna S, Thennarasu K, Reddy S. Exploration of dimensions of psychopathology in neuroleptic-naïve patients with recent-onset schizophrenia/schizophreniform disorder. Psychiatry Research. 2003;121(1):11–20. doi: 10.1016/s0165-1781(03)00199-9. [DOI] [PubMed] [Google Scholar]
- Kay SR. Positive and Negative Syndromes in Schizophrenia. Brunner/Mazel; New York, NY: 1991. [Google Scholar]
- Keefe RS, Mohs RC, Losonczy MF, Davidson M, Silverman JM, Kendler KS, Horvath TB, Nora R, Davis KL. Characteristics of very poor outcome in schizophrenia. American Journal of Psychiatry. 1987;144(7):889–895. doi: 10.1176/ajp.144.7.889. [DOI] [PubMed] [Google Scholar]
- Keefe RS, Bilder RM, Harvey PD, Davis SM, Palmer BW, Gold JM, Meltzer HY, Green MF, Miller del D, Canive JM, Adler LW, Manschreck TC, Swartz M, Rosenheck R, Perkins DO, Walker TM, Stroup TS, McEvoy JP, Lieberman JA. Baseline neurocognitive deficits in the CATIE schizophrenia trial. Neuropsychopharmacology. 2006;31(9):2033–204. doi: 10.1038/sj.npp.1301072. [DOI] [PubMed] [Google Scholar]
- Kerns JG, Berenbaum H. Cognitive impairments associated with formal thought disorder in people with schizophrenia. Journal of Abnormal Psychology. 2002;111(2):211–224. [PubMed] [Google Scholar]
- Kerns JG, Berenbaum H, Barch DM, Banich MT, Stolar N. Word production in schizophrenia and its relationship to positive symptoms. Psychiatry Research. 1999;87(1):29–37. doi: 10.1016/s0165-1781(99)00052-9. [DOI] [PubMed] [Google Scholar]
- Kircher TT, Liddle PF, Brammer MJ, Williams SC, Murray RM, McGuire PK. Neural correlates of formal thought disorder in schizophrenia: Preliminary findings from a functional magenitic resonance imaging study. Archives of General Psychiatry. 2001;58(8):769–774. doi: 10.1001/archpsyc.58.8.769. [DOI] [PubMed] [Google Scholar]
- Kraepelin E. Dementia Praecox and Paraphrenia. Livingstone; Edinburgh: 1919. [Google Scholar]
- Kuperberg GR, McGuire PK, David AS. Reducted sensitivity to linguistic conext in schizophrenic thought disorder: Evidence from on-line monitoring for words in linguistically anomalous sentences. Journal of Abnormal Psychology. 1998;10(3):424–434. doi: 10.1037//0021-843x.107.3.423. [DOI] [PubMed] [Google Scholar]
- Landauer TK, Foltz PW, Laham D. Introduction to Latent Semantic Analysis. Discourse Processes. 1998;25:259–184. [Google Scholar]
- Lezak MD. Neuropsychological assessment. 3rd ed. Oxford University Press; New York: 1995. [Google Scholar]
- Liefker FR, Bowie CR, Harvey PD. Determinants of everyday outcomes in schizophrenia: The influences of cognitive impairment, functional capacity, and symptoms. Schizophrenia Research. 2009;115(1):82–87. doi: 10.1016/j.schres.2009.09.004. [DOI] [PubMed] [Google Scholar]
- McGurk SR, Moriatry PJ, Harvey PD, Parrella M, White L, Davis KL. The longitudinal relationship of clinical symptoms, cognitive functioning, and adaptive life in geriatric schizophrenia. Schizophrenia Research. 2000;42(1):47–55. doi: 10.1016/s0920-9964(99)00097-3. [DOI] [PubMed] [Google Scholar]
- McKenna PJ, Oh TM. Schizophrenic Speech: Making Sense of Bathroots and Ponds that Fall in Doorways. Cambridge University Press; Cambridge: 2005. [Google Scholar]
- Ojeda N, Peña J, Sanchez O, Elizargarate E, Ezcurra J. Processing speed mediated the relationship between verbal memory, verbal fluency, and functional outcome in chronic schizophrenia. Schizophrenia Research. 2008;101(1-3):225–233. doi: 10.1016/j.schres.2007.12.483. [DOI] [PubMed] [Google Scholar]
- Ojeda N, Sanchez P, Peña J, Elizagarate E, Yoller AB, Larumbe J, Gutierrez M, Casais L, Ezcurra J. Verbal fluency in schizophrenia: does cognitive performance reflect the same underlying mechanisms in patients and healthy controls? Journal of Nervous and Mental Diseases. 2010;198(4):286–291. doi: 10.1097/NMD.0b013e3181d61748. [DOI] [PubMed] [Google Scholar]
- O'Leary DS, Flaum M, Kesler ML, Flashman LA, Arndt S, Andreasen NC. Cognitive correlates of the negative, disorganized, and psychotic symptom dimensions of schizophrenia. The Journal of Neuropsychiatry and Clinical Neuroscience. 2000;12(1):4015. doi: 10.1176/jnp.12.1.4. [DOI] [PubMed] [Google Scholar]
- Perlick D, Mattis S, Stastny P, Teresi J. Neuropsychological discriminators of long-term inpatient or outpatient status in chronic schizophrenia. Journal of Neuropsychiatric Clinical Neuroscience. 1992;4(4):428–434. doi: 10.1176/jnp.4.4.428. [DOI] [PubMed] [Google Scholar]
- Petty RG, Barta PE, Pearlson GD, McGilchrist IK, Lewis RW, Tien AY, Pulver A, Vaughn DD, Casanova MF, Powers RE. Reversal of asymmetry of the planum temporale in schizophrenia. American Journal of Psychiatry. 1995;152(5):715–721. doi: 10.1176/ajp.152.5.715. [DOI] [PubMed] [Google Scholar]
- Rapp B, Caramazza A. On the distinction between deficits of access and deficits of storage: A question of theory. Cognitive Neuropsychology. 1993;10(2):113–141. [Google Scholar]
- Rempfer MV, Hamera EK, Brown CE, Cromwell RL. The relations between cognition and the independent living skill of shopping in people with schizophrenia. Psychiatry Research. 2003;117(2):103–112. doi: 10.1016/s0165-1781(02)00318-9. [DOI] [PubMed] [Google Scholar]
- Revheim N, Schechter I, Kim D, Silipo G, Alligham B, Butler P, Javitt DC. Neurocognitive and symptom correlates of daily problem-solving skills in schizophrenia. Schizophrenia Research. 2006;83(2-3):237–245. doi: 10.1016/j.schres.2005.12.849. [DOI] [PubMed] [Google Scholar]
- Ruff RM, Light RH, Parker SB, Levin HS. The psychological construct of word fluency. Brain and Language. 1997;57(3):394–405. doi: 10.1006/brln.1997.1755. [DOI] [PubMed] [Google Scholar]
- Spreen O, Strauss E. A Compendium of Neuropsychological Tests and Norms. 2nd ed. Oxford University Press; New York: 1998. [Google Scholar]
- Stirling J, Hellewell J, Blakey A, Deakin W. Thought disorder in schizophrenia is associated with both executive dysfunction and circumscribed impairments in semantic function. Psychological Medicine. 2006;36(4):475–484. doi: 10.1017/S0033291705006884. [DOI] [PubMed] [Google Scholar]
- Troyer AK, Moscovitch M, Winocur G. Clustering and switching as two components of verbal fluency: evidence from younger and older healthy adults. Neuropsychology. 1997;11(1):138–146. doi: 10.1037//0894-4105.11.1.138. [DOI] [PubMed] [Google Scholar]
- Velligan DI, Mahurin RK, Diamond PL, Hazelton BC, Eckert SL, Miller AL. The functional significance of symptomatology and cognitive functiong in schizophrenia. Schizophrenia Research. 1997;25(1):21–31. doi: 10.1016/S0920-9964(97)00010-8. [DOI] [PubMed] [Google Scholar]
- Wilkinson GS. The Wide Range Achievement Test. Wide Range, Inc; Wilmington, DE: 1993. [Google Scholar]
- White L, Harvey PD, Opler L, Lindenmayer JP. Empirical Assessment of the Factorial Structure of Clinical Symptoms in Schizophrenia. Psychopathology. 1997;30(5):263–274. doi: 10.1159/000285058. [DOI] [PubMed] [Google Scholar]