Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2025 Apr 1.
Published in final edited form as: Psychiatry Res. 2024 Feb 27;334:115831. doi: 10.1016/j.psychres.2024.115831

A Meta-Cognitive Wisconsin Card Sorting Test in People with Schizophrenia and Bipolar Disorder: Self-assessment of Sorting Performance

Mary E Gorora 1, Nina Dalkner 1,2, Raeanne C Moore 3, Colin A Depp 3,4, Varsha D Badal 3, Robert A Ackerman 5, Amy E Pinkham 5, Philip D Harvey 1,6,*
PMCID: PMC10947823  NIHMSID: NIHMS1971601  PMID: 38428288

Abstract

People with serious mental illness have challenged self-awareness, including momentary monitoring of performance. A core feature of this challenge is in the domain of using external information to guide behavior, an ability that is measured very well by certain problem-solving tasks such as the Wisconsin Card Sorting Test (WCST). We used a modified WCST to examine correct sorts and accuracy decisions regarding the correctness of sort. Participants with schizophrenia (n=99) or bipolar disorder (n=76) sorted 64 cards and then made judgments regarding correctness of each sort prior to feedback. Time series analyses examined the course of correct sorts and correct accuracy decisions by examining the momentary correlation and lagged correlation on the next sort. People with schizophrenia had fewer correct sorts, fewer categories, and fewer correct accuracy decisions (all p<.001). Positive response biases were seen in both groups. After an incorrect sort or accuracy decision, the groups were equally likely to be incorrect on the next sort or accuracy decision. Following correct accuracy decisions, participants with bipolar disorder were significantly (p=.003) more likely to produce a correct sort or accuracy decision. These data are consistent with previous studies implicating failures to consider external feedback for decision making. Interventions aimed at increasing consideration of external information during decision making have been developed and interventions targeting use of feedback during cognitive test performance are in development.

Keywords: Executive Functioning, Introspective Accuracy, Schizophrenia, Bipolar Disorder, Insight

1. Introduction

One of the most widely used tests in research on schizophrenia has been the Wisconsin card sorting test (WCST; Goldberg & Weinberger, 1988), largely because of the multiple cognitive processes that it examines (Gold et al., 1997). The WCST is an executive functioning/problem solving test wherein participants identify which of three sorting concepts is correct, sustain that concept through successive sorts, and then change concepts in response to feedback (Heaton, 1981). Feedback regarding a correct sort based on a concept serves as an indication that the current concept should be sustained. Error feedback signals the need for a change in concept. As a result, during task performance, the participant needs to recall the current concept being tested, evaluate every sort for correctness, and make trial by trial judgments to retain or shift sorting concepts. Thus, both internally-originating (concept identification, recall of the current concept, evaluation of the match of the sort to the concept) and externally originating (accuracy feedback) information must be integrated to succeed. Excessive reliance on internally generated information and challenges in discriminating information originating internally and externally has been implicated in previous theoretical models of the origin of delusions (Stirling et al., 1998), hallucinations (Keefe et al., 2002), and communication disorders (Harvey, 1985, Batchelder and Riefer, 1990), as well as impairments in self-assessment of cognitive and functional abilities (Durand et al., 2021). Failing to incorporate feedback would seem to render success on the WCST impossible.

Successful everyday functioning in general is facilitated by the ability to learn from experience and feedback. Unawareness of the presence of illness (Amador et al.,1991), commonly referred to as lack of clinical insight (Medalia and Thysen, 2010), is associated with challenges in the ability to evaluate experiences and incorporate external feedback regarding their plausibility, potentially leading to poor treatment decisions. Treatments targeting clinical insight, such as metacognitive therapy (Penney et al., 2022) or Cognitive Behavioral Therapy (Jones et al., 2000), commonly focus on increasing consideration of external information. An additional form of self-assessment is that of abilities and performance, commonly referred to as introspective accuracy (IA; Harvey & Pinkham, 2015). These impairments common in schizophrenia (Medalia et al., 2010) and bipolar disorder (Burdick et al., 2005) are associated with greater impairments in functional outcomes than would be predicted by performance in tests of cognition and functional capacity alone (Gould et al., 2015) and have been shown to be related to impairments in both social (Silberstein et al., 2018) and nonsocial (Harvey et al., 2019) real-world functional outcomes, as well as poorer performance on tests of cognition (Perez et al., 2020), social cognition (Jones et al., 2019), and functional capacity (i.e., the ability to perform everyday functional and social skills; Durand et al., 2015).

Previous studies of the accuracy of self-assessment of the quality of performance-based assessments show that perceived and actual performance does not align even with immediate self-assessment (Dalkner et al., 2023; Morgan et al., 2022), likely excluding forgetting of performance as a cause. Response biases toward over-underestimation compared to actual performance are common (Bowie et al., 2007; Mervis et al., 2022). People with schizophrenia have been shown in several studies to make judgments based on less information than healthy controls (Moritz & Woodward, 2005), to be more certain about their judgments (Moritz et al., 2015), to be biased toward discounting new information that is inconsistent with existing ideas (Coltheart & Davies, 2021), and to have an enhanced ability to recall self-generated information (Harvey et al., 1986). Pinkham et al. (2018) found that the largest single correlate of impaired social functioning in a large sample of participants with schizophrenia was how confident the participants reported that they were in their responses being correct during two different tests of social cognitive abilities. In a study of confidence in social cognitive performance, Jones et al. (2019) administered an emotion recognition test to participants with schizophrenia and asked them after each response how confident they were in the correctness of their responses. Those participants were overconfident compared to healthy participants across all levels of objective task performance. A subset of those participants (13%) reported that they were 100% certain that they were correct on every one of the scenarios, with this subset found to have the lowest actual task accuracy.

While previous studies have focused on the functional impacts of self-assessment challenges, quantification of sensitivity to external feedback in the process of self-assessment judgments is just starting to be understood. As described above, mastering the WCST requires sustaining and changing sorting concepts in response to external feedback. An individual who did not utilize external information (performance feedback), particularly if accompanied by high levels of confidence in their ability, would be disadvantaged in terms of solving this task. In some ways, the WCST is an ideal test of the ability to incorporate external feedback, while adding self-assessment and confidence components examines tendencies toward response bias during assessment of performance (i.e., always correct) and overconfidence (overly certain of the accuracy of self-assessments).

In an examination of basic sorting performance on a meta-cognitive modification of the 64-card WCST (Kongs et al., 2000) participants with schizophrenia or bipolar disorder were asked to perform each sort, generate an immediate dichotomous accuracy judgment regarding correctness of the sort (“Were you correct or incorrect?”) and produce a confidence rating (0–100) for their certainty regarding their judgment regarding the accuracy of the sort (Tercero et al., 2021). This meta-cognitive component was followed by feedback on the correctness of each sort as in the standard administration of the WCST. At the end of the task, participants were asked to generate a global judgment of their performance over the 64 trials. Participants with schizophrenia manifested a sort-by-sort positive response bias toward overconfidence (48% correct, 78% reported as correct). Sort-by-sort confidence ratings and end of study global performance ratings were uncorrelated with the number of correct sorts despite sort-by-sort accuracy feedback. Despite the zero correlation between global self-assessments and the number of correct sorts, global performance ratings were highly correlated with the number of sorts on which the participants believed that they were correct. These findings suggest very well-preserved abilities to remember self-generated information. They also suggest a response bias toward primarily weighting internally generated information (judgments) as compared to externally provided information (feedback) when evaluating task performance. This finding is clearly contrary to alternative explanations for poor WCST sorting performance such as random responding or low levels of effort and task engagement.

Participants with bipolar disorder also overestimated the correctness of their individual sorts (58% correct, 83% reported as correct), but their sort-by-sort confidence ratings and global performance judgments were correlated with the number of correct sorts (and not the number of sort-by-sort “correct” judgments), suggesting incorporation of the sort-by-sort performance feedback that they received (Tercero et al., 2021). In an analysis of these data focusing on the time course of momentary confidence in the correctness of sorts, Badal et al. (2023) reported that participants with bipolar disorder generated momentary sort-by-sort confidence ratings in the correctness of their sorts that tracked the recent stream of feedback on their performance. These participants manifested notable increases in sort-by-sort confidence following a series of correct sorts and a precipitous decline in confidence after encountering unexpected errors at category transitions, with a rapid recovery of confidence as they discovered the next sorting concept. Participants with schizophrenia had minimal variations in their confidence when sorting correctly across consecutive sorts and manifested a much more attenuated drop in confidence after unexpected errors.

In this final report from our WCST study, we examined the time course of correct sorts, as well as the time course of the correctness of self-generated accuracy judgments regarding the correctness of the immediately prior sort. We also examined the lagged effects of the correctness of the previous sort on the likelihood that the next sort would be correct. We also applied this strategy to sort-by-sort accuracy judgments (i.e., making an accurate judgment of the sort being correct or incorrect) as a predictor of the correctness of the next sort and the accuracy of the next judgment. While previous papers examined confidence, here we focus on whether the correctness of a sort, or an accurate judgment about the correctness of a sort, informed subsequent correct sorting and accuracy judgments. Solving the WCST across typical success metrics should be facilitated by developing the ability to accurately judge the correctness of individual sorts, based on recent feedback and recollection of the current sorting concept, and using those decisions to inform upcoming sorts.

Multiple hypotheses were tested in this study. We anticipated that participants with schizophrenia would show a smaller improvement than the bipolar disorder group in correct sorts as they gained experience, based on prior research highlighting the difficulties individuals with schizophrenia face in effectively utilizing feedback. Similarly, we hypothesized that, with increasing experience, there would be a less pronounced improvement in accurately determining the correctness of sorts. We further hypothesized that the accuracy judgements of previous sorts would have a diminished impact on subsequent correct sorts and accuracy judgements. We also believed that overall cognitive ability, indexed by a composite of neuropsychological test performance, would be correlated with, but not fully explanatory of, correct sorting and accuracy judgments regrading sorts in both samples. We examined these data for the first half of the task, as the participants with schizophrenia took an average of more than 32 sorts to complete their first category. Thus, we were able to focus on the time course of the process of identification of the first correct category, on average, for participants with schizophrenia.

2. Methods

The basic results of this study, including the association between correct sorts, momentary accuracy judgments, confidence in accuracy judgments, and global self-assessment of performance have been published before (Tercero et al., 2021; Badal et al., 2023). In the current report, we note all previously published findings, which overlap minimally with the goals of this sub-study. The overall study structure was such that participants received a baseline diagnostic assessment, answered ecological momentary assessment questions up to 3 times a day for 30 days, and completed an endpoint assessment (Durand et al., 2021). The current data were all collected at that endpoint assessment.

2.1. Participants

Participants who met DSM-V criteria for schizophrenia, schizoaffective disorder, or bipolar disorder (type I or II), with or without current or previous psychotic symptoms, participated in this study. All schizophrenia spectrum patients were grouped into a single group, as were the bipolar disorder participants. They were recruited at three different sites: The University of Miami Miller School of Medicine (UM), the University of California San Diego (UCSD), and The University of Texas at Dallas (UTD). UM participants were recruited from the Jackson Memorial Hospital-University of Miami Medical Center and the Miami VA Medical Center. UCSD participants were recruited from the UCSD Outpatient Psychiatric Services clinic, a large public mental health clinic, the San Diego VA Medical Center, and other local community clinics and by word of mouth. UTD participants were recruited primarily from Metrocare Services, a non-profit mental health services organization in Dallas County, TX, and from other local clinics. The study was approved by each University’s respective Institutional Review Board, and all participants provided written informed consent. Diagnostic information was collected by trained interviewers using the Mini International Neuropsychiatric Interview (MINI; Sheehan et al., 1998) and the psychosis module of the Structured Clinical Interview for DSM Disorders-5 (SCID-5; First et al., 2015), and a local consensus procedure was used to generate final diagnoses.

2.2. Inclusion/Exclusion Criteria

All participants were required to be clinically stable (i.e., no hospitalizations) for a minimum of 6 weeks and to be on a stable medication regimen for a minimum of 6 weeks with no dose changes >20% for a minimum of 2 weeks. Any antipsychotics or antipsychotic combinations were accepted.

For participants in both diagnostic groups exclusion criteria included: (1) current or historical medical or neurological disorders that may affect brain functioning (e.g., CNS tumors, seizures, or loss of consciousness for over 15 minutes), (2) history of intellectual disability (IQ<70) or pervasive developmental disorder according to the DSM-5 criteria, (3) presence of substance use disorder not in remission for at least six months, (4) visual or hearing impairments that interfere with assessment, and (5) lack of proficiency in English. Participants with a Wide Range Achievement Test-3rd edition (WRAT-3; Jastak, 1993) grade equivalent score of less than 8th grade were also not enrolled.

2.3. Performance-based assessment

2.3.1. Neurocognitive Performance

We administered a battery of neurocognitive tests on the same day as the WCST assessment. The cognitive assessments were a subset of the tasks from the MATRICS Consensus Cognitive Battery (MCCB; Nuechterlein et al., 2008), selected for high correlations with composite scores. Each raw score was converted to a t score using the MCCB scoring program. All t-scores were averaged into a single composite.

  • The Trail Making Test Part A (TMT A) is a measure of individual attention and psychomotor speed. Participants are instructed to connect numbers in ascending order as quickly and carefully as possible.

  • The Brief Assessment of Cognition in Schizophrenia (BACS) symbol coding is a sensitive measure of psychomotor processing speed. Using a reference key, participants are asked to pair specific numbers with geometric figures within 90 seconds.

  • The Maryland Letter-Number Sequencing test (LNST) is a valid and sensitive measure of auditory working memory. The examinees hear a series of letters and digits, and then report back the stimuli with the letters in alphabetical order, and digits in ascending numerical order.

  • The Animal Naming Test (ANT) measures semantic verbal fluency and processing speed. Subjects were asked to list as many animals as they could in one minute.

  • The Hopkins Verbal Learning Test (HVLT) is an assessment that consists of three learning trials of a 12-item semantically organized list with the total score as the dependent variable, in line with the typical use of this test in the MCCB. Each form of the HVLT contains four words each from one of three semantic categories and as we only performed one assessment, we used form 1 (animals, precious stones, and places to live).

2.3.2. Metacognitive Wisconsin Card Sorting Test.

This test was like the one used by Gould et al. (2015). There were 64 sorts to be performed. After each sort, to measure IA, the participant was asked “Did you get it correct?” [Accuracy Judgment] and answered with a yes/no response. Participants were then asked to provide a confidence rating on a 1–5 scale as to their certainty in the correctness of their accuracy judgment. Participants were then provided feedback about the correctness of their sort after they had provided both accuracy and confidence judgments. To date, there are no psychometric data available on the modified WCST.

In this report the dependent variables were the standard measures of correct sorts [Previously reported], categories completed, and number of trials to complete the first category. The IA-relevant self-assessment was the sort-by-sort accuracy of judgments regarding the correctness of each sort. There are 4 accuracy variables, described by a 2 × 2 matrix like any signal detection task. These variables are: Correct decisions about a correct sort [Hits], correct decisions about an incorrect sort [Correct Rejections], Positive Accuracy judgments about an incorrect sort [False Positive], and Negative Accuracy judgments about correct sorts [False Negatives]. We examined the data as a time series and analyzed the time course of correct sorts and correct accuracy decisions over the first 32 trials.

2.4. Data Analyses.

We compared the two groups on the previously unreported WCST variables using t-tests. We calculated Pearson Correlations between composite cognitive performance and the WCST variables in each group separately. For these analyses, we used a Bonferroni Corrected significance level of p<.01 because of the number of correlations calculated. These analyses were based on the entire set of 64 sorts, in order to index overall differences in performance. We calculated effect sizes with Cohen’s d, because there was homogeneity of variance across the outcomes variables in the two samples.

Mixed model repeated measures (MMRM) analyses were used to examine the time course of correct sorts and accuracy judgments regarding the correctness of the sorts, limiting the analyses to the first 32 sorts. We used the Generalized Linear Models (GLM) program from SPSS version 28 (IBM corporation, 2022). A mixed effects hierarchical linear modeling (HLM) strategy was used to examine the time course of the two different WCST variables. For these analyses, we entered Sort (1–32) as a repeated-measure, diagnosis as fixed factor, the interaction of diagnosis x sort, and subject as a random intercept. We used the omnibus test to determine that the fitted model improved on the intercept-only model. After computing the first models, we added composite cognitive performance as a fixed covariate.

For a final analysis, we created lagged variables, wherein we first related the correctness of the previous sort with the likelihood of making a correct sort on the next attempt. These variables were lagged by a single sort, so they identified the immediately prior proximal observations regarding correctness of sorting and accuracy of decision-making Then we examined the correctness of the accuracy decision for the previous sort as a predictor of both correct accuracy decisions and correct sorts on the next sort. Finally, we entered both correctness of the previous sort and accuracy of the previous decision to predict the correctness of the next sort to see how these influences overlapped.

3. Results

3.1. Demographic data

Demographic information on the participants is presented in Table 1. These data were presented in the previous publication (Tercero et al., 2021), but it is worth noting that the participants with schizophrenia had less education and lower reading scores, as well as being less likely to have ever been married and more likely to be male and Black than the participants with bipolar disorder. We have previously published the cognitive functioning correlates of those demographic differences in this sample.

Table 1.

Descriptive and Demographic Information on Participants

Variables Schizophrenia n=99 Bipolar Disorder n=67 t p
M SD M SD
Age 41.98 10.44 39.22 11.75 1.63 .11
Years of Education 12.53 2.32 14.22 2.64 4.42 <.001
Mothers Education 13.05 3.54 13.67 3.67 1.81 .069
WRAT-3-Standard Score 95.42 11.85 102.13 11.70 3.67 <.001
% % χ2 p
Sex (% Female) 48 69 8.22 .004
Racial Status (%)
 Caucasian 32 53 15.27 .009
 African American 54 25
 Asian 2 3
 Native American, Hawaiian, Alaskan 1 1
 Other, Multiple, Unknown 11 12
Ethnic Status (%)
 Hispanic 24 29 0.64 .420
 Non-Hispanic 76 81
Ever Married or Equivalent 49 70 7.14 .007
Financially Responsible 71 70 0.02 .880
Unemployed for more than one year 60 45 2.74 .100
Open in a new tab

Note. M = Mean, SD = Standard Deviation, WRAT = Wide Range Achievement Test

3.2. WCST Performance and Judgment Variables.

These data and the significance tests for group differences are presented in Table 2. As can be seen in the table, participants with bipolar disorder performed better on all of the WCST performance variables, including correct sorts (previously reported), categories completed, and trials to the first category. Composite cognitive performance was also better for the participants with bipolar disorder and the effect size for performance differences was d=.54 or more. For the variables regarding accuracy judgments of correctness of sorts, or IA, participants with bipolar disorder had more correct accuracy judgments (hits) and fewer false positive judgments (all d>.34), with no nominally significant differences in correct rejections and false negative judgments. Total correct accuracy decisions were also significantly higher in the participants with bipolar disorder.

Table 2.

Wisconsin Card Sorting Test Performance and Judgement Accuracy

Variables Schizophrenia n=99 Bipolar Disorder n=67 t p d
Performance Data M SD M SD
Correct Sorts* (out of 64) 30.67 11.61 36.95 11.77 3.39 <.001 .72
Categories Completed 1.11 1.25 1.94 1.56 −3.79 <.001 .60
Trials to First Category 35.02 26.52 21.18 24.65 −3.84 <.001 .54
Composite Cognitive performancea 40.39 7.43 45.20 8.33 −3.84 <.001 .62
Judgement Data Presented as % of Decisions M SD M SD t p d
Hits 0.37 0.22 0.51 0.24 −3.25 <.001 .57
Correct Rejections 0.14 0.16 0.12 0.14 1.67 .090 .26
False Positive 0.38 0.19 0.30 0.15 2.13 .017 .34
False Negative 0.11 0.13 0.07 0.11 1.00 .160 .08
Correct Accuracy Decisions* 0.51 0.15 0.61 0.15 −3.28 <.001 .66
Open in a new tab

Note. M = Mean, SD = Standard Deviation,

Cohen’s d: Effect Size

a

t-scores based on MCCB norms

*

Previously reported

3.3. Correlations Between Variables

Intercorrelations of the variables in the two participant samples are presented separately in Table 3. Of interest, likely due to a strong positive decision bias and very few occurrences of this error, false negative responses failed to correlate with any other variables in either sample. Cognitive performance was correlated similarly with the WCST performance and decision performance across the samples. In terms of the accuracy variables, total false positive accuracy judgments were correlated with significantly fewer correct sorts, fewer total categories completed, and more trials to the first category in both samples. Conversely, more hits were associated with more correct sorts and categories completed and fewer trials to the first category in both samples. One area of difference between the samples was the relationship of correct rejections with total correct decisions. Although the correlations were in the same direction, the correlation between correct rejections and total correct decisions was significantly larger in the bipolar participants, z=2.19, p=.035, suggesting a larger contribution of immediately recognizing inaccurate sorts in the bipolar participants.

Table 3.

Intercorrelations of Wisconsin Card Sorting Test Performance and Decision Accuracy Variables and Other Cognitive Indicators in Participants with Schizophrenia and Bipolar Disorder

Schizophrenia Participants (n=99)
Variable 1 2 3 4 5 6 7 8 9
1. Correct Sorts — .83** −.79** .79** .16 −.62** −.03 .73** .35**
2. Categories Completed .88** — −.83** .72** .33** −.53** .02 .71** .32**
3. Trials to first Category −70** −.76** — −.68** −.35** .46** −.07 −.63** −.34**
4. Hits .90** .81** −.63** — −.73** −.15 −.02 .69** .44**
5. Correct Rejections .44** .51** −.48** −.83** — −.47** .05 .18 .26**
6. False Positive −.66** −.61** .42** −.33** −.39** — .02 −.71** −.11
7. False Negative .08 .06 −.03 .06 −.10 −.01 — .02 −.13
8 Correct Accuracy Decisions .90** .84** −.59** .85** .48** −.71** .01 — 36**
9. Composite Cognition .39** .36** −.42** .40** .34** .34** −.18 .18 —
Open in a new tab

Note.

*

p < .01.

**

p < .001.

Bipolar Participants (n=67)

Correlations for Schizophrenia participants are above the diagonal and those for Bipolar Disorder are below.

3.4. Longitudinal analyses

3.4.1. Correct Sorting and Decision Accuracy

The trajectories for the time course of correct sorts and correct accuracy decisions are presented in Figures 1 and 2 respectively and the results of the statistical tests are presented in Table 4. For both correct sorts and correct decisions, there were significant repeated measures effects of sort, with improvements over time. There were significant between-subjects effects of diagnosis and diagnosis x sort interactions, with participants with schizophrenia having reduced gains with sorting experience. When composite cognition was added as a fixed covariate, the effect of cognition was significant, and the interaction of diagnosis x sort became nonsignificant for correct sorts but not correct accuracy judgments. Thus, reduced gains in correct sorts over time in the participants with schizophrenia compared to participants with bipolar disorders seems partially associated with poorer cognitive performance. However, accuracy of decisions regarding sorting accuracy differed between the groups above and beyond the covariate effect of cognitive performance. Because of these findings, we did not run the analyses separately in the two diagnostic groups.

Figure 1.

Figure 1.

Open in a new tab

Time course of correct sorts by diagnosis over the first 32 sorts. See table 4 for statistical test results.

Figure 2.

Figure 2

Open in a new tab

Time course of correct decisions regarding accuracy of a sort prior to feedback by Diagnosis over the first 32 sorts. See table 4 for statistical test results.

Table 4.

Results of Repeated Measures HLM Analyses on The Time Course of Correct Sorts and Correct Accuracy ecisions

Time Course analyses by Sort: Correct Sorts and Correct Decisions
Correct Sorts Correct Accuracy Decisions
χ2 df p χ2 df p
Intercept 6579.28 1 <.001 7315.56 1 <.001
Diagnosis 110.18 1 <.001 74.74 1 <.001
Sort 806.80 31 <.001 469.37 31 <.001
Sort x Diagnosis 46.94 31 .03 56.81 31 .030
Time Course analyses by Sort: Correct Sorts and Correct Decisions with Cognition as a Covariate
Correct Sorts Correct Accuracy Decisions
χ2 df p χ2 df p
Intercept 111.76 1 <.001 44.14 1 <.001
Diagnosis 36.95 1 <.001 25.17 1 <.001
Sort 804.62 31 <.001 481.96 31 <.001
Sort x Diagnosis 44.22 31 .058 48.07 31 .003
Composite Cognition 131.66 1 <.001 80.14 1 <.001
Open in a new tab

3.4.2. Lagged Analyses for Sorting and Decision Accuracy

The lagged analyses are presented in Table 5. In the first analysis, correctness on the immediately previous sort was used to predict correctness of the next sort; there were significant effects of sort, diagnosis, and a significant sort x diagnosis interaction. The largest effect was making a correct previous sort, with composite scores on cognition also a significant predictor.

Table 5.

Lagged Analyses of Correct Accuracy Decisions and Correct Sorts

Lagged Analyses with Previous Sort Correct as a Predictor
Correct Sorts
χ2 df p
Intercept 59.30 1 <.001
Diagnosis 15.38 1 <.001
Sort 886.10 30 <.001
Sort x Diagnosis 59.19 30 .040
Sort Correct before 798.47 1 <.001
Composite Cognition 62.24 1 <.001
Lagged Analyses with Previous Decision Correct as a Predictor
Correct Sorts Correct Decisions
χ2 df p χ2 df p
Intercept 109.76 1 <.001 42.93 1 <.001
Diagnosis 33.75 1 <.001 25.29 1 <.001
Sort 806.76 30 <.001 485.91 30 <.001
Sort x Diagnosis 49.80 30 .040 47.81 30 .030
Decision Correct before 45.22 1 <.001 38.94 1 <.001
Composite Cognition 110.89 1 <.001 80.14 1 <.001
LS Means for the Examination of Correct Sorts and Decisions About Accuracy as Predictors of Subsequent Sorts and Decisions Presented as Proportion Scores
Sorting Performance as a Function of the Correctness of Immediately Prior Sort
Correct Sorts
LS Mean SEM
Schizophrenia
Prior Sort Incorrect .29 .01
Prior Sort Correct .65 .01
Bipolar Disorder
Prior Sort Incorrect .32 .02
Prior Sort Correct .70 .01
Sorting Performance and Decision Accuracy as a Function of Correctness of Immediately Prior Accuracy Decision
Correct Sorts Correct Decisions
LS Mean SEM LS Mean SEM
Schizophrenia
Prior Decision Incorrect .44 .02 .52 .02
Prior Decision Correct .48 .01 .51 .01
Bipolar Disorder
Prior Decision Incorrect .42 .03 .55 .03
Prior Decision Correct .61 .01 .63 .01
Open in a new tab

In the next analyses, correctness of the accuracy decision on the immediately previous sort was used to predict correct sorts and accuracy decisions on the next sort. For both correct sorts and correct decisions on the next sort, there were identical patterns of significant effects of sort, diagnosis, cognition and a significant sort x diagnosis interaction. Correct accuracy decisions on the previous sort were found to be significant predictors of both correctness of the next sort and correct decisions regarding the accuracy of that sort.

The bottom of Table 5 presents the LS means for the correct sorts and accuracy decisions regarding correctness of sorts. For correctness of the previous sort predicting accuracy of the current sort, the response patterns for the two groups were very similar, with the percentage of correct sorts following an incorrect sort at 29% and 36% for schizophrenia and bipolar participants respectively, with rates of correct sorts increasing to 65% and 70% for sorts following a correct sort.

For the impact of prior accuracy decisions on subsequent correct sorts, correctness of the next sort was essentially identical in the two groups following an incorrect accuracy decision. There was a considerably larger difference in the likelihood of a correct sort following a correct accuracy decision in participants with bipolar disorder compared to those with schizophrenia: 19% vs 4%. These differences are consistent with the 2-way interaction of sort x diagnosis. For prior decision accuracy predicting later decision accuracy, there was no difference between the samples in the likelihood that the next decision would be correct when the prior decision was incorrect (52% vs. 55%), with both values close to the 50% correct random guessing threshold. In contrast, for accuracy decisions following a correct decision, participants with bipolar disorder had an increased likelihood that their accuracy decision would be correct, increasing to 63%. For the participants with schizophrenia there was no difference in decision accuracy following correct decisions compared to an incorrect decision.

4. Discussion

In this final analysis of the data from our WCST study, we performed time series analyses focused on correct sorts and correct accuracy judgments regarding sorts. In line with our hypotheses, participants with bipolar disorder generally had more rapid early gains in correct sorts as well as a greater number of correct accuracy judgments with experience. These findings were accompanied by better scores on several other indicators of WCST performance over the entire course of the test, including categories completed and number of sorts required to reach the first category. Previous studies have shown that participants with schizophrenia who do not master the WCST can be detected very early in the sorting process, as early as 4 trials (Prentice et al., 2008). It is of substantial interest that participants with schizophrenia in that sample were 60% likely to generate a correct sort by the 4th sort; this is identical to the value in our sample (see figure 1). Prentice et al. discussed whether error processing and response to negative feedback was specifically impaired in this population. As shown in the growth curves and in the statistical analyses, there are early and generally sustained differences between our two participant groups in both correct sorting and making correct accuracy judgments about sorts.

Our second hypothesis was confirmed in part, in that correct accuracy judgments about the previous sort exerted a greater influence on the correctness of the next sort and the accuracy of the next judgment in participants with bipolar disorder. There was essentially no carry-over benefit from being correct on the prior sort toward correctness of a subsequent sort in the participants with schizophrenia. The most salient difference in performance across the groups was the absence of correlation between prior accuracy decisions and subsequent correct. Correct rejections had different correlational relationships with both correct sorts and overall accuracy decisions in the two groups. These findings likely suggest that participants with bipolar disorder were more able to switch sorting strategies after correctly identifying an incorrect sort and receiving feedback that they were in fact incorrect. Correct overall accuracy decisions (Yes/No) were consistent with chance in the participants with schizophrenia.

The finding that correct sorts predict accurate subsequent sorts in participants with schizophrenia (albeit minimally), despite no correlation between prior and subsequent accuracy judgments, suggests that using hypothesis testing to the correct sorting principal is not how these participants are performing the WCST. The fact that the modal participant with schizophrenia did not achieve a single category in the first half of the test means that participants were not, as a group, sustaining correct sorting for 10 consecutive sorts. One possibility for such “set failures” after a two or more correct responses is that the participants are not aware of the concept underlying their sort, likely because of not basing sorting decisions on feedback regarding correctness of prior sorts. The essentially random association between decision accuracy across proximal sorting attempts again suggests a failure to incorporate external feedback.

Previous studies with different performance and self-assessment indices targeting self-assessment have found that decision making accuracy was correlated with poorer cognitive performance, while response biases were not (Mervis et al., 2022). For instance, in a recent study (Dalkner et al., 2023), when participants with bipolar disorder or schizophrenia were asked to make an immediate judgment regarding their neuropsychological test performance,. poorer cognitive performance was associated with greater absolute mis-estimation of performance in both groups. However, impairments in performance did not correlate with a response bias to overestimate. In the first use of a metacognitive WCST, Koren et al. (2004) found that decision-making biases were correlated with ratings of insight but not with correct sorting performance or categories completed.

Previous research on the WCST has suggested poorer clinical insight (Lysaker and Bell, 1994) was correlated with more perseverative errors. Increased perseverative errors were also found to be present in participants with paranoid features (Abbruzzese, et al., 1996). As perseverative errors are the definitional case of decision making without considering external information, these findings are consistent with the model of Moritz et al. (2015) suggesting that failures to consider external information arise from poor problem solving and sustain paranoid ideation. Cognitive neuroscience research has suggested a substrate that may describe these challenges, referred to as the “Hyperfocusing Hypothesis” (Luck et al., 2019). This model suggests that people with schizophrenia may over-allocate processing resources to an overly limited set of domains, often idiosyncratically selected, leading to better than normal performance in the targeted domains and poorer overall performance. In this study, the exeptional performance of people with schizophrenia in recalling and focusing on their accuracy decisions, while failing to process feedback is consistent with that conceptualization. Also associated with this model is the tendency to engage in “all or nothing “ belief changes (Nassar et al., 2021), which can be unassociated with immediately proximal feedback. Thus, a series of correct sorts on the WCST could still lead to abandonment of that sorting principle, based on internally originating information.

4.1. Limitations

Without a healthy comparison sample, it is not possible to evaluate the sorting and judgment performance of bipolar participants, who performed better than people with schizophrenia. When the large-scale data-base from the Prentice et al. study is examined, their healthy controls were over 90% likely to be correct by the 4th sort, where our bipolar participants were correct on 78% of their 4th sorts. Applying the momentary accuracy decision analyses to healthy controls could quantify the contribution of accuracy decisions to correct sorting. Our participants were prescribed medication, but it is not clear if they were adherent. Participants were not selected for the presence of any particular level of either positive or negative symptoms, mania, or depression. Previous analyses in the overall sample have suggested a connection between more frequent momentary psychotic symptoms and self-report decision bias for cognitive performance in participants with schizophrenia but not bipolar disorder (Gohari et al., 2022) and more intense momentary reports of negative affect in participants with bipolar disorder but not schizophrenia (Dalkner et al., 2023).

4.2. Clinical Implications and Conclusions

The biggest clinical implication of these findings is that they provide more evidence that participants with schizophrenia are not adequately considering even explicitly presented external feedback when making decisions. Although participants with bipolar disorder had better sorting performance and better overall accuracy judgments (and better composite cognition), they still manifested a substantial positive momentary response bias, leading to increased tendencies to report that they were correct when they were not. Although their overly positive bias seems to be overcome by feedback, global judgments in more ambiguous situations without specific feedback could still manifest an excessively positive introspective bias and result in overoptimistic decisions.

Cognitive enhancing treatments, either cognitive remediation therapy or pharmacological, would seem to have the potential to improve WCST performance and global accuracy of self-assessments. However, the minimal correlations between composite cognitive performance and positive introspective bias in this study were similar to the overall literature reviewed by (Mervis et al., 2022). Thus, it seems as if strategic interventions, such as metacognitive therapy (Penney et al., 2022) rather than global cognitive enhancement, would be required to make inroads into the self-assessment biases seen in these participant populations. . A new digital intervention study (iTEST: Introspective Accuracy as a Novel Target for Functioning in Psychotic Disorders; MH129379) is currently in process with US Government funding and results of using this strategy can be compared to the effects of CBT and metacognitive training on other aspects of the illness.

Highlights.

  • People with both bipolar disorder and schizophrenia are challenged in their performance of the Wisconsin Card Sorting test.

  • Both groups had a false positive response bias.

  • People with schizophrenia appear to generate accuracy judgments that do not incorporate prior feedback.

  • People with bipolar disorder do appear to benefit from feedback in performing the test.

Acknowledgments

This research was supported by NIMH grant RO1MH112620 to Dr. Pinkham. The data in this study are being deposited in the NIMH RDOC repository. 6 Months after data lock they will be available for public access. In the interim, the authors are happy to share the data that underlie this paper.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Conflict of interest statement

Dr. Raeanne C. Moore is a co-founder of KeyWise AI, Inc. and a consultant for NeuroUX. Dr. Harvey has received consulting fees or travel reimbursements from Alkermes, Boehringer Ingelheim, Karuna Therapeutics, Minerva Neuroscience, Roche Pharma, and Sunovion Pharma. He receives royalties from the Brief Assessment of Cognition in Schizophrenia, which is owned by WCG-Endpoint Solutions and is part of the MATRICS Consensus Battery. He is chief scientific officer of iFunction, Inc. Dr. Pinkham has served as a consultant for Roche Pharma. The other authors have no potential Biomedical Conflicts of Interest.

References

  1. Abbruzzese M, Ferr S, Scarone S 1996. Performance on the Wisconsin Card Sorting Test in schizophrenia: perseveration in clinical subtypes. Psychiatry Res 64(1):27–33. [DOI] [PubMed] [Google Scholar]
  2. Amador XF, Strauss DH, Yale SA, Gorman JM, 1991. Awareness of illness in schizophrenia. Schizophr. Bull 17 (1), 113–132. [DOI] [PubMed] [Google Scholar]
  3. Badal VD, Depp CA, Pinkham AE, Harvey PD, 2023. Dynamics of task-based confidence in schizophrenia using seasonal decomposition approach. Schizophr. Res. Cogn 32, 100278. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Batchelder WH, Riefer DM, 1990. Multinomial processing models of source monitoring. Psychol. Rev 97 (4), 548. [Google Scholar]
  5. Bowie CR, Twamley EW, Anderson H, Halpern B, Patterson TL, Harvey PD, 2007. Self-assessment of functional status in schizophrenia. J. Psychiatr. Res 41(12), 1012–1018. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Burdick KE, Endick CJ, Goldberg JF, 2005. Assessing cognitive deficits in bipolar disorder: are self-reports valid?. Psychiatry Res 136 (1), 43–50. [DOI] [PubMed] [Google Scholar]
  7. Coltheart M, Davies M, 2021. Failure of hypothesis evaluation as a factor in delusional belief. Cogn. Neuropsychiatry 26 (4), 213–230. [DOI] [PubMed] [Google Scholar]
  8. Dalkner N, Moore RC, Depp CA, Ackerman RA, Pinkham AE, Harvey PD, 2023. Negative mood states as a correlate of cognitive performance and self-assessment of cognitive performance in bipolar disorder versus schizophrenia. Schizophr. Res 252, 1–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Dalkner N, Moore RC, Depp CA, Ackerman RA, Pinkham AE, Harvey PD,. 2023. Immediate post performance judgements about cognitive performance in schizophrenia and bipolar disorder: associations with test performance and subjective overall judgments regarding abilities Cogn. Neuropsychiatry 28(6):450–466 [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Durand D, Strassnig MT, Moore RC, Depp CA, Ackerman RA, Pinkham AE, et al. , 2021. Self-reported social functioning and social cognition in schizophrenia and bipolar disorder: Using ecological momentary assessment to identify the origin of bias. Schizophr. Res 230, 17–23. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Durand D, Strassnig M, Sabbag S, Gould F, Twamley EW, Patterson TL, et al. , 2015. Factors influencing self-assessment of cognition and functioning in schizophrenia: implications for treatment studies. Eur. Neuropsychopharmacol 25 (2), 185–191. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. First MB, Williams JBW, Karg RS, Spitzer RL, 2015. Structured Clinical Interview for DSM-5—Research Version (SCID-5 for DSM-5, Research Version; SCID-5-RV) Arlington, VA, American Psychiatric Association. [Google Scholar]
  13. Gold JM, Carpenter C, Randolph C, Goldberg TE, Weinberger DR, 1997. Auditory working memory and Wisconsin Card Sorting Test performance in schizophrenia. Arch. Gen. Psychiatry 54 (2), 159–165. [DOI] [PubMed] [Google Scholar]
  14. Goldberg TE, Weinberger DR, 1988. Probing prefrontal function in schizophrenia with neuropsychological paradigms. Schizophr. Bull 14 (2), 179–183. [DOI] [PubMed] [Google Scholar]
  15. Gohari E, Moore RC, Depp CA, Ackerman RA, Pinkham AE, Harvey PD, 2022. Momentary severity of psychotic symptoms predicts overestimation of competence in domains of everyday activities and work in schizophrenia: An ecological momentary assessment study. Psychiatry Res 310, 114487. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Gould F, McGuire LS, Durand D, Sabbag S, Larrauri C, Patterson TL, et al. , 2015. Self-assessment in schizophrenia: Accuracy of evaluation of cognition and everyday functioning. Neuropsychology 29 (5), 675–682. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Harvey PD, 1985. Reality monitoring in mania and schizophrenia: The association of thought disorder and performance. J. Nerv. Ment. Dis 173 (2), 67–73. [DOI] [PubMed] [Google Scholar]
  18. Harvey PD, Earle-Boyer EA, Weilgus MS, Levinson JC, 1986. Encoding, memory, and thought disorder in schizophrenia and mania. Schizophr. Bull 12 (2), 252–261. [DOI] [PubMed] [Google Scholar]
  19. Harvey PD, Deckler E, Jones MT, Jarskog LF, Penn DL, Pinkham AE, 2019. Autism symptoms, depression, and active social avoidance in schizophrenia: Association with self-reports and informant assessments of everyday functioning. J. Psychiatr. Res 115, 36–42. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Harvey PD, Pinkham A, 2015. Impaired self-assessment in schizophrenia: Why patients misjudge their cognition and functioning. Curr. Psychiatry 14 (4), 53–59. [Google Scholar]
  21. Heaton RK, 1981. Wisconsin card sorting test manual; revised and expanded Psychological Assessment Resources, Lutz, FL [Google Scholar]
  22. IBM Corporation, 2022. Statistical package for the social sciences (SPSS) Edition 28 [Computer software] Armonk, NY: 2022 [Google Scholar]
  23. Jastak S, 1993. Wide-Range Achievement Test, 3rd ed. San Antonio, TX: Wide Range, Inc; 1993. [Google Scholar]
  24. Jones C, Cormac I, Mota J, Campbell C, 2000. Cognitive behaviour therapy for schizophrenia. Cochrane Database Syst. Rev (2), CD000524. [DOI] [PubMed] [Google Scholar]
  25. Jones MT, Deckler E, Laurrari C, Jarskog LF, Penn DL, Pinkham AE, Harvey PD 2019. Confidence, performance, and accuracy of self-assessment of social cognition: A comparison of schizophrenia patients and healthy controls. Schizophr Res Cogn 19:002–2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Keefe RS, Arnold MC, Bayen UJ, McEvoy JP, Wilson WH, 2002. Source-monitoring deficits for self-generated stimuli in schizophrenia: multinomial modeling of data from three sources. Schizophr. Res 57 (1), 51–67. [DOI] [PubMed] [Google Scholar]
  27. Kongs SK, Thompson LL, Iverson GL, Heaton RK, 2000. Wisconsin card sorting test-, 64 card version: WCST-64 Lutz, FL: PAR. [Google Scholar]
  28. Koren D, Seidman LJ, Poyurovsky M, Goldsmith M, Viksman P, Zichel S, Klein E 2004. The neuropsychological basis of insight in first-episode schizophrenia: a pilot metacognitive study. Schizophr Res 70(2–3):195–202. [DOI] [PubMed] [Google Scholar]
  29. Luck SJ, Hahn B, Leonard CJ, Gold JM 2019. The Hyperfocusing Hypothesis: A New Account of Cognitive Dysfunction in Schizophrenia. Schizophr. Bull 45(5):991–1000. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Lysaker P, Bell M 1994. Insight and cognitive impairment in schizophrenia. Performance on repeated administrations of the Wisconsin Card Sorting Test. J. Nerv. Ment. Dis 182(11):656–60. [DOI] [PubMed] [Google Scholar]
  31. Medalia A, Thysen J, 2010. A comparison of insight into clinical symptoms versus insight into neurocognitive symptoms in schizophrenia. Schizophr. Res 118 (1–3), 134–139. [DOI] [PubMed] [Google Scholar]
  32. Mervis JE, Vohs JL, Lysaker PH 2022. An update on clinical insight, cognitive insight, and introspective accuracy in schizophrenia-spectrum disorders: symptoms, cognition, and treatment. Expert Rev Neurother 22(3):245–255. [DOI] [PubMed] [Google Scholar]
  33. Morgan O, Strassnig MT, Moore RC, Depp CA, Ackerman RA, Pinkham AE, et al. , 2022. Accuracy of immediate self-assessment of neurocognitive test performance: Associations with psychiatric diagnosis and longitudinal psychotic symptoms. J. Psychiatr. Res 156, 594–601. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Moritz S, Göritz AS, Gallinat J, Schafschetzy M, Van Quaquebeke N, Peters MJ, et al. , 2015. Subjective competence breeds overconfidence in errors in psychosis. A hubris account of paranoia. J. Behav. Ther. Exp. Psychiatry 48, 118–124. [DOI] [PubMed] [Google Scholar]
  35. Moritz S, Woodward TS, 2005. Jumping to conclusions in delusional and non-delusional schizophrenic patients. Br. J. Clin. Psychol 44 (Pt 2), 193–207. [DOI] [PubMed] [Google Scholar]
  36. Nassar MR, Waltz JA, Albrecht MA, Gold JM, Frank MJ 2021. All or nothing belief updating in patients with schizophrenia reduces precision and flexibility of beliefs. Brain 2021 Apr 12;144(3):1013–1029. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Nuechterlein KH, Green MF, Kern RS, Baade LE, Barch DM, Cohen JD, et al. , 2008. The MATRICS Consensus Cognitive Battery, Part 1: Test Selection, Reliability, and Validity. Am. J. Psychiatry 165 (2), 203–213. [DOI] [PubMed] [Google Scholar]
  38. Penney D, Sauvé G, Mendelson D, Thibaudeau É, Moritz S, Lepage M, 2022. Immediate and Sustained Outcomes and Moderators Associated With Metacognitive Training for Psychosis: A Systematic Review and Meta-analysis. JAMA Psychiatry 79 (5), 417–429. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Perez MM, Tercero BA, Penn DL, Pinkham AE, Harvey PD, 2020. Overconfidence in social cognitive decision making: Correlations with social cognitive and neurocognitive performance in participants with schizophrenia and healthy individuals. Schizophr. Res 224, 51–57. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Pinkham AE, Harvey PD, Penn DL, 2018. Social Cognition Psychometric Evaluation: Results of the Final Validation Study. Schizophr. Bull 44 (4), 737–748. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Prentice KJ, Gold JM, Buchanan RW, 2008. The Wisconsin Card Sorting impairment in schizophrenia is evident in the first four trials. Schizophr. Res 106 (1), 81–87. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Sheehan DV, Lecrubier Y, Sheehan KH, Amorim P, Janavs J, Weiller E, et al. , 1998. The Mini-International Neuropsychiatric Interview (M.I.N.I.): the development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. J. Clin. Psychiatry, 59 Suppl 20, 22–57. [PubMed] [Google Scholar]
  43. Silberstein JM, Pinkham AE, Penn DL, Harvey PD, 2018. Self-assessment of social cognitive ability in schizophrenia: Association with social cognitive test performance, informant assessments of social cognitive ability, and everyday outcomes. Schizophr. Res 199, 75–82. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Stirling JD, Hellewell JS, Quraishi N, 1998. Self-monitoring dysfunction and the schizophrenic symptoms of alien control. Psychol. Med 28 (3), 675–683. [DOI] [PubMed] [Google Scholar]
  45. Tercero BA, Perez MM, Mohsin N, Moore RC, Depp CA, Ackerman RA, et al. , 2021. Using a Meta-cognitive Wisconsin Card Sorting Test to measure introspective accuracy and biases in schizophrenia and bipolar disorder. J. Psychiatr. Res 140, 436–442. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES