Has the Generalized Deficit Become the Generalized Criticism?

Abstract

The “generalized cognitive deficit problem” refers to a situation in which a generalized deficit gives the false appearance of a specific deficit due to the psychometric properties of tests, and it is an important methodological consideration in schizophrenia research. However, it also generates considerable confusion and is often used indiscriminately as a scientific criticism, even in situations to which it does not apply. Further, the generalized deficit problem creates few concerns in interpretation for many central questions in contemporary schizophrenia research. The research literature has shifted away from the traditional goal of identifying generalized vs differential deficits, and the field now demonstrates (1) increased recognition that a generalized deficit, broadly defined, probably does not exist in schizophrenia, (2) increased emphasis on explaining both shared and unique variance across measures to understand the mechanisms through which cognition relates to external variables (eg, functional outcome), and (3) increased use of neuroscientific methods to explore cognition in schizophrenia in which the structure and richness of data can be used to minimize misinterpretation of the sort that can occur when using only behavioral measures. Clearly, consideration of the generalized deficit still remains essential in certain experimental contexts, but criticisms based on this concern are unwarranted in many other situations in schizophrenia research. This commentary is intended to help clarify the distinctions between these 2 situations so that concerns will be expressed in a more selective, less reflexive, manner.

The generalized deficit in schizophrenia research has become a confound in all senses of the word. On the one hand, it is a well-recognized methodological challenge that has produced spirited debate. On the other hand, this long-standing issue has generated confusion through its indiscriminant use as a criticism, both in situations to which it applies and in situations to which it does not.

The “generalized deficit” in schizophrenia refers to the tendency for patients to perform poorly relative to controls across a range of cognitive tasks. The “generalized deficit problem” occurs when the generalized deficit gives the false appearance of a specific deficit (ie, a differential deficit) due to the psychometric properties of the tests used. Although the topic of generalized vs differential deficits in schizophrenia has nearly disappeared from the research literature (see below), it is alive and well in the reviews of papers and grants. This commentary attempts to help discriminate between situations in which concerns about the generalized deficit are appropriate and warranted, and those in which the generalized deficit creates few, if any, concerns.

Background

Traditionally, experimental psychopathology has had the aim of “characterizing the disruptions in complex abilities and basic psychological processes that occur in mental disorders.”¹ This interest reflects an early emphasis on studying the psychological nature of cognitive impairments in schizophrenia,² separate from treatment, functional, or neuroscientific implications. Because patients with schizophrenia perform poorly on a range of cognitive tests, it is difficult to determine whether any 1 deficit is larger or more central than any other. At the time experimental psychopathology was emerging (post-World War II), classical clinical neuropsychology was influenced by the study of discrete brain lesions (eg, bullet wounds)³ that can lead to an entire syndrome. Similarly, one perspective in psychopathology was that damage to a single brain area and its associated consequences might explain broad impairment across cognitive domains in schizophrenia. This broad pattern of impairment, or generalized performance deficit, can be due directly to computational problems that affect the entire cognitive landscape, or indirectly to a motivational impairment that affects any demanding task.

To address the generalized deficit problem, investigators started to use a differential deficit strategy that typically included 2 groups (schizophrenia patients and healthy controls) and at least 2 behavioral measures. Larger differences (effect sizes) on one task than another (ie, a group by task interaction) were taken as evidence of a differential deficit. Starting in the 1970s, however, Loren and Jean Chapman provided compelling critiques of the differential deficit strategy.^4,5 They demonstrated that simply showing larger group differences on one task than another does not, by itself, constitute evidence of a differential deficit. If tasks differ in discriminating power (ie, true score variance defined as reliability times observed score variance), an apparent differential deficit could be due to this methodological artifact. For example, scale attenuation (a floor or ceiling effect) can reduce discriminatory power. How to match tasks on discriminating power, and whether it is wise to do so, became a focus of considerable debate.^1,6,7 A process of task matching can help reduce the likelihood that a differential deficit is a psychometric artifact. However, task matching is not easy to do, is not always successful, and could compromise construct validity, eg, when elements are added to a task to increase difficulty.^7–9

A discussion of the generalized deficit in schizophrenia research appeared over 10 years ago as a special issue of the Journal of Abnormal Psychology, ^1,7,10 but since then the topic has been barely visible. A PubMed search with the terms “differential deficit” and “schizophrenia,” and the 5 years 2007–2011 (entered separately) yielded only 3 articles that apply to the topics discussed in this commentary. Using the term “generalized deficit” instead of differential deficit yielded 6 articles. Some of these articles argued that a generalized deficit provides important clues to pathophysiology, as opposed to being only an artifact (see below). We noted a few important additional relevant articles^8,9 that our search did not pick up. The sizable presence of the generalized deficit in grant and article critiques is far out of proportion to its slim presence in the research literature.

Everyone agrees that the generalized deficit is a valid and important concern if one compares cognitive profiles of patients and controls across multiple domains and claims that some domains are more impaired than others.¹¹ In this commentary, we briefly describe 3 trends that should temper concern that the generalized deficit is a ubiquitous problem. First, it is becoming increasingly recognized that the generalized deficit, broadly defined, probably does not exist in schizophrenia. Second, there is an increasing focus on research questions, such as how cognition relates to external variables (eg, functional outcome), which employ joint (multivariable) relational models, emphasizing both shared variance and unique explanatory power. Third, many modern widely used neuroscientific methods, including electroencephalography (EEG) and functional magnetic resonance imaging (fMRI), are less susceptible to some of the types of measurement artifacts that originally generated concerns about the generalized deficit.

Failure to Document a Broad Generalized Deficit in Schizophrenia

Concerns may be calmed by the observation that the generalized deficit, broadly defined, probably does not exist in schizophrenia. The generalized cognitive deficit concept implies two things: (1) all areas of cognition show deficits and (2) all (or most) of the variance in cognitive performance is shared. We now have many examples of intact cognitive and emotional functions in schizophrenia, as well as measures with minimal shared variance. A provocative and well-documented review by Gold et al lists examples of spared functioning, including attentional selection for working memory storage, speed of attention shifting, forms of gradual implicit learning, and emotional experience.¹² Beyond the impressive list of domains in this review article, our laboratory has found intact functioning for certain aspects of emotion processing, such as subjective and physiological responses during exposure to rewarding stimuli, and context processing for facial emotion identification.^13,14 Another example of intact function comes from visual perception. Although patients show deficits in perception of shapes and objects, we find that the earliest components of vision are often intact across a range of difficulty levels (eg, contrast thresholds for identification, primary visual cortex activation, visible persistence, and color priming).^15–17

These findings of intact cognitive and emotional processes across laboratories and methods are inconsistent with a broad generalized deficit. Similarly, it is challenging to explain in the context of a generalized deficit why patients who are matched to controls on intelligence quotient show impairment on neurocognitive tests.¹⁸ Findings from a range of tasks (many that require active participation) also soothe worries that patients generally perform poorly on tests due to reduced motivation or inadequate effort.

Many of the examples of spared functions in schizophrenia come from highly specialized methods adapted from cognitive, social, and affective neuroscience. There are clear advantages to developing and adapting measures that isolate cognitive subprocesses or define narrow cognitive mechanisms, and indeed this was one goal of the Cognitive Neuroscience Treatment Research to Improve Cognition in Schizophrenia (CNTRICS) Initiative.^19,20 Such measures of discrete subprocesses would also be expected to have minimal shared variance with other similarly narrow measures, making them useful for identifying areas of specific deficit. As might be expected, some measures that grew out of the CNTRICS Initiative showed minimal correlations with other tasks.¹⁹ For example, the 2 measures of visual perception (visual integration and contrast-contrast effect for gain control) showed minimal correlations with memory and goal maintenance, as well as with each other.

If there is no broadly defined generalized deficit, then why are people worried about it? One reason may be that tasks that show intact performance or lack of shared variance are often highly specialized and therefore not widely used. Although the tasks are specialized, the range of areas that they cover is quite diverse and includes crucial aspects of cognition and emotion that are the focus of much contemporary research in schizophrenia. Hence, they are not limited to marginal or esoteric topics.

Concerns about a generalized deficit are easier to understand when interpreting results from commonly used neurocognitive batteries. When looking at standardized performance-based cognitive batteries that are designed to assess key domains of higher cognition one often finds 2 conditions that suggest a generalized deficit: (1) relatively comparable levels of impairment in schizophrenia across cognitive domains and (2) at least moderate interrelationships among the tests.^21,22 A single general cognitive process will usually account for some proportion of performance on each task. Although islands of intact functioning render a broad generalized deficit untenable, studies frequently use performance-based batteries that are derived from a particular framework of cognitive functions. In such situations, a generalized deficit will present interpretative problems when trying to compare relative levels of impairment.

Although some investigators consider shared variance across measures to be a problem, others consider it to be an important clue for underlying neural processes. Some studies have reversed focus from a search for a specific deficit to consider the implications of the shared variance on a neurocognitive battery for understanding pathophysiology.^23,24 For example, a relatively specific lesion could create a processing bottleneck or a reduction in coordination across processing modules. We find this focus on cross-domain processes such as coordination to be intriguing, and these more integrative processes might explain much of the profound functional impairment in schizophrenia. Also, the performance pattern may reflect a neurobiology underlying cognitive dysfunction in schizophrenia that is diffuse across brain regions. Prominent models of pathophysiology for schizophrenia implicate widespread dysfunction at the cellular level (eg, abnormalities in the NMDA receptor and/or GABA interneurons).^25,26 Such abnormalities would be expected to have similarly widespread implications for cognitive functioning and result in some degree of shared variance across measures. Indeed, viewed from these models of pathophysiology, it is surprising that so many islands of intact functioning can be demonstrated.

Research Focus on Shared Vs Unique Variance in Linkages Between Cognition and External Variables

Not all scientific questions regarding cognition in schizophrenia involve 2 groups or compare 2 measures. Another major trend is the focus on the relevance of cognitive impairments as they relate to predicting downstream real world outcomes. Models that predict an outcome variable are an entirely different ballgame than the models of cognitive performance comparing groups across tasks (ie, interactions) that have traditionally generated concern about the generalized deficit problem. Predictive analyses frequently occur within a single group so that relative sizes of group differences across tasks are not relevant. The central question is how well the tasks, individually and jointly, predict a particular outcome variable (eg, functional outcome). Furthermore, shared variance, which can range across perception to negative symptoms and functional outcome, is not the same as the generalized deficit. Reviewers who confuse the 2 concepts are likely to pounce on one by conflating it with the other.

Consider, for instance, a consistently replicated finding in schizophrenia that cognitive impairment relates to daily functioning. Statistical models can be constructed to show how constructs early in the processing stream can feed into later, more integrative, constructs and lead to daily functioning. Research indicates that early perception relates to cognition, nonsocial cognition relates to social cognition, and social cognition relates to motivation and functioning.²⁷

For these models of prediction of functional outcome (or any type of model with a network of components), there is no advantage in only isolating variables that are largely independent of one another (ie, only specific deficits). The interpretative advantage comes from finding which components of the model have variance that is shared across other latent and measured variables, and which components have unique explanatory power, above and beyond that shared variance. If a measure has unique explanatory power, then observed relationships are not easily attributable to a generalized deficit.

Some of these models can be relatively complex and wide-ranging. For instance, social cognition reliably mediates the relationship between nonsocial neurocogni tion and daily functioning.²⁸ True mediation requires that the potential mediator (eg, social cognition) be related to the outcome even after controlling for the initial variable—ie, it has some unique explanatory power. With a truly generalized deficit, that unique explanatory power would not exist; there simply would be associations, not mediation. With more complex relational structures examined with techniques such as structural equation models (eg, starting with early perception, including negative symptoms), the generalized deficit cannot explain good model fit because each path controls for the others. The informational value of such models comes from understanding the interplay between shared and unique variance, and models simply do not work if there is only shared variance.

Such models are extremely important for understanding relationships among cognition and functioning and for exploring the underlying neural mechanisms. However, there are important interpretive caveats: one is that under unusual conditions it is possible for a generalized deficit to masquerade as a combination of shared and unique variance. For example, suppose that 2 cognitive measures tap an underlying generalized deficit but one of them does a better job of capturing the construct (ie, has less noise or explains more variability) than the other. In this case, the psychometrically superior measure may appear to explain unique variance simply because of measurement artifacts. This situation can be avoided by selecting measures in the battery with good reliability, range, and construct validity. Although measures certainly differ in their psychometrics, it is unreasonable to believe that all cases of unique variance can be attributed to lack of construct validity or reliability. Nor is it plausible that outcome models that include a range of domains beyond cognition (eg, perception and negative symptoms) would have systematically poor measures of the same general deficit and produce artifacts large enough to create the impression of unique variance for well-established constructs.

In the other direction, even if all the relationships in a model are entirely explained by shared variance (consistent with a generalized deficit), that should not be taken to mean that the individual constructs are uninteresting or unimportant. For example, variables may represent pieces of a sequential process or cascade in which A directly causes B which in turn directly causes C. If this case, each of A, B, and C represents a crucial component of the underlying mechanism and should not be regarded as irrelevant even though they will have no unique explanatory power in a predictive model.

Finally, insights from this type of modeling paradigm suggest that unique and shared variance provides an alternative explanation for a problem that was typically viewed as part of the generalized deficit: when a deficit on Task A disappears while controlling for Task B, rendering it non-specific. For years, researchers debated whether observed patient-control differences in facial affect identification hold up after controlling for facial (hold the emotion) identification. After many years of studies the answer is sometimes yes and sometimes no. Few people questioned the importance of the exercise. But facial affect identification and structural (nonemotional) face processing have both shared and unique variance. Depending on specific measures and samples, the correlation between structural face processing and affect identification sometimes will happen to be high enough so that statistical control will eliminate the group difference, and sometimes not. Because of this shared variance, controlling for face identification renders facial affect identification less ecologically valid—the residual (unique) portion of the task is no longer comparable to processing facial emotions in the real world.

Widespread Use of Neuroscientific Methods, Including EEG and fMRI, in Studies of Schizophrenia Cognition

Another factor that helps to diminish concerns about the generalized deficit in schizophrenia is the increasing application of neuroscientific methods such as EEG or fMRI to examine cognitive impairment. To be clear, the use of EEG or fMRI by itself does not make the generalized deficit problem disappear. The psychometric properties of an activation task in the scanner deserve just as much attention as in a behavioral study. However, these methods can lessen concerns in a few ways. Ceiling and floor effects, which are one source of erroneous conclusions about a differential deficit with behavioral measures, do not mean the same thing for physiological methods. A physiological ceiling effect on EEG/fMRI reflects the limit of a biological system; a behavioral ceiling effect reflects the limit of a measurement system. The generalized deficit problem occurs when a generalized deficit gives the false appearance of a differential deficit, not when ceiling or floor effects (and corresponding presence/absence of group differences) represent a biological reality.

Moreover, because many interesting and reliable findings between patients and controls in EEG come from tasks that are ridiculously simple, they are not easily explained by a motivational impairment in patients. Some tasks, such as the odd ball paradigm or mismatch negativity involve difference waves between rare and common stimuli. Although the tasks often involve “catch trials” to ensure the subject is attending, differences between stimuli are so obvious that no real effort (or even awareness) is needed. Also, one approach in EEG is to look at well-established waveforms at early, middle, or late components (eg, P100, N100, P300, late positive potentials) for the same paradigm. Group differences that emerge at some stages but not others are likely to reflect valid (not artifactual) neural processing differences.

Because fMRI yields a wealth of information, it provides added ways to avoid misinterpretations that might occur with behavioral tasks alone.²⁹ The images are based on contrasts, usually both within and between groups. Some degree of protection from the generalized deficit problem comes with a thoughtful choice of control tasks that include similarly demanding processes not of interest in the experimental condition. For example, one can control for nonsocial cognitive processes of reading, memory, or speed of processing while assessing social cognition in the scanner. One can then isolate the neural underpinning of social vs nonsocial cognitive processing but with reduced ecological validity (because real world interactions never involve purely social cognitive processes). Of course, fMRI has its own methodological and inferential issues including challenges in establishing reliability of measurement and avoiding circularity in data processing.^30,31 These concerns, although important and substantial, are different from the generalized deficit confound being discussed here.

Because fMRI can reveal qualitative patterns on multiple dimensions, it provides ways to determine whether results are consistent with a unidimensional generalized deficit. Groups may differ in brain regions activated in a certain task, or they may differ qualitatively when difficulty of a task is parametrically manipulated. Regional measurement typically involves patterns of responses that are multidimensional or nonlinear, such as the inverse-U response seen with working memory tasks.³² Brain activation differences between groups at similar levels of performance may reflect how each group responds on a task to achieve a level of competence (eg, through compensatory use of alternative brain regions). Aside from magnitude of response, groups can also differ in how efficiently they perform a task at different levels, eg, in the number of voxels recruited. As with EEG, these patterns are thought to reflect real physiological differences between groups, not artifacts that underpin the generalized deficit problem.

Summary

In this limited space, our intention was to offer a counter-point to the view that generalized deficits in schizophrenia are always lurking and always problematic. We fully believe that concerns about a generalized deficit are often warranted, such as when interpreting performance profiles across domains or group by task interactions. Further, the psychometric qualities of a performance test are always an important consideration, even if combined with neuroscientific methods such as EEG or fMRI. However, reviewers need not adopt a default assumption that generalized deficits always confound interpretation of findings.

In this commentary, we gave examples of areas that are relatively intact in schizophrenia, which challenges the notion of a broad generalized deficit in schizophrenia. We also discussed the importance of understanding both shared and unique variance across cognitive measures in predicting an outcome variable. Models of perception, cognition, motivation, and outcome in schizophrenia celebrate shared variance across domains as clues to causal mechanisms, just as some researchers focus on the commonalities of cognitive deficits as clues to pathophysiology. Concerns about the psychometrics of measures do not go in and out of style, but research topics do. A focus on generalized vs differential cognitive deficits in schizophrenia has given way to a host of other scientific questions, including clinical trials for psychopharmacological and training interventions, models of functional and clinical outcome, and exploration of underlying neural and genetic substrates, among many others. As cognitive research in schizophrenia expands into unexplored new territory, the implications of a generalized deficit will continue to be explored. We hope that concerns will continue to be expressed—but in a selective, not reflexive, manner.