Measurement Issues in the Use of Cognitive Neuroscience Tasks in Drug Development for Impaired Cognition in Schizophrenia: A Report of the Second Consensus Building Conference of the CNTRICS Initiative

Abstract

This overview describes the goals and objectives of the second conference conducted as part of the Cognitive Neuroscience Treatment Research to Improve Cognition in Schizophrenia (CNTRICS) initiative. This second conference was informed by a series of online surveys and brought together basic and clinical scientists from academia and industry to address the concerns central to each field of research. Our goal was to develop recommendations for future research addressing the psychometric and practical challenges involved in translating paradigms from cognitive neuroscience into tasks that are feasible for use in the treatment discovery and development process. In this overview article, we describe the series of talks that were presentations at the conference. This article serves as an introduction to the set of articles included in this special issue that provide overviews and discussions of the issues raised and the recommendations made in these talks and in the subsequent discussions at the meeting. In addition, we describe the online surveys conducted in the month before the conference that were used to obtain suggestions from the field as to important task selection criteria and to generate initial benchmark goals for psychometric development for cognitive neuroscience tasks.

Over the past decade and a half, there has been a growing awareness of the importance of impaired cognition in schizophrenia as a critical “glass ceiling” that limits functional outcome for people with the illness.¹ For example, many people with schizophrenia continue to have problems with memory and problem solving, along with difficulties of living and working independently, despite the fact that their hallucinations and delusions may be well controlled by their current antipsychotic medications. During the 1990s, there was initial enthusiasm that second-generation antipsychotic drugs would confer significant advantages over first-generation agents for this aspect of the illness. However, it has now become clear that the data are disappointing in this regard.² This understanding has resulted in a growing awareness of an urgent need for the discovery and development of new treatments for schizophrenia that will enhance cognitive functioning in the illness and improve functional outcome. It is widely recognized that this is one of the major challenges for psychopharmacology in the 21st century, with the stakes highest for patients with a diagnosis of schizophrenia and their families.³

An important milestone in the quest for cognition-enhancing therapies for schizophrenia was the successful completion of the Measurement and Treatment Research to Improve Cognition in Schizophrenia (MATRICS) initiative.^4,5 This National Institute of Mental Health (NIMH)–sponsored initiative brought together clinical researchers from academia, the pharmaceutical industry, and the Food and Drug Administration to map out a process by which approval could be obtained for drugs targeting cognition in schizophrenia and a battery of cognitive tasks that could be rapidly developed for use in phase III clinical trials.⁵ As part of this process, the MATRICS Neurocognition Committee went through a very careful process of using theoretical and empirical data (eg, factor analysis) to identify 7 separable cognitive domains in schizophrenia.^6,7 MATRICS then selected a set of clinical neuropsychological measures (the MATRICS Consensus Cognitive Battery) to assess these 7 domains based on criteria such as psychometric characteristics (eg, test-retest reliability and repeatability), relationship to functional outcome, practicality and tolerability, and potential changeability in response to pharmacological agents.⁵ The Cognitive Neuroscience Treatment Research to Improve Cognition in Schizophrenia (CNTRICS) initiative grew out of the last of 6 meetings conducted during MATRICS. In this last meeting, “new approaches” such as those for measuring behavior and related brain activity from cognitive neuroscience were considered for future development (see Schizophrenia Bulletin, Volume 31, 2005). It was agreed at this meeting that such measures could convey significant advantages as they would facilitate translational research by more seamlessly bridging animal models of cognition to human models, as well as providing more specific measurement of cognitive processing that could be related to discrete neural systems. In particular, it was noted that increased specificity had the potential to convey increased sensitivity to drug effects that targeted discrete neural systems supporting cognition in the brain. This would result if the increased specificity removed the influence of confounding factors that might not be sensitive to drug effects. However, cognitive neuroscience-based measures were not included in the MATRICS battery because it was felt there was insufficient data with regard to their psychometric properties as well as concerns regarding the practicalities of administration of these tasks in the clinical trials setting.

The CNTRICS initiative was designed to begin to address these concerns regarding cognitive neuroscience paradigms, starting with the first of 3 conferences in Washington, DC, in February 2007. Funded by an R13 grant from the NIMH, a detailed description of the process has been published elsewhere.⁸ Briefly, the goal of CNTRICS was to bring basic cognitive scientists, clinical investigators, and those involved in drug discovery and development together to initiate the translations of a set of cognitive neuroscience–based tools that can be used to facilitate drug development targeting impaired cognition in schizophrenia. At the initial meeting, a set of cognitive systems and component processes were identified as targets for treatment development in schizophrenia. These included: (1) executive control, (2) working memory, (3) long-term learning and memory (including reinforcement learning), (4) attention, (5) perception, and (6) social and emotional processing. The consensus building process was guided by a set of formal criteria developed through a web-based survey. These criteria included: (1) strong evidence for the validity of the construct in the field of cognitive neuroscience, (2) evidence of impairment in schizophrenia, (3) established links with known neural circuits and neurotransmitter systems, (4) ease of implementation in functional imaging studies; and (5) the availability of (or potential for) animal homologues. Lack of data regarding psychometric properties and practicalities of administration for measures of these constructs were explicitly not considered at this meeting and in the initial construct selection process. Further, the decision was not to use the absence of psychometric data as an exclusion criterion during the task selection process that would occur during the third and final meeting in March of 2008. However, it was still very clear to all those involved in CNTRICS that psychometric and practical issues were critical, and that we needed to gain a better sense of the challenges we would face in the translation process, and that we needed to develop suggestions or recommendations for how to think about psychometric and practicality goals during the translation of cognitive neuroscience tasks into a format useful in clinical trials.

The objectives of the second CNTRICS meeting, which we are reporting in this special issue of Schizophrenia Bulletin, were 2-fold. The first objective was to directly address the problem of a lack of psychometric data for cognitive neuroscience tasks, as well issues related to the practicalities of integrating cognitive neuroscience into the drug development process. In doing so, we sought to overcome one of the chief barriers to the translation of basic cognitive neuroscience methods into the drug development process. Specifically, this meeting focused on developing guidelines for addressing concerns regarding the psychometric properties of such tasks, including some preliminary consensus-building “benchmark” values to be pursued during task development. The second objective was to explicitly bring together basic scientists and clinical scientists so that they could begin to understand and appreciate the concerns and unique challenges facing each of their research domains. Basic cognitive neuroscientists spend a great deal of time thinking about and establishing the construct validity of their paradigms, their ability to measure specific processes (and hence deficits), and their links to neural systems. Many (if not most) basic scientists do not spend a great deal of time worrying about psychometric issues such as test-retest reliability, practice effects, etc, or about how easy it would be to use their tasks across many sites in a clinical trial because these are not issues critical to their research programs. In contrast, clinical scientists interested in drug development and treatment spend a great deal of time concerned with psychometric and practical issues that can have an major impact on the ability to validly assess cognitive change and enhancement. In initiating a dialogue between basic and clinical scientists around these issues, we sought to facilitate the translation of cognitive neuroscience paradigms into formats feasible for use in clinical trials.

To achieve these goals, we asked a set of both basic scientists and clinical scientists to give talks that addressed 6 major areas of interest and challenge from both the basic and clinical perspectives. From the basic science side, we addressed: (1) how construct validity is established in cognitive psychology and cognitive neuroscience and how it might be maintained during translation (Jonathan Cohen) and (2) how an individual differences perspective can inform construct validity (Randy Engle). From the clinical science side we addressed: (1) strategies for the measurement of differential deficits in specific cognitive processes, as opposed to generalized deficits that can impair task performance⁹,¹¹ (eg, poor motivation, sedation due to medications, general inattention or poor test taking skills) (Steve Silverstein); (2) the theoretical and practical challenges of translating basic cognitive experimental paradigms into tasks to be used in clinical populations (Steve Luck and Jim Gold); (3) the practicalities of measuring cognitive functioning in the clinical trials context, including the challenges involved in multisite implementation of cognitive tasks (Phil Harvey and Richard Keefe); and (4) statistical and power concerns in clinical trials in relationship to reliability, multiplicity, and practice effects (Andy Leon).

In addressing these issues, the meeting participants sought to provide guidance for future efforts to translate experimental paradigms from basic cognitive science and neuroscience into useful tools for the drug development process. The results of this meeting were not meant to be definitive or prescriptive but rather to provide food for thought and guidance for the next generation of translational studies. The presentations are summarized in the 5 articles that comprise this special section of the Schizophrenia Bulletin. As described in this series of articles, there were a number of important issues discussed at this meeting that helped to inform the design of the third meeting and the task nomination process. These included: (1) ways in which to think about what evidence serves to establish construct validity of a task; (2) the types of psychometric challenges different tasks are likely to face (eg, accuracy vs reaction time–based tasks); (3) the tension between a desire to establish differential deficits and the subtractive design logic often guiding cognitive neuroscience task development; and (4) the sorts of challenges tasks with several dependent measures would provide for the design and analysis of clinical trials. The slide presentations may be viewed online at cntrics.ucavis.edu. It is important to note that both the process and content of the meeting were developed in close consultation with the members of the CNTRICS Executive Committee and informed by a series of web-based surveys that sought to develop a set of priority issues to be addressed at the conference. In the next section, we will describe these surveys and present the results.

Results of the Preconference Survey and Overview of the Recommendations

There are a number of different characteristics of tasks that are relevant to their utility in clinical trials focusing on cognition-enhancing treatments. These characteristics include construct validity, psychometric properties (test-retest, practice effect, etc.) and practical issues (length, ease of administration). Although an ideal task would perform well on all such dimensions, in reality it is often the case that some tasks do very well on some dimensions, and less well on others. In addition, it may be the case that different types of task characteristics are more or less important for different types of clinical trial settings. For example, one distinction that the CNTRICs Executive Committee felt might be important was the distinction between small-scale clinical trials, such as Phase 1 or Phase 2 trials, and large-scale clinical trials, such as Phase 3 trials. As such, the CNTRICs Executive Committee felt it would be important to survey individuals from academia and industry in both basic and clinical areas as to the relative importance of different task characteristics for small and large-scale clinical trials. This survey was conducted online in the month prior to the conference.

To do so, the CNTRICS Executive Committee developed a list of task characteristics likely to be relevant for cognition enhancement clinical trials, including: (1) internal consistency; (2) test-retest reliability; (3) alternative form reliability; (4) short length; (5) adequate number of trials for robust estimate of condition; (6) construct validity; (7) lack of practice effects; and (8) lack of floor/ceiling effects. We asked survey respondents to rate the importance of these characteristics using the following scale: 0-Not Important; 1-Somewhat Helpful; 2-Very Helpful But Not Essential; 3-Somewhat Essential; 4-Very Essential. We asked respondents to make these ratings twice, once for the relative importance of these characteristics in small-scale clinical trials and once for their relative importance for large-scale clinical trials. We invited over 200 individuals from academia and industry to participate in the survey. Their domains of expertise included individuals from academia and industry, the basic and clinical sciences, as well as individuals with experience in clinical trials and cognitive rehabilitation in schizophrenia. We used several methods to generate the list of individuals asked to participate in the survey, including: (1) the names of those individuals that were involved in the MATRICS project; (2) individuals serving on the editorial boards of basic and clinical cognitive science, cognitive neuroscience, and schizophrenia related journals; (3) individuals from as many small and large industry organization as could be identified by the CNTRICS steering committee. We received full responses from 96 individuals. Of these 96 responders, 77% were from academia or government, and 23% were from industry. Further, of these 96 responders, 57% percent labeled themselves as being involved in clinical trials or cognitive research in schizophrenia, 37% labeled themselves as being involved in human or animal cognitive neuroscience, and 6% labeled themselves as being involved in psychometrics. The rankings provided by these respondents are shown in table 1 for both the small and large-scale clinical trials.

Table 1.

Results of Preconference Survey for Second CNTRICS Meeting: Rankings of Task Characteristics for Small- and Large-Scale Clinical Trials

N = 96	Small-Scale Clinical Trials				Large-Scale Clinical Trials
Task Characteristics	Mean	Mode	Median	SD	Mean	Mode	Median	SD
Test-retest reliability	3.5	4	4	0.74	3.3	4	4	0.81
Construct validity	3.3	4	3.5	0.79	3.2	4	3	0.95
Lack of floor/ceiling effects	3.2	4	3	0.92	3.2	4	3	0.90
Internal consistency	2.9	3	3	1.03	3.0	3	3	0.96
Adequate number of trials	2.9	3	3	0.96	2.9	3	3	0.94
Short length	2.2	2	2	0.83	2.8	3	3	0.96
Alternate form reliability	2.4	2	2	0.95	2.6	2	3	1.00
Lack of practice effects	2.4	2	2	1.03	2.5	2	2	0.95

Note: Task characteristics are listed in descending order of ranked importance using a combination of mean, mode, and median values. Rating scale was: 0—not important; 1—somewhat helpful; 2—very helpful but not essential; 3—somewhat essential; 4—very essential.

As can be seen in table 1, there was a relatively similar ranking of task characteristics for their importance for both small and large-scale clinical trials. Test-retest reliability was considered to be the most important characteristic for both types of clinical trials, followed closely by construct validity and a lack of floor and ceiling effects. The importance of internal consistency and an adequate number of trials were ranked somewhat lower, but similarly for both small and large scale clinical trials. The importance of a short task length was ranked as more important for a large-scale clinical trial than for a small-scale trial, which makes sense in terms of the different demands of such trials in terms of the number of participants to be included and the intensity of the procedures. Alternate form reliability was rated somewhat higher for large than small scale clinical trials, while a lack of practice effects was rated similarly for the 2 types of trials. Thus, these survey results suggest that the 3 most important characteristics to attend to when translating paradigm from cognitive neuroscience into a format useful to clinical trials will be to ensure adequate test-retest reliability, to maintain construct validity, and to ensure a lack of floor and ceiling effects. Obviously the other task characteristics are important as well, but may need to be balanced against the importance of the task characteristics rated as most necessary.

The survey data reviewed above provide some guidance as to where to focus initial efforts to assess and enhance the psychometric characteristics of paradigms derived from cognitive neuroscience. However, they do not give a sense of what precise values or goals we should be aiming for during this process. For example, although we know that test-retest reliability is important, what is an optimal, or even a sufficient level of test-retest reliability for the purposes of a clinical trial? Clearly a lack of floor and ceiling effects is important, but how does one gauge whether there is an adequate range of scores to allow valid interpretation of cognitive change? To provide some guidelines for answering these questions, we also asked survey participants to provide us with their estimates as to the optimal values for a range of task characteristics, as well as what they saw as the lowest acceptable values for these characteristics. Further, we asked respondents to rate their confidence in their estimates, given the range of expertise among the survey participants.

The estimates provided by the survey respondents are shown in table 2, both for the whole sample and for those respondents who rated themselves as highly confident in their responses. As can be seen in table 2, the recommended values provided by the entire sample were fairly similar to the values provided by those who rated themselves as very or extremely confident in their estimates. However, the values for the highly confident responses tended to be higher for the reliability estimates, more stringent in terms of the floor and ceiling effects, and smaller in terms of the task length and number of trials. The values suggested for optimal reliability characteristics fall within the range of what is generally considered to be good reliability and we suggest provide clear guidance for the assessment and improvement of reliabilities in studies of cognitive neuroscience tasks. The values provided for the floor and ceiling effects (particularly ceiling effects) may appear somewhat daunting to basic scientists, who often develop tasks with mean accuracy values in the 90% range. This is often because such paradigms focus on patterns in reaction times rather than accuracy rates. However, much of this initial basic science research has been done with college populations who typically score much better than patient samples or even than community control samples. Thus, we may find that tasks that have what we consider to be unacceptable ceiling effects in undergraduates perform quite well in this regard in samples more akin to those that would be studied in clinical trials.

Table 2.

Results of Preconference Survey for Second CNTRICS Meeting: Estimates of Benchmark Values

	All Responses		Responses rated as “Very” or “Extremely” Confident
Task Characteristic	Optimal Value	Worst Acceptable Value	Optimal Value	Worst Acceptable Value
Test-retest reliability	0.90	0.70	0.90	0.70
Floor effect (% difference from 0)	20	10	25	10
Ceiling effect (% difference from 100)	25	15	37.5	12.5
Internal consistency	0.85	0.60	0.88	0.60
Minimum number of trials	30	15	25	10
Length (in min)	15	30	10	30
Alternate form reliability	0.90	0.65	0.90	0.70
Practice effects (in SD units)	0.20	0.50	0.20	0.50

Some basic scientists may also find the value estimates for the number of trials and the length of tasks rather low. Many articles in the basic science literature report on studies in which hundreds of trials are collected in a single task, which may often take more than 30 minutes to acquire. Thus, an initial reaction might be to feel that one cannot possibility obtain a valid estimate of the construct of interest in as few as 25 or 30 trials or in a task that takes only 10 or 15 minutes. However, there are actually many examples in the basic cognitive science literature of tasks that are short enough to meet these suggested values, so it is clearly possible to design such tasks. These would include versions of the single-trial Stroop, attentional flanker tasks, versions of the Simon Task, and N-back working memory tasks. Thus, one of the challenges facing the translation process will be to determine an optimal balance between length and the maintenance of construct validity, so that cognitive neuroscience paradigms can be optimized to be as efficient as possible without sacrificing the ability to provide a valid estimate of a specific cognitive construct.

As noted above, the responses acquired during the pre-conference survey for the second CNTRICS conference are not meant to be definitive. Rather, they are meant to provide some initial guidance for those investigators willing to take on the challenges of translating and optimizing the tasks that will be selected in the third CNTRICS meeting, or any additional tasks that investigators wish to move into a clinical trials context. Further, we should note that there is no “perfect” value for any of the psychometric characteristics that we discussed above. As any clinical trials statistician will report, a low reliability or high practice effect value (eg) does not definitively preclude one from using that measure in a clinical trial. However, unless the factors that lead to the low reliability or practice effects can be dealt with proactively in the study design, it will have specific ramifications for sample size and power that may or may not be feasible at a financial or practical level. Thus, we hope that providing investigators with a sense of the task characteristics on which they should focus, and providing some “benchmark” values to guide future studies will help to make the process more manageable for scientists who might be new to this particular type of endeavor. Further, we hope that the process of bringing together basic and clinical scientists to share their diverging views and to learn about the challenges and concerns each face in their own research domains will foster a collaborative environment that will lead to concrete and productive collaborations in the process of translation cognitive neuroscience paradigms for use in clinical trials.

Funding

National Institute of Mental Health (MH0786710, MH066629, MH45156, AG10129, MH55714, MH47073, 2R01MH059883 to C.S.C.; MH066031, MH62130, MH056584, MH0786710 to D.M.B.); Robert Wood Johnson Foundation (to C.S.C.); National Alliance for Research on Schizophrenia and Depression (NARSAD; to C.S.C., D.M.B.); McDonnell Foundation (to D.M.B.).