Skip to main content
Schizophrenia Bulletin logoLink to Schizophrenia Bulletin
. 2011 Nov 28;38(1):114–124. doi: 10.1093/schbul/sbr146

Relational and Item-Specific Encoding (RISE): Task Development and Psychometric Characteristics

John D Ragland 1,*, Charan Ranganath 2, Deanna M Barch 3, James M Gold 4, Brittaney Haley 1, Angus W MacDonald III 5, Steven M Silverstein 6, Milton E Strauss 7, Andrew P Yonelinas 2, Cameron S Carter 1
PMCID: PMC3245591  PMID: 22124089

Abstract

Objectives: The Relational and Item-Specific Encoding task (RISE) was designed to assess contributions of specific encoding and retrieval processes to episodic memory in schizophrenia. This manuscript describes how a cognitive neuroscience functional imaging paradigm was translated for clinical research. Methods: The RISE manipulates encoding by requiring participants to decide whether stimuli are “living/nonliving” (item-specific) or whether one stimulus fits inside the other (relational) and estimates familiarity (F) and recollection (R) by examining receiver operator characteristics (ROC) and assessing item and associative recognition. Two studies examined psychometric characteristics and tested the hypothesis that patients have differential deficits in relational vs item-specific encoding and disproportionate impairments in recollection vs familiarity. Results: Study 1, using visual objects, provided support for the encoding hypotheses and revealed good internal consistency and alternate forms reliability, with small differences between test forms. ROC analysis revealed R and F deficits, with F deficits most prominent following relational encoding. Study 2 used word stimuli, which lowered item recognition, but patients had difficulty understanding task demands, and words were less desirable for non-English speaking clinical trials, leading to the decision to proceed with the original task. Conclusions: The RISE is a valid and reliable measure of item-specific and relational memory that is well tolerated, with good psychometric characteristics and equivalent forms to facilitate treatment studies. Results indicate that episodic memory in schizophrenia is most preserved under conditions promoting item-specific encoding that is supported by familiarity-based recognition and is most impaired under relational encoding and recollection-based retrieval conditions.

Keywords: neurocognition, episodic memory, relational processing, item-specific processing, schizophrenia

Introduction

Episodic memory1 refers to the ability to remember past events. Although individuals with schizophrenia have pronounced episodic memory impairments,2,3 it remains unclear whether these deficits are due to problems at encoding, information retrieval, or some combination. The Relational and Item-Specific Encoding task (RISE) was designed to assess contributions of different encoding and retrieval processes to episodic memory in schizophrenia. Here, we describe how this paradigm, inspired by basic cognitive neuroscience research,4 was translated for clinical research. In addition to reporting group differences, psychometric data are provided on internal consistency, alternate forms reliability, presence of ceiling or floor effects, and difficulty levels between task conditions.

The encoding manipulation in the RISE comes from basic research distinguishing between “relational” encoding that promotes memory for associations between items and “item-specific” encoding that enhances distinctiveness of specific items.57 In the RISE, relational encoding was operationalized by presenting pairs of items and prompting participants to decide whether, in real life, one of the items could fit inside the other (see figure 1). Item-specific encoding was operationalized by presenting single items and prompting participants to decide whether each was living or nonliving. Studies have shown that, although both processes are effective, they tend to have different effects on memory.5,6,8,9 For instance, item-specific encoding is optimal when source memory for item details is tested, whereas relational encoding is optimal when memory for associations between items is tested (eg, associative recognition).10 Nonetheless, both item-specific and relational encoding can benefit performance on item recognition tasks to an equivalent degree.4,9 Little work has been done to dissociate these 2 encoding processes in schizophrenia. However, retrieval studies using list learning,11 inference tasks,12 and transverse patterning tasks13 have found that patients have severe deficits when relational information must be retrieved,14 but relatively unimpaired performance when provided with item-specific encoding strategies and asked to recognize whether items were previously studied.15 Thus, our first hypothesis was that individuals with schizophrenia have a disproportionate impairment in their use of relational vs item-specific encoding and, as a result, show disproportionate deficits for items studied during relational encoding.

Fig. 1.

Fig. 1.

Illustration of item-specific and relational test procedures and task stimuli. (A) Memory encoding and (B) memory retrieval.

The RISE also assesses retrieval by incorporating tests of item and associative recognition. During item recognition, participants make “old/new” judgments on a series of items and indicate decision confidence (low, medium, or high). Item recognition can be supported by assessing familiarity of studied items or by recollecting contextual details associated with study events.16,17 By combining old/new and confidence data, receiver operator characteristic (ROC) analysis18 can examine recognition discriminability as a function of response criterion. In addition, a dual-process model19 can be used to estimate the extent to which item recognition is supported by familiarity (F) or recollection (R). During the associative recognition test, participants are presented with “intact” studied item pairs or “rearranged” item pairs consisting of studied items not originally paired together. Because all items on the test are familiar, associative recognition relies largely on recollection of some aspect of the encoding event to discriminate studied and rearranged pairs. Previous research of schizophrenia patients using the remember/know (R/K) test procedure20 have suggested that patients have a specific recollection deficit.2123 Therefore, our second hypothesis was that patients have disproportionate deficits in recollection vs familiarity-based retrieval that would be evident in ROC analysis of item recognition. Also, because associative recognition relies more on recollection than does item recognition, we predicted that patients would be disproportionately impaired on associative recognition.

Methods

Study 1

Participants.

As described in our companion article,24 recruitment was through the Cognitive Neuroscience Test Reliability And Clinical applications for Schizophrenia (CNTRACS) Consortium, including 5 research sites: University of California—Davis, Maryland Psychiatric Research Center at the University of Maryland, University of Medicine and Dentistry of New Jersey, University of Minnesota—Twin Cities, and Washington University in St Louis. Participants were recruited nearly equally across the 5 sites and were recruited from outpatient psychiatric clinics, community centers, and local settings via flyers, and online advertisements.

Data were obtained on 74 healthy controls and 104 patients with schizophrenia. One control and 9 patients were excluded for below-chance performance on any version of the item and associative recognition tasks, with 2 additional patients excluded for a positive drug screen, leaving a final sample of 73 controls and 93 patients (table 1). Below-chance performance was defined as a hit rate (ie, correct identification of studied items) below 0.50. T-tests revealed that excluded patients were older (47.9 ± 5.2 vs 37.5 ± 11.7 y, P < .0001) and had lower estimated community function (Specific Level of Functioning [SLOF] scale total = 4.00 ± 0.25 vs 4.28 ± 0.65, P < .05). All but one patient was receiving antipsychotic medication (14 typical and 78 atypical). Groups were matched on age and parental education. However, patients had a larger proportion of males and lower estimated premorbid intellectual abilities (Wechsler Test of Adult Reading [WTAR]). Therefore, any between-group differences in the full sample will also be investigated using a smaller control sample that is also matched on gender and WTAR. After complete description of the study, written informed consent was obtained. The study was approved by the Institutional Review Board at all participating research sites.

Table 1.

Study 1 Demographic Characteristics

Variable Controls (n = 73)
Patients (n = 93)
P Value
Mean SD Mean SD
Age (y) 36.9 12.2 37.5 11.7 ns
WTAR 37.0 8.3 32.9 9.0 <.0001
Education (y) 14.7 2.2 13.2 2.3 <.0001
Parental education (y) 13.6 2.6 13.3 2.8 ns
Gender (% male) 51 68 <.05
Handedness (% right) 82 81 ns
BPRS—depressed –– –– 7.5 3.7 ––
BPRS—positive –– –– 8.8 4.9 ––
BPRS—disorganized –– –– 5.2 1.9 ––
BPRS—negative –– –– 2.2 1.2 ––
BPRS—manic –– –– 5.1 2.0 ––
UPSA-B –– –– 76.9 12.4 ––
SLFS_P –– –– 4.3 0.4 ––
SLFS_I –– –– 4.1 0.6 ––

Note: ns, not significant group difference at P < .05, 2 tailed. BPRS, Brief Psychiatric Rating Scale; UPSA, University of California San Diego Performance-Based Skills Assessment.

Materials.

Stimuli consisted of 432 visual object representations of word stimuli employed in the original study.4 Visual objects were used to improve understanding of encoding instructions and facilitate use in future non-English speaking clinical trials. Visual objects were selected from a standardized corpus of photographic images (http://cvcl.mit.edu/MM/)25 and divided into groups of 144 stimuli for each of the 3 test forms. For each form, a list of 36 items was assigned to item encoding, 36 assigned to relational encoding, and 72 served as foils for recognition testing. Lists and test forms were matched for the word length and frequency of the original word stimuli based upon study norms.4

Procedure.

Tasks were presented with E-Prime(r) (version 2.0). Subjects performed 2 incidental encoding tasks (figure 1A): (1) item-specific encoding—36 stimuli were presented for 2 seconds each, with a 1-second interstimulus interval (ISI), and subjects made a 2-button “yes/no” response indicating whether objects were “living.” This semantic encoding manipulation controlled for potential group differences in strategy generation15 and was superior to a “pleasant/unpleasant” decision that produced ceiling effects during piloting. (2) Relational encoding—18 object pairs were presented for 4 seconds, with a 1-second ISI, and subjects made a 2-button yes/no response indicating whether 1 item could fit inside the other. Because individuals with schizophrenia have prominent task-switching difficulties,26 encoding conditions were alternated in a pseudorandom block design (3 item blocks—12 trials each and 3 relational blocks—6 trials each). Three-second instruction screens announced changes between encoding blocks and encoding decisions (“living?” or “inside?”) remained visible to remind subjects of the current condition.

After encoding, 2 retrieval tasks were administered (figure 1B): (1) item recognition—all 72 studied objects (36 item-specific and 36 relational targets) were randomly intermixed and presented with 72 new unstudied foils, and subjects indicated whether each item was “old” (left hand response) or “new” (right hand response) and rated confidence using 1 of 3 buttons (ie, 3 = high, 2 = medium, and 1 = low). (2) Associative recognition—all 18 object pairs studied during relational encoding were randomly intermixed and presented with 18 rearranged object pairs consisting of items presented on different trials during relational encoding and not originally paired together (eg, left item on trial 5 paired with right item on trial 16). Subjects made 2-button yes/no responses indicating whether items in each pair had been presented “together.” To prevent additional encoding of relational object pairs during retrieval, item recognition preceded associative recognition.

Both retrieval tasks were self-paced, and subjects were instructed to, “Work as quickly and accurately as you can.” As an index of overall recognition accuracy, we subtracted the proportion of false alarms (ie, new items incorrectly accepted as old) from the proportion of hits (ie, old items correctly accepted as old) in each condition. F and R estimates were also calculated and used to identify group differences in retrieval processes on the item recognition task. These were calculated by using confidence ratings to plot ROC curves, which were then fitted to the dual-process signal detection model (DPSD)25to derive independent estimates of F and R.9,10 ROCs are generated by plotting hits against false alarms as a function of response confidence. The first point of the function reflects the strictest criterion such that only the most confidently recognized items (3) are treated as a hit or a false alarm. Each subsequent point reflects a less strict response criterion (eg, accepting both “2” and “3” as acceptable hits or false alarms). The shape of the ROC is used to infer the contribution of R and F to overall performance using a curve-fitting search algorithm.25 The method effectively measures R as the y-intercept and F as the degree of curvilinearity of the function. Subjects successfully completed practice tasks prior to beginning. Task duration was 15–20 minutes.

Results

Study 1

Reliability.

As described in the companion article,24 participants received each of the 3 forms of the RISE and the other CNTRACS tasks over 3 testing sessions, which occurred within approximately a 1-month period. During a given session, the version of the RISE was counterbalanced using a latin squares design, and participants never did more than one version of the task in a single session.

Alternate forms reliability was investigated in a subsample of 64 controls with complete data on all 3 forms. As seen in figure 2, recognition accuracy was generally equivalent, with small differences between forms. There were no form differences in the item recognition task following item-specific encoding (F2,62 < 1). However, in the relational encoding condition, there was an effect of test form on item recognition (F2,62 = 6.2, P < .005), with lower performance on form C than on either form A (F1,63 = 11.1, P < .005) or form B (F1,63 = 10.1, P < .005). The size of these differences was small (Cohen's d = 0.32 and 0.38, respectively). For the associative recognition task, there was also an effect of test form (F2,62 = 7.4, P < .005), with higher performance on form A than form B (F1,63 = 8.6, P < .005; Cohen's d = 0.33) or form C (F1,63 = 12.4, P < .001; Cohen's d = 0.23). Examination of alternate forms reliability coefficients (Pearson's r) confirmed that tasks had acceptable levels of agreement (r values from .61 to .79) for all possible combinations (table 2).

Fig. 2.

Fig. 2.

Mean (± SD) recognition accuracy (hit rate—false alarm rate) in healthy control subjects on the 3 test forms of the RISE paradigm. (A) Item recognition performance and (B) associative recognition performance.

Table 2.

Study 1 Alternative Forms Reliability Coefficients in Healthy Volunteers

Retrieval Task A_B A_C B_C
Item recognition 0.79 0.77 0.61
Associative recognition 0.65 0.69 0.63

Internal consistency was examined by calculating intraclass correlation coefficients (ICC) for test item responses in the same subsample of 64 healthy controls. As seen in table 3, there was acceptable internal consistency (ICCs from .61 to .91) for item and associative recognition across forms and encoding conditions. Given acceptable levels of alternative forms reliability and internal consistency, data were collapsed across forms for subsequent analyses.

Table 3.

Study 1 Internal Consistency in Healthy Volunteers

Test Version Item Recognition
Associative Recognition
Item Encoding Relational Encoding Relational Encoding
Version A 0.78 0.65 0.71
Version B 0.78 0.76 0.72
Version C 0.79 0.75 0.61

Difficulty.

Difficulty levels were examined to identify any floor or ceiling effects and confirm equivalent difficulty to facilitate investigation of differential deficits. Table 4 summarizes task performance in all conditions for the full sample of 73 controls and 93 patients. In controls, accuracy on the item recognition task was high (above 85%), but below ceiling (less than 100%), and was substantially better than on the associative recognition task. When these values were entered into an ANOVA, there was a main effect of test variable (F2,71 = 152, P < .0001). Follow-up analyses revealed that this was due to better item recognition in the item encoding than in the relational encoding condition (F1,72 = 24.0, P < .0001). Item recognition was also better than associative recognition for both item encoding (F1,71 = 309.9, P < .001) and relational encoding conditions (F1,71 = 273.6, P < .0001). Encoding condition differences in item recognition were small (Cohen's d = 0.24). However, associative recognition was substantially more difficult than item recognition (Cohen's d = 1.72), which was not surprising because it requires not only memory for each item but also for which items were paired together at study. These differences in task difficulty suggest caution in interpreting differential deficit, as the discriminating power of the tasks may not be equivalent27—particularly in comparisons of item and associative recognition. Therefore, item recognition and associative recognition were examined separately.

Table 4.

Study 1 Performance on Item and Associative Recognition Tasks

Task Controls (n = 73)
Patients (n = 93)
Group Difference
Mean SD Mean SD F Value P Value Effect Size
Item recognition: item encoding condition
    HR 0.90 0.08 0.79 0.12 52.2 <.0001 1.07
    Accuracy (HR − FAR) 0.87 0.11 0.71 0.18 41.2 <.0001 1.07
    Familiarity (F)a 1.40 1.06 1.14 0.86 2.9 .09 0.27
    Recollection (R)a 0.79 0.20 0.59 0.25 29.1 <.0001 0.88
Item recognition: relational encoding condition
    HR 0.88 0.08 0.73 0.13 66.3 <.0001 1.39
    Accuracy (HR − FAR) 0.84 0.11 0.66 0.18 54.5 <.0001 1.21
    Familiarity (F)a 1.61 0.81 1.06 0.74 19.4 <.0001 0.82
    Recollection (R)a 0.73 0.22 0.52 0.24 30.2 <.0001 0.91
Item recognition: item and relational encoding condition
    FAR 0.04 0.05 0.08 0.09 8.6 <.005 0.55
Associative recognition: relational encoding condition
    HR 0.71 0.12 0.63 0.14 14.6 <.001 0.61
    FAR 0.13 0.10 0.25 0.17 31.4 <.0001 0.86
    Accuracy (HR − FAR) 0.58 0.19 0.37 0.19 47.7 <.0001 1.10

Note:HR, high rate; FAR, false alarm rate.

a

Familiarity and Recollection scores reported for 69 controls and 90 patients.

Effect of Schizophrenia.

Encoding Responses

Although patients had a higher rate of nonresponses than controls across the 2 encoding tasks (t163 = −5.1, P < .0001), all participants were fully engaged, and the number of nonresponses across the 72 encoding trials remained low in both groups (mean ± SD: controls = 1.6 ± 1.3; patients = 3.8 ± 3.3).

Item Recognition

Figure 3A illustrates item recognition discriminability for the full sample of 73 controls and 93 patients. ANOVA revealed effects of group (F1,164 = 50.0, P < .0001), encoding condition (F1,164 = 58.3, P < .0001), and a group by condition interaction (F1,164 = 4.9, P < .05). As can be seen in table 4, the effect of schizophrenia on item recognition was larger following relational (Cohen's d = 1.21) than item-specific encoding (Cohen's d = 1.07).

Fig. 3.

Fig. 3.

Mean (± SE) recognition accuracy (hit rate—false alarm rate). (A) Item recognition in healthy controls (dark bars) and patients with schizophrenia (light bars) reveals a group by encoding condition interaction, with disproportionate patient impairments following relational vs item-specific encoding and (B) associative recognition reveals a main effect of group, with lower patient vs control performance.

Although controls were not at ceiling, their performance was at the upper range of recognition accuracy, raising the possibility that the group by condition interaction was influenced by a truncated range of control performance (ie, there may have been an even stronger item recognition impairment in patients with a more difficult task). To investigate this possibility, a quartile split was performed, and the highest performing quartile of patients and controls were eliminated, leaving a sample of 55 controls and 70 patients matched on age and parental education. Eliminating this top quartile moved controls further away from ceiling for item recognition following item-specific (0.85 ± 0.13 vs 0.87 ± 0.12) and relational encoding (0.82 ± 0.12 vs 0.84 ± 0.11). As in the full sample, accuracy in controls remained significantly higher in the item than relational encoding condition (F1,54 = 16.6, P < .0001; Cohen's d = 0.23). When the between-group analysis was repeated, significant main effects of group (F1,123 = 46.7, P < .0001) and encoding condition (F1,123 = 48.2, P < .0001) and a group by condition interaction (F1,123 = 6.2, P < .05) were observed. Again, effects of schizophrenia on item recognition were larger for relational than item-specific encoding.

Finally, because the full sample was not matched on gender and estimated premorbid intellectual ability (see table 1), the analysis was also repeated for all 93 patients and a subsample of 61 controls matched on age, gender distribution, parental education, and intellectual ability. This analysis produced a similar pattern, with main effects of group (F1,152 = 39.4, P < .0001) and encoding condition (F1,152 = 59.2, P < .0001). However, the group by condition interaction was reduced to a trend level effect (F1,152 = 2.9, P = .09). Thus, more severe patient deficits in item recognition following relational vs item-specific encoding did not appear secondary to a restricted range of control performance but were less prominent when group differences in premorbid intellectual ability and gender were taken into account.

Familiarity and Recollection

Confidence ratings were used to plot ROC curves and fit them with the DPSD model18 to obtain F and R estimates in the full sample of 93 patients and 73 controls. Examination of these parameter estimates revealed 3 patients and 4 controls with extreme F values (ie, > 2.5 SD beyond their respective group mean) because they did not distribute their responses. These participants were removed as outliers, leaving 90 patients and 69 controls for subsequent analyses. As can be seen in figure 4, the remaining sample utilized the full range of confidence ratings to distribute their responses. A small residual sum of squares error in patients (SSE = 0.0018) and controls (SSE = 0.0012) confirmed an excellent fit when the DPSD model was applied to these ROC data.

Fig. 4.

Fig. 4.

Distribution of confidence ratings (mean ± SE) for new and old items during the item recognition task in patients (red bars) and healthy controls (blue bars). Confidence ratings for previously studied targets (old) are represented by solid bars, and confidence ratings for never-studied foils (new) are represented by shaded bars. As can be seen, the largest proportion of responses are high confident responses during correct identification of targets as old (right most end of graph) and correct rejection of foils as new (left most end of graph).

The ANOVA of R and F revealed main effects of group (F1,157 = 23.7, P < .0001), and test parameter (F1,157 = 87.2, P < .0001). There were also 2-way interactions between encoding condition and group (F1,157 = 6.7, P < .05) and encoding condition and test parameter (F1,157 = 4.6, P < .05) and a 3-way interaction between encoding condition, test parameter, and group (F1,157 = 5.8, P < .05). Contrary to predictions, patients had deficits in recollection- and familiarity-based retrieval (table 4). Whereas recollection was impaired across conditions (F1,157 = 31.0, P < .0001), familiarity deficits were most prominent following relational vs item-specific encoding (F1,157 = 6.4, P < .05) as illustrated in figure 5.

Fig. 5.

Fig. 5.

Mean (± SE) familiarity and recollection during item recognition in healthy controls (dark bars) and patients with schizophrenia (light bars). (A) Familiarity (F) reveals a group by encoding condition interaction, with disproportionate patient impairments following relational vs item-specific encoding, (B) Recollection (R) reveals a main effect of group, with lower patient vs control performance.

Associative Recognition

Associative recognition discriminability for the full sample of 73 controls and 93 patients is illustrated in figure 3B and summarized in table 4. The ANOVA revealed a large effect (Cohen's d = 1.07) of schizophrenia, such that patients were significantly impaired (F1,164 = 95.4, P < .0001). The same pattern of results was observed in the comparison between patients and the demographically matched subsample of 61 controls (F1,152 = 37.1, P < .0001).

Summary.

Study 1's revisions of the original task4 appeared to optimize tolerability and comprehension, as all patients understood the task, and only 6% were excluded for below-chance performance. Psychometric analysis revealed that there were small effects of encoding condition on item recognition accuracy in controls, with better performance following item-specific than relational encoding. However, the difference in control performance on associative and item recognition tasks was large, with substantially better performance on the item recognition task. The 3 test forms also appeared generally well matched, with acceptable levels of internal consistency and alternate forms reliability, despite some small test form differences. Between-group analysis supported our hypothesis of a disproportionate patient deficit in item recognition deficit following relational vs item-specific encoding. Contrary to predictions, patients had both R and F retrieval deficits, with R impaired across conditions, and F showing the strongest impairments following relational encoding. There was also a large effect of schizophrenia on associative recognition, consistent with the patients' overall recollection deficit.

One concern raised in study 1 was that item recognition was quite high in the control group. Therefore, the aim of study 2 was to modify the RISE to increase the difficulty of the item recognition task while equating the difficulty of item-specific and relational encoding conditions.

Methods

Study 2

Humans have a large capacity for storing and retrieving visual object details,25 and we speculated that this may have driven the high item recognition in controls. We, therefore, replaced visual objects used in study 1 with word stimuli used in the original version of the task.4 Because words are less salient than objects, we reasoned that this change would reduce item recognition equally for the 2 encoding conditions.

Participants.

Multisite data were obtained on 64 controls and 49 patients. However, 2 controls and 13 patients were excluded for below-chance performance, leaving a final sample of 62 controls and 36 patients. Excluded patients had a larger proportion of males (89% vs 55%, P < .0001), lower daily skills (University of California San Diego Performance-Based Skills Assessment total = 68.4 ± 16.7 vs 82.5 ± 10.9, P < .005) and social acceptability (SLOF social acceptability = 4.1 ± 0.5 vs 4.5 ± 0.4, P < .05) but higher interpersonal skills (SLOF interpersonal skills = 4.9 ± 0.1 vs 4.5 ± 0.4, P < .05) than included patients. Diagnosis, assessment, and exclusion procedures were identical to study 1. Thirty-three patients were receiving antipsychotic medication (2 typical and 31 atypical), and 3 were not medicated. Groups were matched on age, gender, handedness, and intellectual ability (table 5). However, patients had lower education and parental education. After complete description of the study, written informed consent was obtained. The study was approved by the IRB at all participating research sites.

Table 5.

Study 2 Demographic Characteristics

Variable Controls (n = 62)
Patients (n = 36)
P Value
Mean SD Mean SD
Age (y) 36.4 11.8 40.4 11.0 ns
WTAR 39.4 7.3 36.2 11.6 ns
Education (y) 15.0 1.8 13.6 2.1 <.005
Parental education (y) 12.5 1.9 13.7 2.8 <.05
Gender (% male) 60 56 ns
Handedness (% right) 90 83 ns
BPRS—depressed –– –– 8.4 3.5 ––
BPRS—positive –– –– 9.1 4.4 ––
BPRS—disorganized –– –– 5.0 1.6 ––
BPRS—negative –– –– 7.0 2.5 ––
BPRS—manic –– –– 4.6 1.6 ––
UPSA-B –– –– 82.5 10.9 ––
SLFS_P –– –– 4.2 0.4 ––
SLFS_I –– –– 4.2 0.5 ––

Note:Abbreviations are explained in the first footnote to table 1.

Materials and Procedure.

The RISE2 was developed following identical procedures to study 1. The only change was that visual objects were replaced with words.

Results

Difficulty

Table 6 summarizes recognition accuracy in controls across the 3 retrieval conditions. A comparison of control performance on study 1 and study 2 confirmed that use of words lowered item recognition accuracy (F1,133 = 46.7, P < .0001). In experiment 2, healthy controls showed a small trend toward differential item recognition between the item-specific and relational encoding conditions (F1,61 = 3.1, P = .08), and performance was significantly lower on the associative than on the item recognition task (F1,61 = 38.3, P < .0001).

Table 6.

Study 2 Recognition Accuracy (HR − FAR)

Task Controls (n = 62)
Mean SD
Item recognition
    Item encoding 0.73 0.12
    Relational encoding 0.71 0.12
Associative recognition 0.61 0.19

Note:HR, high rate; FAR, false alarm rate.

Reliability and Effects of Schizophrenia

Briefly, internal consistency and alternative forms reliability were somewhat lower than in study 1 but remained within an acceptable range (ie, above 0.60 in most cases). Examination of group differences in item recognition replicated the study 1 finding of a greater familiarity deficit in patients following relational vs item-specific encoding but did not find a group by condition interaction for recognition accuracy. As previously, recollection was impaired across conditions, and there was a large effect of schizophrenia on associative recognition. Detailed results are provided in online supplementary materials.

Summary

Study 2 attempted to lower item recognition in controls by replacing visual objects with words. This approach was successful as controls had lower item recognition in study 2 than in study 1. Difficulty was also more closely matched between item-specific and relational encoding conditions, although associative recognition remained more difficult than item recognition. Unexpectedly, patients had greater difficulty understanding task instructions, with a 2-fold increase in subject exclusion for below-chance performance in study 2 (13%) vs study 1 (6%). Given this increased data loss and the potential difficulty of adapting a verbal task for future non-English speaking clinical trials, the decision was made to proceed with the original visual object version of the RISE for ongoing development.

Discussion

In the present study, we translated a paradigm designed to examine memory and brain function in young healthy individuals4 into a clinical measure to be used as a biomarker to facilitate discovery of memory-enhancing agents and identify neural mechanisms of specific encoding and retrieval deficits in schizophrenia. In healthy controls, the RISE had an absence of ceiling or floor effects, acceptable internal consistency, and good reliability between 3 alternative forms. Item recognition accuracy in healthy participants was somewhat higher following item-specific encoding than following relational encoding. Between-group comparisons revealed that the magnitude of episodic memory deficits in schizophrenia depended upon how information was studied, and what processes were required for retrieval. As in a previous levels-of-processing study,15 memory appeared less impaired when patients were required to focus on item features to make a semantic decision during encoding and then use the familiarity strength of those feature representations to recognize items as old or new. However, familiarity and recognition accuracy on that same item recognition task was severely impaired when patients were required to process relationships between items during encoding instead of focusing on item features. Reduced R estimates and severe associative recognition impairments also suggested that patients were experiencing prominent recollection deficits regardless of whether they had performed item-specific or relational encoding.

These behavioral results are pertinent to an ongoing debate about the nature of memory impairment in schizophrenia. Several previous studies have indicated that schizophrenia patients have a specific impairment on memory tasks requiring conscious recollection and/or associative recognition and intact familiarity-based item recognition. For instance, studies using subjective report methods2123 indicate that patients might have specific recollection deficits and unimpaired or increased reliance on familiarity-based retrieval. Studies using associative and transitive inference paradigms12,28 and eye-movement memory measures29,30 have, likewise, provided evidence that relational memory may be specifically impaired, but item memory may be relatively spared. However, in a meta-analysis of associative and item recognition studies in schizophrenia,31 investigators found medium-sized effects of schizophrenia on item and associative recognition that were of similar magnitude (ie, d = 0.48 and d = 0.40, respectively). Several source retrieval studies have also cast doubt upon the specific recollection deficit hypothesis,32 and a recent patient study that carefully manipulated retrieval processes found evidence of familiarity and recollection deficits.33

Present results indicate that, although conscious recollection and associative recognition are severely impaired in schizophrenia, familiarity is also significantly impaired depending on the way in which patients are oriented to encode information. When instructed to semantically encode item-specific information, patients showed a relatively small familiarity deficit (d = 0.27). However, when instructed to actively process relationships among pairs of items, patients showed a similar magnitude deficit in familiarity and recollection (d = 0.82 and d = 0.91, respectively). This familiarity deficit occurred because patients, unlike controls, failed to benefit from the relational vs item-specific encoding condition. This beneficial encoding effect in controls may appear counterintuitive as relational processing facilitates encoding of associations between items rather than encoding of specific item features.6 However, creating item associations also activates lexical network representations, which could also increase the semantic strength of individual items, thereby facilitating familiarity-based retrieval (see ref. 34). The miss-match in stimulus presentation between relational encoding and item recognition conditions (ie, stimulus pairs vs individual items) may have also reduced the likelihood that these associations would be reactivated and used to support recollection at the time of retrieval.

In a previous functional Magnetic Resonance Imaging (fMRI) study of the original version of this task, Murray and Ranganath4 found that activity in the dorsolateral prefrontal cortex (DLPFC) was higher during relational than item-specific encoding and that DLPFC activity specifically predicted successful memory for associations among items. This finding led the authors to conclude that the DLPFC may contribute to episodic memory encoding through its processing of relationships among items that are active in working memory. Thus, impaired DLPFC function during memory encoding might account for the current pattern of disproportionate item recognition deficits in patients following relational encoding. Further support for this hypothesis is provided by a recent meta-analysis of fMRI studies of episodic memory in schizophrenia35 that found consistent DLPFC deficits in patients during episodic encoding. It is also notable that studies using a variety of process estimation methods, such as R/K, ROC, and source recognition paradigms, have indicated that PFC damage can impair both recollection and familiarity.3639 In addition, functional imaging studies suggest that the PFC contributes to both retrieval processes.40

In addition to these potential frontal lobe mechanisms, there is strong evidence from lesion studies of animals and studies of human amnesics17,4143 that hippocampal dysfunction is specifically associated with impaired recollection. Thus, the current pattern of generalized recollection deficits and associative recognition impairments may also be reflecting hippocampal dysfunction in schizophrenia. However, these same lesion studies, and functional imaging studies in healthy participants (for reviews, see ref. 4446), show that familiarity deficits are unrelated to hippocampal function, suggesting that current results cannot be explained by a focal hippocampal deficit. Other candidate regions include the perirhinal cortex,47 ventrolateral PFC,48 and/or DLPFC.47,49 Implementation of the RISE in an fMRI paradigm will allow us to directly assess these possibilities.

Although neural substrates need to be clarified, present results suggest that the RISE paradigm has sufficient reliability and sensitivity to detect memory deficits in schizophrenia. The ability to compare different encoding conditions and dissociate metrics of familiarity and recollection with a single task are unique strengths of this paradigm that distinguishes it from most current neuropsychological measures. The inclusion of an associative recognition task provides an added measure that is highly sensitive to memory impairment in schizophrenia.14

However, there are some limitations and areas for future task development worth noting. One limitation of the RISE paradigm is that individuals are very good at recognizing visual objects, which created potential ceiling effects for the item recognition task. Although controls were not at ceiling, and prominent group and group by condition differences remained when lower performing participants were examined, the discriminating power of the task might be increased by reducing control performance through further modifications such as addition of a retention delay or decrease in stimulus presentation times. This is being investigated in ongoing research. A second limitation is that difficulty levels of different task conditions were not perfectly matched, reducing confidence in strong claims of differential deficits because of potential differences in discriminating power.27 This was particularly true for the associative recognition task, which was substantially harder than item recognition. We, therefore, do not recommend use of the RISE as a sole measure of differential deficits in associative vs item recognition and, instead, suggest that the RISE associative recognition task be used as a supplementary measure, particularly in high-performing populations in which item recognition might be close to ceiling. Inclusion of the RISE in a larger battery of item and associative recognition tasks may also enable use of the process-dissociation procedure50 to establish a differential deficit despite differences in discriminating power. It is unlikely, however, that differences in discriminating power can explain patients' disproportionate item recognition deficit following relational encoding. In healthy controls, relational encoding led to lower item recognition accuracy but higher item familiarity, as compared with item-specific encoding. Nonetheless, patients showed disproportionate deficits following relational encoding on both item recognition accuracy and item familiarity measures, indicating that the group by condition interactions for both of these measures could not simply be due to the higher discriminating power of one condition vs the other. However, if these same differences in task difficulty are observed in ongoing research, item analysis procedures will be used to adjust test items to better match control performance between the 2 encoding conditions. Item analysis procedures may also be used in future task development to adjust alternate forms if the small differences in control performance between forms observed in the current study are replicated in subsequent research.

In conclusion, this study indicates that the RISE is a valid measure of item-specific and relational encoding and retrieval, with acceptable internal consistency and alternate forms reliability that can facilitate longitudinal treatment studies. Results suggest that episodic memory dysfunction in schizophrenia is driven by specific deficits in encoding and retrieval. Impairments are most prominent when patients must form relational representations during encoding and recollect these relationships during retrieval. However, patients' item-specific processing and familiarity-based retrieval is also affected to a lesser extent, suggesting that both aspects of episodic memory are viable targets for treatment development. One possible strategy may be to develop cognitive training interventions to further strengthen patients' ability to encode item details and use familiarity to aid retrieval, while developing pharmacological agents and remediation procedures to improve relational encoding processes and recollection-based retrieval.

Funding

National Institutes of Health (NIH) (5R01MH084840-03 to D.M.B., 5R01MH084826-03 to C.S.C., 5R01MH084828-03 to S.M.S., 5R01MH084821-03 to J.M.G., 5R01MH084861-03 to A.W.M.). D.M.B. has received grants from the National Institute of Mental health (NIMH), National Institute on Aging, National Alliance for Research on Schizophrenia and Depression (NARSAD), Allon, Novartis, and the McDonnell Center for Systems Neuroscience. C.S.C. has received research grants from the NIMH, National Institute on Drug Abuse, the Robert Wood Johnson Foundation and from Glaxo Smith Kline and has been an external consultant for Roche, Servier, Lilly, Merck, and Pfizer. J.M.G. has received grants from NIMH, receives royalty payments from the Brief Cognitive Assessment for Schizophrenia, and has consulted with Pfizer, Merck, Astra Zenaca, Solvay, and Glaxo Smith Kline. A.W.M. and J.D.R. have received research grants from the NIH and NARSAD. C.R. has received research grants from the NIH and has been an external consultant for Helicon Pharmaceuticals. S.M.S. has received research grants from NIMH, Pfizer, and AstraZeneca. A.P.Y. has received research grants from the NIH.

Supplementary Material

Supplementary Data

Acknowledgments

We thank the staff at each of the CNTRACS sites for their hard work and our participants for their time, energy, and cooperation. We also thank Sam Lockhart for assistance with piloting early versions of the memory task. B.H. and M.E.S. have no currently active grant or contract support from private or public sources. The authors have declared that there are no conflicts of interest in relation to the subject of this study.

References

  • 1.Tulving E. Episodic memory: from mind to brain. Annu Rev Psychol. 2002;53:1–25. doi: 10.1146/annurev.psych.53.100901.135114. [DOI] [PubMed] [Google Scholar]
  • 2.Ranganath C, Minzenberg MJ, Ragland JD. The cognitive neuroscience of memory function and dysfunction in schizophrenia. Biol Psychiatry. 2008;64:18–25. doi: 10.1016/j.biopsych.2008.04.011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Aleman A, Hijman R, de Haan EHF, Kahn RS. Memory impairment in schizophrenia: a meta-analysis. Am J Psychiatry. 1999;156:1358–1366. doi: 10.1176/ajp.156.9.1358. [DOI] [PubMed] [Google Scholar]
  • 4.Murray LJ, Ranganath C. The dorsolateral prefrontal cortex contributes to successful relational memory encoding. J Neurosci. 2007;27:5515–5522. doi: 10.1523/JNEUROSCI.0406-07.2007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Hunt RR, Einstein GO. Relational and item-specific information in memory. J Verbal Learn Verbal Behav. 1981;20:497–514. [Google Scholar]
  • 6.Bower GH. Imagery as a relational organizer in associative learning. J Verbal Learn Verbal Behav. 1970;9:529–533. [Google Scholar]
  • 7.Blumenfeld RS, Ranganath C. Dorsolateral prefrontal cortex promotes long-term memory formation through its role in working memory organization. J Neurosci. 2006;26:916–925. doi: 10.1523/JNEUROSCI.2353-05.2006. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Hunt RR, Mcdaniel MA. The enigma of organization and distinctiveness. J Mem Lang. 1993;32:421–445. [Google Scholar]
  • 9.Bower GH. Organizational factors in memory. Cognit Psychol. 1970;1:18–46. [Google Scholar]
  • 10.Blumenfeld RS, Parks CM, Yonelinas AP, Ranganath C. Putting the pieces together: the role of dorsolateral prefrontal cortex in relational memory encoding. J Cogn Neurosci. 2011;23:257–265. doi: 10.1162/jocn.2010.21459. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Iddon JL, McKenna PJ, Sahakian BJ, Robbins TW. Impaired generation and use of strategy in schizophrenia: evidence from visuospatial and verbal tasks. Psychol Med. 1998;28:1049–1062. doi: 10.1017/s0033291798006758. [DOI] [PubMed] [Google Scholar]
  • 12.Titone D, Ditman T, Holzman PS, Eichenbaum H, Levy DL. Transitive inference in schizophrenia: impairments in relational memory organization. Schizophr Res. 2004;68:235–247. doi: 10.1016/S0920-9964(03)00152-X. [DOI] [PubMed] [Google Scholar]
  • 13.Hanlon FM, Weisend MP, Yeo RA, et al. A specific test of hippocampal deficit in schizophrenia. Behav Neurosci. 2005;119:863–875. doi: 10.1037/0735-7044.119.4.863. [DOI] [PubMed] [Google Scholar]
  • 14.Lepage M, Montoya A, Pelletier M, Achim AM, Menear M, Lai S. Associative memory encoding and recognition in schizophrenia: an event-related fMRI study. Biol Psychiatry. 2006;60:1215–1223. doi: 10.1016/j.biopsych.2006.03.043. [DOI] [PubMed] [Google Scholar]
  • 15.Ragland JD, Moelter ST, McGrath C, et al. Levels-of-processing effect on word recognition in schizophrenia. Biol Psychiatry. 2003;54:1154–1161. doi: 10.1016/s0006-3223(03)00235-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Mandler G. Recognizing: the judgment of previous occurrence. Psychol Rev. 1980;87:252–271. [Google Scholar]
  • 17.Yonelinas AP. The nature of recollection and familiarity: a review of 30 years of research. J Mem Lang. 2002;46:441–517. [Google Scholar]
  • 18.Yonelinas AP. Receiver-operating characteristics in recognition memory–evidence for a dual-process model. J Exp Psychol Learn Mem Cogn. 1994;20:1341–1354. doi: 10.1037//0278-7393.20.6.1341. [DOI] [PubMed] [Google Scholar]
  • 19.Yonelinas AP. Components of Episodic Memory: The Contribution of Recollection and Familiarity. In: Episodic Memory: New Directions in Research. London, UK: Oxford University Press; 2002. pp. 31–52. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Tulving E. Elements of Episodic Memory. New York, NY: Oxford University Press USA; 1985. [Google Scholar]
  • 21.Danion JM, Kazes M, Huron C, Karchouni N. Do patients with schizophrenia consciously recollect emotional events better than neutral events? Am J Psychiatry. 2003;160:1879–1881. doi: 10.1176/appi.ajp.160.10.1879. [DOI] [PubMed] [Google Scholar]
  • 22.Thoma P, Zoppelt D, Wiebel B, Daum I. Recollection and familiarity in negative schizophrenia. Neuropsychologia. 2006;44:430–435. doi: 10.1016/j.neuropsychologia.2005.05.017. [DOI] [PubMed] [Google Scholar]
  • 23.van Erp TG, Lesh TA, Knowlton BJ, et al. Remember and know judgments during recognition in chronic schizophrenia. Schizophr Res. 2008;100:181–190. doi: 10.1016/j.schres.2007.09.021. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Henderson D, Poppe AB, Barch DM, et al. Optimization of a goal maintenance task for use in clinical applications. Schizophr Bull. 2012;38:104–113. doi: 10.1093/schbul/sbr172. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Brady TF, Konkle T, Alvarez GA, Oliva A. Visual long-term memory has a massive storage capacity for object details. Proc Natl Acad Sci U S A. 2008;105:14325–14329. doi: 10.1073/pnas.0803390105. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Ravizza SM, Moua KC, Long D, Carter CS. The impact of context processing deficits on task-switching performance in schizophrenia. Schizophr Res. 2010;116:274–279. doi: 10.1016/j.schres.2009.08.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Chapman LJ, Chapman JP. The measurement of differential deficit. J Psychiatr Res. 1978;14:303–311. doi: 10.1016/0022-3956(78)90034-1. [DOI] [PubMed] [Google Scholar]
  • 28.Armstrong K, Kose S, Williams L, Woolard A, Heckers S. Impaired associative inference in patients with schizophrenia. Schizophr Bull. 2010;37:1–8. doi: 10.1093/schbul/sbq145. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Hannula DE, Ranganath C, Ramsay IS, et al. Use of eye movement monitoring to examine item and relational memory in schizophrenia. Biol Psychiatry. 2010;68:610–616. doi: 10.1016/j.biopsych.2010.06.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Williams LE, Must A, Avery S, et al. Eye-movement behavior reveals relational memory impairment in schizophrenia. Biol Psychiatry. 2010;68:617–624. doi: 10.1016/j.biopsych.2010.05.035. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Achim AM, Lepage M. Is associative recognition more impaired than item recognition memory in schizophrenia? A meta-analysis. Brain Cogn. 2003;53:121–124. doi: 10.1016/s0278-2626(03)00092-7. [DOI] [PubMed] [Google Scholar]
  • 32.Weiss AP, Goff DC, Duff M, Roffman JL, Schacter DL. Distinguishing familiarity-based from source-based memory performance in patients with schizophrenia. Schizophr Res. 2007;99:208–217. doi: 10.1016/j.schres.2007.05.028. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Lefebvre AA, Cellard C, Tremblay S, et al. Familiarity and recollection processes in patients with recent-onset schizophrenia and their unaffected parents. Psychiatry Res. 2010;175:15–21. doi: 10.1016/j.psychres.2009.01.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Wang WC, Lazzara MM, Ranganath C, Knight RT, Yonelinas AP. The medial temporal lobe supports conceptual implicit memory. Neuron. 2010;68:835–842. doi: 10.1016/j.neuron.2010.11.009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Ragland JD, Laird AR, Ranganath C, Blumenfeld RS, Gonzales SM, Glahn DC. Prefrontal activation deficits during episodic memory in schizophrenia. Am J Psychiatry. 2009;166:863–874. doi: 10.1176/appi.ajp.2009.08091307. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Duarte A, Ranganath C, Knight RT. Effects of unilateral prefrontal lesions on familiarity, recollection, and source memory. J Neurosci. 2005;25:8333–8337. doi: 10.1523/JNEUROSCI.1392-05.2005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Farovik A, Dupont LM, Arce M, Eichenbaum H. Medial prefrontal cortex supports recollection, but not familiarity, in the rat. J Neurosci. 2008;28:13428–13434. doi: 10.1523/JNEUROSCI.3662-08.2008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.MacPherson SE, Bozzali M, Cipolotti L, Dolan RJ, Rees JH, Shallice T. Effect of frontal lobe lesions on the recollection and familiarity components of recognition memory. Neuropsychologia. 2008;46:3124–3132. doi: 10.1016/j.neuropsychologia.2008.07.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.Kishiyama MM, Yonelinas AP, Knight RT. Novelty enhancements in memory are dependent on lateral prefrontal cortex. J Neurosci. 2009;29:8114–8118. doi: 10.1523/JNEUROSCI.5507-08.2009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Blumenfeld RS, Ranganath C. Prefrontal cortex and long-term memory encoding: an integrative review of findings from neuropsychology and neuroimaging. Neuroscientist. 2007;13:280–291. doi: 10.1177/1073858407299290. [DOI] [PubMed] [Google Scholar]
  • 41.Fortin NJ, Wright SP, Eichenbaum H. Recollection-like memory retrieval in rats is dependent on the hippocampus. Nature. 2004;431:188–191. doi: 10.1038/nature02853. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Aggleton JP, Vann SD, Denby C, et al. Sparing of the familiarity component of recognition memory in a patient with hippocampal pathology. Neuropsychologia. 2005;43:1810–1823. doi: 10.1016/j.neuropsychologia.2005.01.019. [DOI] [PubMed] [Google Scholar]
  • 43.Turriziani P, Oliveri M, Salerno S, et al. Recognition memory and prefrontal cortex: dissociating recollection and familiarity processes using rTMS. Behav Neurol. 2008;19:23–27. doi: 10.1155/2008/568057. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44.Eichenbaum H, Yonelinas AP, Ranganath C. The medial temporal lobe and recognition memory. Annu Rev Neurosci. 2007;30:123–152. doi: 10.1146/annurev.neuro.30.051606.094328. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Skinner EI, Fernandes MA. Neural correlates of recollection and familiarity: a review of neuroimaging and patient data. Neuropsychologia. 2007;45:2163–2179. doi: 10.1016/j.neuropsychologia.2007.03.007. [DOI] [PubMed] [Google Scholar]
  • 46.Diana RA, Yonelinas AP, Ranganath C. Imaging recollection and familiarity in the medial temporal lobe: a three-component model. Trends Cogn Sci. 2007;11:379–386. doi: 10.1016/j.tics.2007.08.001. [DOI] [PubMed] [Google Scholar]
  • 47.Ranganath C. A unified framework for the functional organization of the medial temporal lobes and the phenomenology of episodic memory. Hippocampus. 2010;20:1263–1290. doi: 10.1002/hipo.20852. [DOI] [PubMed] [Google Scholar]
  • 48.Raposo A, Han S, Dobbins IG. Ventrolateral prefrontal cortex and self-initiated semantic elaboration during memory retrieval. Neuropsychologia. 2009;47:2261–2271. doi: 10.1016/j.neuropsychologia.2008.10.024. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49.Turriziani P, Smirni D, Oliveri M, Semenza C, Cipolotti L. The role of the prefrontal cortex in familiarity and recollection processes during verbal and non-verbal recognition memory: an rTMS study. Neuroimage. 2010;52:348–357. doi: 10.1016/j.neuroimage.2010.04.007. [DOI] [PubMed] [Google Scholar]
  • 50.Knight RA, Silverstein SM. A process-oriented approach for averting confounds resulting from general performance deficiencies in schizophrenia. J Abnorm Psychol. 2001;110:15–30. doi: 10.1037//0021-843x.110.1.15. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary Data

Articles from Schizophrenia Bulletin are provided here courtesy of Oxford University Press

RESOURCES