Where Have I Heard That Before? A Validity Study of Source Memory Indices from the California Verbal Learning Test – Second Edition

Abstract

Objective:

We investigated the construct and criterion validity of the source memory (SM) indices within the California Verbal Learning Test-II (CVLT-II).

Method:

Participants included 77 individuals with HIV-associated neurocognitive disorders (HAND+), 287 HIV+ neurocognitively normal individuals (HAND−), and 203 seronegative HIV comparisons (HIV−). CVLT-II SM impairment status (normative scores ≤ 1 standard deviation) was determined using Total Across-List Intrusions and Source Recognition Discriminability (d’). Participants also completed a comprehensive neuropsychological battery, assessments of everyday functioning, and experimental measures of SM.

Results:

CVLT-II SM impairment was significantly associated with increased errors on experimental SM measures and lower scores on measures of passage recall and executive functions, but not visuospatial skills. In a logistic regression controlling for clinicodemographic factors, CVLT-II SM impairment was a significant independent predictor of HAND, with the HAND+ group showing higher rates of SM impairment than both the HAND− and HIV− groups. Finally, CVLT-II SM impairment was significantly related to a composite measure of everyday functioning, but this effect disappeared after adjusting for covariates. Note that, the overall pattern of findings across this study also held when CVLT-3 normative standards were applied to the SM indices.

Conclusions:

Results provide initial support for the construct and criterion validity of a CVLT-II SM index in the setting of HIV disease. Future studies should examine the validity of CVLT-II SM variables in other neuropsychological populations.

The California Verbal Learning Test – Second Edition (CVLT-II; Delis, Kramer, Kaplan, & Ober, 2000) is a well-validated measure of word list learning and memory that is reliably among the most common tests used by clinical neuropsychologists in North America (Rabin, Paolillo & Barr, 2016). In healthy samples, the CVLT-II has good test-retest reliability (Woods et al., 2006) and internal consistency (Delis et al., 2000). The CVLT-II is sensitive to neural injury across a variety of neurological (Alexander, Stuss, & Fansabedian, 2003; Baldo, Delis, Kramer, & Shimamura, 2002), medical (Stegen et al., 2010; Delis, Wetter, Jacobson, Peavy et al., 2005), and psychiatric populations (Stone et al., 2011, 2015). Of further clinical relevance, the CVLT-II shows evidence of ecological validity and can aid in risk assessment for everyday functioning problems (e.g., medication mismanagement; Heinrichs et al., 2008).

The CVLT-II is an especially valuable tool for clinical neuropsychologists, as its multidimensional design informed by cognitive psychology produces dozens of supplementary indices that provide valuable information about the many different cognitive processes by which examinees learn, recall, and recognize words. The primary indices from the CVLT-II (e.g., Total Trials 1–5, Delayed Free Recall, and Recognition Discriminability d’) are generally more reliable than quantitative process measures (e.g., Woods et al., 2006), which can suffer from low base rates and Type I error risk. However, these quantitative process indices enable clinicians to draw inferences about component cognitive mechanisms (Donders, 2008) that can be useful in differentiating neuropsychological disorders (e.g., Fine et al., 2008), identifying risks for everyday functioning problems and health behaviors (e.g., Obermeit et al., 2015), and making targeted recommendations for remediation (e.g., Jacobs & Donders, 2008).

To that end, the CVLT-II produces qualitative process indices measuring source memory (SM) ability, which is an aspect of declarative memory that describes one’s ability to recall the characteristics, conditions, or context related to a particular episodic memory. SM is distinct from item memory, which refers to the specific event or detail that is to be remembered (Glisky, Polster, & Routhieaux, 1995). For example, one might accurately recall the details of a recipe (i.e., item memory), but at the same time forget whether they had learned the recipe from a grandparent, a friend, or the Internet (i.e., SM). The source monitoring framework proposed by Johnson (1993) posits that there are several factors that contribute to the ease and accuracy of identifying a source including: a) the type and amount of the characteristics of the particular episodic memory (e.g., perceptual, contextual, affective, semantic, and cognitive detail), b) the distinctiveness of the characteristics from source to source, and c) the efficacy of the decision process used to evaluate the plausibility and consistency of the source (see Mitchell & Johnson, 2009 for review). Not surprisingly then, studies using lesion samples (Baldo et al., 2002), neuropsychological data (Craik et al., 1990), and various neuroimaging approaches (Dobbins et al., 2004) suggest that SM is dependent on the integrity of prefrontal networks, particularly the dorsolateral prefrontal cortex, as well as executive functions including concept shifting, divided attention, and higher-order encoding strategies (e.g., Glisky et al., 1995).

Although the experimental cognitive psychology literature is replete with experimental SM paradigms, our literature review did not reveal any current “gold standard” clinical measure of SM. Among the top 15 memory tests used by clinical neuropsychologists (Rabin et al., 2016), only the CVLT-II formally includes standardized measures of SM. On the CVLT-II, examinees use SM to distinguish items from the primary (List A) and interference (List B) word lists, each of which contain 16 words belonging to four semantic categories (two categories are shared across List A and List B). In the first five learning trials, examinees are presented with only List A and are asked after each trial to recall as many words as they can from that list. Examinees are then presented with List B, which is comprised of novel words and asked to recall as many words from the new list as possible (while inhibiting recall from List A). On the subsequent free- and cued- short and long delayed recall trials, examinees are asked to recall words only from List A (while inhibiting recall from List B). Finally, the CVLT-II includes a 48-item yes/no recognition trial in which examinees are presented with the 16 target words from List A interspersed with the 16 interference words from List B, and 16 novel lure words.

The CVLT-II includes two complementary, standardized ways to measure SM (Delis et al., 2000). The first is Total Across-List Intrusions, which reflects the number of times that an examinee erroneously generates a word: 1) from List A during the List B trial, or 2) from List B during any of the delayed recall trials of List A. Total Across-List Intrusions are measured as a raw score for which age-referenced percentiles are provided. The second CVLT-II SM measure is Source Recognition Discriminability (d’), which indexes an examinee’s ability to discriminate between List A target items and List B interference items on the yes/no recognition trial. Source Recognition Discriminability (d’) is equivalent to a contrast z-score in that it measures the absolute difference in standard deviation units between the examinee’s hit rate (List A words) and source error false-positive rate (List B words). The Source Recognition Discriminability (d′) raw score on the CVLT-II can range from a high of +4.0 (16 hits, 0 List B false positives) to a low of −4.0 (0 hits, 16 List B false positives), while a d’ score of 0 indicates that an examinee’s hit rate and List B false-positive rate were both at 50% accuracy. The CVLT-II expanded report provides a raw score for Source Recognition Discriminability (d’) along with a sex and age-adjusted normative z-score.

Previous literature suggests that the CVLT SM variables, particularly Source Recognition Discriminability (d’), are able to discriminate between populations with known frontal systems dysfunction and those with memory problems. Specifically, impairment on Source Recognition Discriminability (d’) has been demonstrated in individuals with frontal lobe lesions (Baldo et al., 2002), Huntington’s disease (Fine et al., 2008), and Mild Cognitive Impairment (Clark et al., 2012). Evidence from structural MRI also supports an association between Source Recognition Discriminability (d’) and frontal systems integrity (Butler et al., 2012). The CVLT Total Across-List Intrusions measure has received comparatively less attention as a SM index in the literature. However, higher rates of List B intrusions have been reported during free recall of List A in ischemic vascular dementia (Davis et al., 2002) and autism (Bennetto et al., 1996). Taken together, the limited extant literature suggests that the CVLT SM indices may be sensitive to frontotemporal dysfunction, but we know little about most other aspects of validity.

Validity is typically a matter of degree rather than an all-or-none property; as such the assessment of validity should be viewed as a process involving the accumulation of empirical evidence and theoretical rationales to support the adequacy and appropriateness of proposed score interpretations (AERA, APA, & NCME, 2014). With the understanding that “validity” is not a static, unidimensional property of a test, we undertook a multidimensional, comprehensive, and clinically relevant investigation of CVLT-II SM indices. We adopted Crocker and Algina’s test validity framework (2006), focusing specifically on construct and criterion validity. Construct validity (i.e., the degree to which a test measures what it claims to be measuring) can be assessed by examining: (1) convergent validity, when tests measuring the same construct are related to one another (e.g., the relationship of CVLT-II SM indices to existing measures of SM and related constructs, such as item memory and executive functions); (2) incremental validity, when the test is associated with the construct of interest above an existing method of assessment (i.e., independence of the CVLT-II SM indices relative to the CVLT-II primary memory indices in relating to measures of SM and retrospective episodic memory); (3) discriminant validity, when tests measuring different constructs are not related to one another (e.g., the relationship of the CVLT-II SM indices to a measure of an unrelated construct, such as visuospatial ability); and (4) diagnostic validity, when the measure differentiates between groups that would be expected to differ on the construct (e.g., the ability of the CVLT-II SM index to differentiate between groups that are susceptible to SM issues). For criterion validity (i.e., the extent to which items adequately represent a performance domain that cannot be directly measured by the test), we focused specifically on ecological validity, which is when the measure relates to a criterion relevant to real-world behaviors, (e.g., the CVLT-II SM index in relation to everyday functioning outcomes).

The present study examined the validity of CVLT-II SM indices in the setting of human immunodeficiency virus (HIV) disease. Frontal systems dysfunction is common in HIV disease (Du Plessis et al., 2014) and individuals with HIV-associated neurocognitive disorders (HAND) commonly show mild-to-moderate executive dysfunction (Walker & Brown, 2018) and deficits in the strategic aspects of encoding and retrieval from episodic memory (e.g., Carey et al., 2006). For example, HIV disease is associated with limited use of higher-order encoding strategies, such as semantic clustering during list learning (e.g., Cattie et al., 2012; Gongvatana et al., 2007). HIV is also associated with deficits in SM. Morgan et al. (2009) reported that HIV+ individuals were moderately impaired on both verbal and visual SM tasks relative to demographically similar seronegative participants (Cohen’s d = .44 to .50). Additionally, among the HIV+ group, SM performance was related to cognitive flexibility (e.g., Trail Making Test B) and semantic clustering on the CVLT-II, but not to semantic knowledge (e.g., Boston Naming Test), thus providing evidence of both convergent and discriminant validity for the association between executive functions and SM. Morgan et al. (2009) also reported that an experimental measure of visual SM was associated at a trend level with raw CVLT-II Total Across-List Intrusions in 60 HIV+ persons with a small effect size (Spearman’s rho = 0.22). However, this small study did not examine the CVLT-II Source Recognition Discriminability (d’) index.

With this literature in mind, we evaluated the construct and criterion validity of the CVLT-II SM indices. In evaluating the construct validity, we hypothesized that the CVLT-II SM indices would: (1) be associated with experimental measures of SM, and measures of the related constructs of retrospective episodic memory and executive functions (convergent validity); (2) be related to SM and retrospective episodic memory deficits even when controlling for primary CVLT-II primary memory measures (incremental validity); (3) not be associated with a measure of visuospatial ability (discriminant validity); and (4) differentiate seronegative comparison participants from groups known to be sensitive to SM impairment (i.e., HIV and HAND; diagnostic validity). In investigating the criterion validity, we predicted that impairment on CVLT-II SM indices would be associated with dependence in activities of daily living (ADLs) (ecological validity). In addition to providing the first known systematic analysis of the validity of the CVLT-II SM indices, our HIV sample also allowed us to contribute to the cognitive neuropsychology literature on HAND, for which episodic memory impairment is among the most commonly affected domains (Carey et al., 2004) and is sensitive to everyday functioning (e.g., Obermeit et al., 2015).

Methods

Participants

The institutional research review board approved the study protocol. The sample included 567 participants recruited from local HIV clinics, community-based organizations, and the general community. Enzyme-linked immunosorbent assay (ELISA) tests or MedMira Rapid Tests were used to determine HIV status. Study exclusions included neuromedical conditions (e.g., seizure disorders, head injury, and stroke), severe psychiatric disorders (e.g., psychosis), an estimated verbal IQ < 70 based on the Wechsler Test of Adult Reading (WTAR; Wechsler, 2001), current substance use disorder (SUD), and positive screening on a Breathalyzer or urine toxicology test for alcohol or illicit drugs on the day of testing.

Participants were classified into three groups: HAND+ (n = 77), HAND− (n = 287), and HIV− (n = 203). The demographic and clinical characteristics of the study groups are shown in Table 1. The three study groups were broadly comparable on age and ethnicity (ps >.10). However, there was a higher frequency of men and diagnoses of lifetime major depressive disorder (MDD), generalized anxiety disorder (GAD), and SUD in the two HIV+ groups (ps < .05). The HIV− group reported more years of formal education, had higher estimated verbal IQ scores, and fewer current mood symptoms (ps < .05). The HIV+ groups did not differ significantly on any disease or treatment variables (ps > .10).

Table 1.

Demographic and clinical characteristics of the samples

Variable	HIV− n = 203	HAND−n = 287	HAND+n = 77	p
Sociodemographics
Age (years) a+	42.8 (14.8)	45.7 (11.1)	45.5 (13.1)	.14
Education (years) a+^	14.2 (2.7)	13.7 (2.6)	13.0 (2.6)	.001
Sex (% women) +^	35.0	12.9	19.5	<.001
Ethnicity (%)				.26
Caucasian	55.7	60.9	49.4
Hispanic	17.7	15.2	19.5
Black	23.2	21.1	31.2
Asian	2.0	2.1	0.0
Other	1.5	0.1	0.0
Estimated Verbal IQ a#^	103.6 (10.9)	103.8 (11.2)	96.5 (11.8)	<.001
Psychiatric
POMS Total (of 200) a+#^	42.7 (27.1)	55.1 (34.4)	68.8 (42.6)	<.001
Lifetime GAD (%) +^	4.4	12.9	18.7	<.001
Lifetime MDD (%) +^	34.7	56.9	60.5	<.001
Lifetime SUD (%) +^	55.4	73.1	68.4	<.001
Medical
Hepatitis C Virus (%)	--	17.7	20.2	.64
AIDS (%)	--	58.0	50.6	.24
Nadir CD4a	--	205.9 (172.2)	236.9 (224.6)	.54
Current CD4a	--	565.5 (282.1)	582.4 (319.6)	.67
cART (%)	--	86.1	84.4	.72
Plasma RNA Detectable (%)	--	24.7	26.7	.73
Estimated duration of HIV Infection (months)a	--	162.0 (96.8)	145.9 (96.5)	.19

Note. Data represent M (SD) or %. HIV = human immunodeficiency virus, HAND = HIV-associated neurocognitive disorders, POMS = Profile of mood states, GAD = generalized anxiety disorder, MDD = major depressive disorder, SUD = substance use disorder, AIDS = acquired immune deficiency syndrome; CD4 = cluster of differentiation 4; cART = combination antiretroviral medication.

^a= non-parametric Wilcoxon test

⁺indicates significant difference between HIV− and HAND− groups

^{^}indicates significant difference between HIV− and HAND+ groups

^#indicates significant difference between HAND− and HAND+ groups.

Neuropsychological evaluation

All participants were administered a comprehensive neuropsychological test battery by certified research assistants. HAND diagnoses were determined across the sample using the following 10 measures: (1) Logical Memory I from the Wechsler Memory Scale, 3rd edition (WMS-III; Wechsler, 1997), (2) WMS-III Logical Memory II, (3) Trail Making Test, Part A time (Army Individual Test Battery, 1944; Heaton et al., 2004), (4) Trail Making Test, Part B time, (5) WMS-III Digit Span subtest, (6) Total Execution Time from the Tower of London Test (Drexel Version; Culbertson & Zillmer, 1999), (7) Total Moves score from the Tower of London Test, (8) total words generated on a verbal (action) fluency test (Woods et al., 2005), and (9) Grooved Pegboard dominant and non-dominant hand completion times (Heaton et al., 2004; Klove, 1963). Demographically adjusted T-scores for each test were converted into deficit scores (range = 0 [normal] to 5 [severe]) and averaged to generate a Global Deficit Score (GDS; Carey et al. 2004) from which a cut-point of > .5 was used to classify HIV+ participants with and without HAND. Importantly for this study, CVLT-II scores were not included in generation of the GDS and subsequent HAND determination in order to avoid issues with criterion contamination. Within the HIV− comparison group, 86.7% were cognitively normal, according to the GDS cutoff, which is consistent with expectations based on a 1.0 standard deviation cut-point.

Neuropsychological and everyday functioning domain scores

Source memory

California Verbal Learning Test-II.

Participants were administered the California Verbal Learning Test-II according to the procedures outlined in the test manual (CVLT-II; Delis et al., 2000).

CVLT-II source memory indices.

As described in detail above, the CVLT-II output scores included two SM variables: Total Across-List Intrusions and Source Recognition Discriminability (d’). For this study, we used the normative scores of these two measures (i.e., the age-referenced percentile of CVLT-II Total Across-List Intrusions and the age- and sex-adjusted z-score for CVLT-II Source Recognition Discriminability d’). Our decision to use normed scores rather than raw scores was informed by our aim to create a standardized, clinician-friendly index of SM impairment. However, the results reported below did not differ meaningfully if raw scores were used instead of normed scores (descriptive group-level raw data are displayed in Table 2). Likewise, our decision to report CVLT-II normative standards herein rather than those of the newly published CVLT-3 was driven by the fact that the data were originally collected and normed using the CVLT-II and there will likely be a lag in the uptake of the CVLT-3 in clinical and research settings. Nevertheless, because the CVLT-3 uses the same calculations as the CVLT-II to generate the Total Across-Trial Intrusions and Source Recognition Discriminability (d’) (referred to as List A vs. List B Recognition Discriminability in the CVLT-3 manual) variables, we were able to provide both scores for the interested reader (see Table 2). The general agreement between SM index impairment rates on the CVLT-II and CVLT-3 was good (i.e., Kappa = .73, 95% confidence interval [CI], .67 to .79). Overall, there was 89.1% agreement, with 66.5% of the sample demonstrating intact performance on both CVLT versions and 22.6% showing impairment across versions.

Table 2.

Comprehensive descriptive data on the source memory variables derived from the California Verbal Learning Test (CVLT) and experimental tasks across the three-level HAND groups

Source Memory Variables	HIV− (n = 203)	HAND−(n = 287)	HAND+(n = 77)
California Verbal Learning Test (CVLT)
CVLT-II Raw Scores
Total Across-List Intrusions (no.)	0.5 (1.1)	0.6 (1.2)	1.0 (1.5)
Source Recognition Discriminability (d’)	3.1 (0.7)	2.9 (0.8)	2.4 (0.9)
CVLT-II Normative Standards
CVLT-II Source Memory Index (% impaired)	28.1	32.4	51.9
Total Across-Trial Intrusions
Continuous (%)+	52.2 (27.2)	48.7 (26.7)	40.7 (28.9)
Impaired (% < 16.0 percentile)	20.2	23.0	33.8
Source Recognition Discriminability (d’)
Continuous (z-score)#	0.3 (1.0)	0.1 (1.0)	-0.5 (1.1)
Impaired (% < 1SD)	15.8	19.9	39.0
CVLT-3 Normative Standards
CVLT-3 Source Memory Index (% impaired)	15.3	22.3	42.9
Total Across-List Intrusions
Continuous (cumulative %)+	80.0 (34.3)	77.9 (35.4)	66.9 (39.1)
Impaired (% < 16.0 percentile)	10.8	10.5	19.5
Source Recognition Discriminability (d’)
Continuous (scaled score) +	10.2 (2.6)	9.7 (2.7)	8.0 (2.9)
Impaired (% < 1SD)	8.9	15.7	35.1
Experimental Measures^
Self-Ordered Pointing Test source errors	9.2 (3.6)	10.4 (3.2)	12.3 (2.2)
Source and Item Memory Test source errors	2.8 (2.2)	3.3 (2.2)	3.9 (2.3)

Note. The CVLT-II Total Across-List Intrusions normative percentile represents the mean of the percentage of subjects in a given age group that had the same number of intrusions while the CVLT-3 Total Across-List Intrusions normative percentile is cumulative and represents the percentage of subjects in a give age group that had the same or more intrusions. CVLT = California Verbal Learning Test, HIV = human immunodeficiency virus, HAND = HIV-associated neurocognitive disorders

⁺= age-adjusted normative data

^#= sex- and age-adjusted normative data

^{^}= Experimental subsample n = 217 (HIV− n = 55; HAND− n = 138; HAND+ n = 24)

Across the entire study sample (N = 567), normative scores for CVLT-II Total Across-Trial Intrusions and Source Recognition Discriminability (d’) were significantly related, with a medium effect size (r_s = .31, p < .001). The general agreement between impairment rates for the normative scores of the CVLT-II Total Across-List Intrusions and Source Recognition Discriminability (d’) was fair (i.e., Kappa = .35, 95% CI, .26 to .44). Overall, there was a 77.4% rate of agreement between the two CVLT-II SM indices, with 10.9% of the sample exhibiting impairment on both SM indices and 66.5% demonstrating intact performance on both SM indices. Thus, the two CVLT-II SM indices measured related, but also separable, aspects of SM. As such, and in order to create a clinician friendly measure of SM (and limit our risk of Type I error), we collapsed these two CVLT-II SM metrics into a single indicator. Specifically, we generated a dichotomous CVLT-II SM impairment composite variable (herein referred to as CVLT-II SM index) in which participants were classified as having either no impairment on both CVLT-II Across Trial Intrusions and CVLT-II Source Recognition Discriminability (d’) (66.5%) or as being impaired (i.e., z-score ≥ 1.0 standard deviation below the normative mean) on either (22.6%) or both (10.9%) variables. A total of 33.5% of individuals were categorized as having impairment in one or both CVLT-II SM variables in the entire study sample. Thus, we only reported statistical analyses on the CVLT-II SM index, which allowed us to exercise some Type I error control by limiting the number of analyses conducted.

Experimental source memory tasks.

Participants were recruited in two funding phases and the 217 participants (HIV− n = 55; HAND− n = 138; HAND+ n = 24) recruited in the first phase completed experimental SM tasks as part of normal study procedures. The participants recruited in the first phase had a smaller proportion of persons with HIV than those recruited in the second phase that did not complete the experimental SM measures (p < .05), but did not otherwise differ in demographic or clinical factors (ps > .10). The subset of 217 participants completed modified versions of the Self-Ordered Pointing Test (SOPT; Petrides & Milner, 1982; Morgan et al., 2009) and the Source and Item Memory (SIM) Test (van Niekerk et al., 2004; Morgan et al., 2009), which was administered during the 5-minute delay of the SOPT. Raw scores for SOPT and SIM SM errors (see Table 2) were converted into sample-based T-scores then averaged to create a continuous experimental SM errors composite variable. Thus, use of a single measure allowed for control of Type I error.

Self-Ordered Pointing Test.

The modified SOPT included two trials in which participants were presented 12 patterned designs each with a colored border. In Trial 1, all designs had a red border and in Trial 2 all designs had a black border. During each trial, 12 stimulus pages were sequentially presented to the participant which each contained all 12 designs. The 12 designs were located in different positions in the array on each page. For each page, the participant was instructed to point to a patterned design that they had not previously pointed to on a previous page. Trial 2 was administered immediately after Trial 1 and followed an identical format with 12 novel patterned designs bordered in a different color than the first trial (i.e., all with black borders if the first trial designs had red borders).

Following a five-minute delay, participants completed a forced recognition task during which they were shown each of the previously presented patterned designs without a border and paired with a foil (i.e., novel) design and told to (a) choose which patterned design had been previously presented (measure of item memory), and (b) identify which color border had surrounded the patterned design (measure of SM). An SOPT SM error score was calculated for the total number of incorrect color borders identified. Possible SM error scores ranged from 0 (correct color border identified for each trial) to 24 (incorrect color border identified for each trial) and within the current sample ranged from 0–17. Aspects of these data in 60 HIV+ participants were also reported in relation to CVLT-II Total Across-List Intrusions in Morgan et al. (2009).

Source and Item Memory Test.

During the SIM Test, participants read 10 sentences aloud to the examiner (i.e., item information), each of which was attributed to one of five individuals (i.e., source information: grandmother, father, mother, son, or daughter); e.g., ‘Grandmother says: I like baked beans on toast.’

After a five-minute delay (during which the participant completed the SOPT recognition task described above), participants were presented with a 10-item paper and pencil task. For each item, the participant was told to choose from three possible sentences — one they had seen previously and two foils— which one they had seen before (item memory) and then were asked to choose from three options which individual the sentence had been attributed to (e.g., grandmother, father, or mother) (SM). A SIM SM error score was calculated for the total number of incorrect source attributions made. SM error scores ranged from 0 (choosing the correct source individual for each sentence) to 10 (choosing an incorrect source individual for each sentence), the range in the current sample was 0–10.

CVLT-II primary index

We created a dichotomous variable based on the primary memory indices from the CVLT-II (i.e., Total Trials 1–5 and Long Delayed Free Recall) for the purpose of anchoring the incremental validity of the SM index. Participants were coded as impaired if their score fell ≥ 1.0 standard deviation below the normative mean on either or both of these indices (28.4% impaired). There was a moderate association between this CVLT-II primary index and the CVLT-II SM index (Kappa = .45, 95% CI, .37 to .53). There were 14.0% of cases in which the SM index was impaired in the setting of normal primary scores and 9.2% of cases in which the SM index was normal despite impaired primary memory scores.

Rey-Osterrieth Complex Figure

The 217 participants that completed the experimental SM measures also completed the Rey Osterrieth Complex Figure test (ROCF) during which participants were instructed to reproduce a complex line drawing. The copy trial was scored with the Boston Qualitative Scoring System for the ROCF (BQSS; Somerville, Tremont & Stern, 2000) in which qualitative scales ranging from 0 to 4 were used to score different aspects of performance, whereby higher scores reflect better performance. For example, to determine the extent to which the examinee correctly planned and drew the correct size of the original figure, a transparent template was overlaid on the copied figure and lines on the template were marked to demonstrate deviation severity (0 [extreme horizontal/vertical expansion] to 4 [no horizontal/vertical expansion]). The sample-based z-scores of the BQSS Vertical Expansion (size distortion on the vertical axes) and Horizontal Expansion (size distortion on the horizontal axes) were averaged to create a visuospatial composite score. This score was selected to investigate the discriminant validity of the CVLT-II SM indices, as we were unaware of any literature to support a strong relationship between verbal SM and visuospatial ability.

Executive functions

An executive functions composite was generated from the full sample (N = 567) and was comprised of Trail Making Test, Part B time (Army Individual Test Battery, 1944; Heaton et al., 2004), Tower of London Total Moves score, and total words generated on a verbal (action) fluency test (Woods et al., 2006). Scores were converted into sample-based T-scores then averaged to create an executive functions composite.

Episodic memory composite

We generated a dichotomous retrospective episodic memory composite variable for which participants with scores ≥ 1.0 standard deviation below the normative mean on either WMS-III Logical Memory I or Logical Memory II were classified as impaired (19.2% of the sample).

Everyday functioning

Participants completed several everyday functioning measures that were used to create a dichotomous global indicator of their manifest everyday functioning. For instrumental and basic activities of daily living (iADLs and bADLs), participants completed the Heaton et al. (2004) version of the Lawton and Brody (1969) ADL scale. Global everyday functioning impairment was operationalized by the presence of two or more domains of disrupted everyday functioning. Our operationalization drew from the Frascati criteria of determining functional dependence (i.e., unable to maintain employment, assistance needed with more than two IADLS, and everyday cognitive symptoms; Antinori et al., 2007), an approach commonly used as an outcome measure in HIV studies (e.g., Blackstone et al., 2012). Participants were determined as employed or unemployed (not due to elective retirement) through self-report. All participants completed the Karnofsky Performance Status Scale (Karnofsky & Burchenal, 1949), which is a clinician’s rating of overall functioning (range: 0 [dead] to 100 [normal, no evidence of disease]) previously applied in HIV samples (Gandhi et al., 2011; Morgan et al., 2012), and ‘dependence’ for this domain was operationalized as having a score < 90. Participants were assessed using the Profile of Mood States (POMS; McNair, Lorr, & Droppleman, 1981) confusion/bewilderment scale and impairment was was operationalized as a score ≥ 1.0 standard deviations above the mean of age- and sex-adjusted scores (Nyenhuis et al., 1999).

Data analyses

To evaluate the convergent validity of the CVLT-II SM index, we conducted logistic regressions and one-way ANOVAs with CVLT-II SM index as the independent variable and the experimental SM error composite, retrospective episodic memory impairment, and executive functions composite as outcome variables. In order to simultaneously investigate the incremental validity of the CVLT-II SM index, we controlled for CVLT-II primary index impairment when evaluating its relationship to the experimental SM error composite and retrospective episodic memory impairment. To examine the discriminant validity of the CVLT-II SM index, we conducted a Wilcoxon rank-sum test to measure its relationship to the visuospatial composite. Next, we conducted a logistic regression examining the relationship between the CVLT-II SM index and the three-level HAND variable. For this analysis, possible covariates (i.e., demographics, psychiatric and medical factors) were selected using a data-driven approach by only including variables in the logistic regression that were related to both the CVLT-II SM index and the three-level HAND group variable. In the current sample, education, lifetime MDD, and lifetime substance use disorder were concurrently related to both the CVLT-II SM index and to the three-level HAND variable, and as such, these variables were included as covariates in the logistic regression. A secondary logistical regression was conducted using the experimental SM composite as the independent variable and the three-level HAND variable (HIV−, HAND−, HAND +) as the dependent variable to ascertain whether the HIV/HAND groups in our sample differed in SM ability. In order to investigate its ecological validity, we conducted a logistic regression examining the association between the CVLT-II SM index and everyday functioning impairment status. Given their importance and possible impact on everyday functioning, we included in this logistic regression a priori selected covariates, which included neuropsychological impairment, AIDS status, and lifetime MDD.

Results

Construct validity

Convergent and incremental validity

Within the subset of participants that completed the experimental SM task (n = 217), we conducted a logistic regression model with CVLT-II SM index and CVLT-II primary index impairment as the predictor variables and the experimental SM errors composite as the criterion measure. The overall model was significant, Adjusted R² = 0.14, F (2, 214) = 19.01, p < .001. The CVLT-II SM index (β = 0.64, p < .001) and the CVLT-II primary index impairment (β = 0.65, p = .03) were both significantly associated with the experimental SM errors composite. Post-hoc analyses showed that the CVLT-II SM index had a medium effect size association with both the visual (i.e., SOPT; d = .69) and verbal (i.e., SIM; d = .75) experimental measures of SM.

Within the whole sample (N = 567), we conducted a logistic regression model with CVLT-II SM index and CVLT-II primary index impairment as predictors of impairment on the WMS-III Logical Memory scales. The overall model was significant, X²[2] = 73.34, p < .001, and both the CVLT-II SM index (X²[1] = 7.12, p = .008, OR [odds ratio] = 1.96, [95% CI, 1.20 to 3.23]) and CVLT-II primary index (X²[1] = 36.02, p < .001, OR = 4.57, [95% CI, 2.78 to 7.50]) were significantly associated WMS-III Logical Memory impairment. Likewise, analysis of variance showed a significant association between the CVLT-II SM index and the executive functions composite, F (1, 554) = 22.15, p < .001, Cohen’s d = 0.42. Post-hoc analyses revealed the CVLT-II SM index was related to each measure of executive functions (i.e., Trails B, action verbal fluency, and total moves from the Tower of London) with small-to-medium effect sizes (Cohen’s d range = .21 to .43).

Discriminant validity

Impairment on the CVLT-II SM index was not associated with scores on the BQSS visual construction composite (Z = −0.91, p = .364, Cohen’s d = .20) (see Table 3).

Table 3.

Descriptive data for the three CVLT-II source memory impairment metrics across the primary study outcomes

Variables	CVLT-II Source Memory Variables
	Source Memory Index		Source Recognition Discriminability d’		Total Across-List Intrusions
	Normal (n=377)	Impaired (n=190)	Normal (n=448)	Impaired (n=119)	Normal (n= 434)	Impaired (n=133)
Experimental SM Errors (T-Score)+	47.9 (7.8)	54.2 (7.1)	48.9 (8.1)	53.8 (7.1)	48.5 (7.7)	54.9 (7.6)
Episodic Memory (WMS-III LM % impaired)	11.9	33.6	13.4	41.2	16.4	28.6
Executive Functions (T-Score)	49.6 (7.4)	46.5 (7.4)	49.2 (7.4)	46.0 (7.4)	49.1 (7.5)	46.5 (7.3)
BQSS Visual Construction (T-Score)	50.6 (8.3)	48.7 (10.4)	50.2 (8.8)	49.0 (10.0)	50.5 (8.5)	48.0 (10.8)
Everyday Functioning (% dependent)*	38.1	49.6	38.6	54.0	40.8	46.7

Note: Data represent M(SD) or frequencies (%)

⁺= Includes only participants that completed the experimental source memory task.

^*= Includes only participants with HIV (n = 364). SM = source memory, HAND = HIV-associated neurocognitive disorders, BQSS = Boston Qualitative Scoring System of the Rey Complex Figure, CVLT-II = California Verbal Learning Test – Second Edition. WMS-III-LM = Wechsler Memory Scale – Third Edition Logical Memory.

Diagnostic validity

A logistic regression was conducted within the whole sample (N = 567) with the CVLT-II SM index, education, lifetime MDD, and lifetime SUD predicting the three-level HAND variable (HIV−, HAND−, HAND+). The overall model was significant (χ²[8] = 57.24, p < .001). The CVLT-II SM index was significantly and independently associated with the three-level HAND variable (χ²[2] = 9.78, p = .008). Descriptive data are presented in Figure 1. Post-hoc pairwise chi-square comparisons showed that the HAND+ group had a significantly higher rate of impairment on the CVLT-II SM index relative to the HAND− (χ²[1] = 9.72, p = .002, OR = 2.25 [95% CI, 1.35 to 3.76]) and HIV− (χ²[1] = 13.65, p < .001, OR = 2.77 [95% CI, 1.61 to 4.76]) groups. There was no significant difference between the HAND− and the HIV− groups (χ²[1] = 1.05, p = .305, OR = 1.23 [95% CI, 0.83 to 1.82]). All other covariates in the model (education, lifetime MDD, lifetime SUD) were also significant independent predictors of the three-level HAND variable (ps < .05).

Figure 1.

Frequencies of California Verbal Learning Test – Second Edition source memory errors across the study groups. HIV = human immunodeficiency virus, HAND = HIV-associated neurocognitive disorders.

Next, a logistic regression was conducted within the subset of 217 participants that completed the experimental SM tasks to investigate the relationship between the experimental SM errors composite and HAND groups. The experimental SM errors composite was significantly associated with HAND group status, F(2, 214) = 6.54, p = .002). Post-hoc Tukey-Kramer HSD analyses indicated that the HAND+ group had significantly higher T-scores (indicating more errors made) than the HIV− (p < .001, Cohen’s d = 0.82) and HAND− group (p = .035, Cohen’s d = 0.52), while difference in errors made between the HAND− and HIV− groups fell at the level of a trend and was accompanied by a small-to-medium effect (p = .060, Cohen’s d = 0.34).

Criterion validity

Ecological validity

Within the HIV+ group, a univariate (unadjusted) chi-square test revealed a significant association between the CVLT-II SM index and global everyday functioning impairment (χ²[1] = 4.58, p = .032, OR = 1.60 [95% CI, 1.04 to 2.46]) (see Table 3). In parallel, a univariate chi-square test did not reveal a significant association between the CVLT-II primary index and global everyday functioning impairment (p = .160, OR = 1.37 [95% CI, 0.88 to 2.13]).

We conducted a follow-up multivariable logistic regression with neuropsychological impairment status, AIDS status, lifetime MDD, and the CVLT-II SM index predicting everyday functioning impairment status. The overall model was significant (χ²[4] = 28.32, p < .001), but only AIDS status and lifetime MDD were significantly associated with everyday functioning (ps < .01). The CVLT-II SM index was not significantly related to everyday functioning impairment in this model (χ²[1] = 1.93, p = .165, OR = 1.38 [95% CI, 0.88 to 2.18]).

Discussion

The current study provides the first systematic analysis of the construct and criterion validity of CVLT-II SM measures (i.e., Total Across-List Intrusions and Source Recognition Discriminability [d’]). The convergent validity of the CVLT-II SM index was supported by its modest association with experimental measures of SM. The CVLT-II SM index was related with a medium effect size to both the visual and verbal experimental measures of SM. As the CVLT-II SM variables are derived from solely verbal tasks (e.g., verbal recall of a list presented auditorily), the strength of its association with tasks measuring both visual and verbal SM provide additional evidence for the construct validity of the CVLT-II variables as a measure of SM. As is true of most higher-order cognitive tests, the CVLT-II SM index is multifactorial, and also demonstrated expected convergent relationships with tests of its various cognitive subcomponents: retrospective episodic memory and executive functions. CVLT-II SM index impairment was associated with poorer performance on well-validated measures of executive functions, including speeded cognitive flexibility (i.e., Trails B), generativity (i.e., action verbal fluency), and visual planning (i.e., total moves from the Tower of London) (Unterrainer et al., 2010; Cattie et al., 2012; Woods et al., 2005; Crowe, 1998). From a neural perspective, the association between executive functions and SM is supported by their shared dependence on frontal systems (e.g., Baldo et al., 2002), while from the perspective of cognitive models, successful SM places demands on executive processes like concept shifting and divided attention (e.g., Glisky et al., 1995). These findings bolster the convergent validity of the CVLT-II SM index, as well as provide further support for previous studies that have related experimental SM impairment to poor performance on tests of executive function in the setting of HIV (e.g., Morgan et al., 2009).

Findings from this study also support the potential incremental validity of the CVLT-II SM index. First, the CVLT-II provided unique information in approximately one-quarter of cases (e.g., participants were impaired on the CVLT-II SM index but not on the CVLT-II primary index and vice versa). Second, the CVLT-II SM index was associated with experimental SM and episodic retrospective memory deficits above and beyond the primary CVLT memory indices (i.e., Total Trials 1–5 and Long Delay Free Recall). This association is particularly compelling given the conservative nature of our analyses, which suffered from shared method variance because both predictor variables were from the CVLT-II. These results were in line with the tradition of the CVLT-II, which provides component process measures to complement the interpretation of the reliable and robust traditional indices (Delis et al., 2000).

Results supported the discriminant validity of the CVLT-II SM index, as the CVLT-II SM index was not associated with a visuospatial composite. Vertical and horizontal expansion scores on the Copy portion of the ROCF visuospatial tasks were chosen a priori as measures of discriminant validity as they were theorized to place minimum demand on executive functions and declarative memory. Additionally, the graphomotor output modality of the visuospatial task requires construction as opposed to the auditory-verbal format of the CVLT-II.

The diagnostic validity of the CVLT-II SM index was supported by its ability to identify patients with HAND. Consistent with the findings of Morgan et al. (2009) in an HIV sample, HAND was associated with moderate deficits on the experimental SM tasks independent of clinicodemographic factors. In parallel, patients with HAND were two to four times more likely to be impaired on the CVLT-II SM index as compared to neurocognitive normal HIV+ persons and HIV− individuals. Thus, the CVLT-II SM index discriminates between clinical groups with known SM impairment. Significant differences in CVLT-II impairment rates between diagnostic groups persisted when controlling for education, lifetime MDD, and lifetime SUD. The pattern of results was expected, as the frontal-striatal-thalamic circuit abnormalities that can disrupt SM and related cognitive functions are reliably observed in individuals with HIV (Du Plessis et al., 2014).

Findings were mixed regarding the criterion (i.e., ecological) validity of the CVLT-II SM index. Univariate analyses revealed that within the HIV+ group, persons with impairment on the CVLT-II SM index were more than 1.5 times more likely to have impaired global everyday functioning; however, these findings became null when we covaried for AIDS status, lifetime MDD, and global neuropsychological impairment. Unlike the CVLT-II SM index, the CVLT-II primary index was not significantly associated with everyday functioning outcomes at the univariate level, though comparison of the two odds radios show highly overlapping confidence intervals (CVLT-II SM OR = 1.60 [95% CI, 1.04 to 2.46]; CVLT-II primary index OR = 1.37 [95% CI, 0.88 to 2.13]). Therefore, by considering effect sizes there does not appear to be a meaningful difference in the strength of association between source memory or traditional memory indices on the CVLT-II for predicting everyday functioning outcomes. Overall, while the CVLT-II SM variables did not provide compelling evidence of incremental ecological validity, the very conservative nature of the covariates included in the model should give pause to dismissing the relationship between SM and everyday functioning.

Indeed, we are aware of only one prior study that has investigated the relationship between SM and everyday functioning outcomes. In a sample of 82 older adults with and without mild cognitive impairment (Schmitter-Edgecombe, Woo, & Greeley, 2009), visual SM errors were negatively correlated with all subdomains (mean r = .27) of the Alzheimer’s Disease Activities of Daily Living – International Scale (ADL-IS; Resiberg et al., 2001) in unadjusted analyses. It may certainly be the case that SM errors lead to different types of daily malfunction not captured by standard everyday functioning assessments. For example, assessing medication adherence in standard everyday functioning assessments may capture one’s ability to take their medication everyday but may not capture that due to a SM misattribution one might be diligently taking the wrong kind of medicine or holistic treatment thinking that it was highly recommended by a reliable health care source. Thus, the association between SM and everyday functioning should not be wholly dismissed without further investigation into these subtler everyday errors that may have large impacts on daily living. For example, future work may examine the possibility that SM plays a role in distilling reliable from unreliable information in everyday life to make health decisions (e.g. distinguishing between advice from a healthcare provider, friend, or newspaper as the source for health information; see Memory for Health Information task, McDaniel et al., 2014).

There are a number of limitations in the current study that are important to consider. First, the interpretation of our findings is limited due to the relatively high rates of psychiatric, medical, and substance use comorbidities in the HIV− sample. Although the use of a comparison (vs. genuinely healthy) sample such as this is useful for methodologically controlling for common confounds, it nevertheless increases our risk of Type II error. Our sample also was comprised of predominantly men and a majority of individuals were Caucasian, which further limits generalizability. Also of note, our HIV+ sample had a relatively low rate of HAND (21%) compared to those found in other population-based studies (Heaton et al., 2011), which may further restrict our generalizability and increase our risk of Type II error. Future studies should investigate SM in the context of HIV+ samples with rates of HAND that more closely align with those of current population estimates. While we were not able probe whether similar versus differing semantic category across-trial intrusion errors were differentially associated with our outcome measures due to the low frequency of across-list intrusions in the current sample, this may be an area for future exploration. Finally, we elected to focus our analyses on the CVLT-II rather than the CVLT-3, given the myriad issues in adopting new tests (e.g., Loring et al., 2010). As reported in the Methods and Table 2, our findings remained largely unaltered when we used CVLT-3 normative data. Nevertheless, it may be important for clinicians to note that there were some SM impairment frequency differences when using CVLT-3 (22.6%) versus CVLT-II (33.5%) normative standards on this sample (see Table 2). This discrepancy may be a function of the CVLT-3 normative sample base rates demonstrating overall more SM errors than the CVLT-II normative sample base rates, resulting in more conservative norms for source variables on the CVLT-3.