Abstract
Background
Cognitive deficits are common, long-term sequelae in children and adolescents with congenital heart disease (CHD) who have undergone surgical palliation. However, there is a lack of a validated brief cognitive screening tool appropriate for the outpatient setting for adolescents with CHD. One candidate instrument is the Montreal Cognitive Assessment (MoCA) questionnaire.
Objective
The purpose of the research was to validate scores from the MoCA against the General Memory Index (GMI) of the Wide Range Assessment of Memory and Learning, 2nd Edition (WRAML2), a widely accepted measure of cognition/memory, in adolescents and young adults with CHD.
Methods
We administered the MoCA and the WRAML2 to 156 adolescents and young adults ages 14-21 (80 youth with CHD and 76 healthy controls who were gender and age matched). Spearman rank order correlations were used to assess concurrent validity. To assess construct validity, the Mann Whitney U-test was used to compare differences in scores in youth with CHD and the healthy control group. Receiver operating characteristic curves (ROC) were created and area under the curve, sensitivity, specificity, positive predictive value, and negative predictive value were also calculated.
Results
The MoCA median scores in the CHD versus healthy controls were (23 [range, 15-29] versus 28 [range, 22-30; p < .001]), respectively. With the screening cutoff scores at < 26 points for the MoCA and ≤ 85 for GMI (<1 SD, m = 100, SD = 15), the CHD versus healthy control groups showed sensitivity = .96 and specificity = .67 versus sensitivity = .75 and specificity = .90, respectively, in the detection of cognitive deficits. A cutoff score of 26 on the MoCA was optimal in the CHD group; a cutoff of 25 had similar properties except for a lower negative predictive value. The area under the ROC curve (95% CI) for the MoCA was 0.84 (95% CI [0.75, 0.93]; p = < .001) and 0.84 (95% CI [0.62, 1.00]; p = .02) for the CHD and controls, respectively.
Discussion
Scores on the MoCA were valid for screening to detect cognitive deficits in adolescents and young adults aged 14-21 with CHD when a cutoff score of 26 is used to differentiate youth with and without significant cognitive impairment. Future studies are needed in other adolescent disease groups with known cognitive deficits and healthy populations to explore the generalizability of validity of MoCA scores in adolescents and young adults.
Keywords: adolescents, concurrent validity, congenital heart disease, Montreal Cognitive Assessment, youth
Cognitive deficits are the most common, and potentially the most harmful, sequelae of adolescents and young adults with congenital heart disease (CHD) who have undergone surgical palliation. The prevalence of cognitive deficits has been estimated in up to 50% of survivors with complex CHD (Markowitz, Ichord, Wernosky, Gaynor, & Nicholson, 2007; Wernovsky, 2006). Mechanisms contributing to cognitive deficits in CHD are complex and multifactorial including hypoxic/ischemic injury triggered by the CHD condition, hypoperfusion during cardiac surgery, and a wide range of genetic, prenatal, and other pre- and postoperative risk factors (Gaynor et al., 2015; Dominguez, Wernovsky, & Gaynor, 2007; Ballweg, Wernovsky, & Gaynor, 2007). A distinct pattern of mild cognitive and behavioral impairment associated with problems with reasoning, learning, memory, executive function, inattention, and impulsive behavior, language, and social skills has emereged (Bellinger et al., 2015; Bellinger et al., 2011; Cassidy, White, DeMaso, Newburger & Bellinger, 2015; von Rhein et al., 2015; Murphy et al., 2015). However, many of these deficits may not become apparent until school age, when higher-level organizational skills are required. Despite efforts aimed at prevention and early detection to minimize cognitive deficits, many with CHD will have deficits reaching into adulthood, which can impact educational achievement (Shillingford et al., 2008, Wray & Sensky, 2001) employability (Niwa et al., 2002), self-care (McCabe et al., 2015) and health-related quality of life (Pike et al., 2012; Kahr, Radke, Orwat, Baumgartner, & Diller, 2015).
In an effort to promote early detection of developmental and cognitive deficits, the American Heart Association and American Academy of Pediatrics recently issued surveillance, screening, and evaluation guidelines for neurodevelopmental assessment, including cognitive factors (Marino et al., 2012). Periodic reevaluation of high-risk children is recommended by the medical home provider at 12 to 24 months, three to five years, and 11-12 years of age (Marino et al., 2012). However, the ability of the medical home provider to objectively screen cognitive function in the clinical setting can be challenging because a wide variety of instruments is available, highly trained test administrators are needed, administration requires time, additional appointments may be needed, families may incur additional out-of-pocket costs, and follow-up recommendations for school may be needed—if an individualized education or 504 plan is indicated. In addition, there are no cognitive screening recommendations for the CHD population after transition into adulthood. It remains unclear to what extent these cognitive deficits—identified at a younger age—persist or worsen into adulthood. Currently, there is a lack of brief screening tools to detect cognitive deficits in multiple domains that can be administered in the outpatient setting that could provide objective data needed to refer for a more comprehensive neurocognitive evaluation.
Various tools for specific cognitive domains (e.g., intelligence, processing, visual perception, executive functioning) have been used in the adolescent CHD population (Marino et al., 2012). However, consensus on the best instruments to measure cognition in youth with CHD is lacking. One commonly accepted instrument is the Wide Range of Assessment Memory and Learning 2nd edition (WRAML2; Sheslow & Adams, 2003). The WRAML2 is a cognitive test that takes approximately one hour to administer, with scores validated in patients age five to 90 years. Although the WRAML2 is an excellent assessment tool of general cognition and memory, it lacks many of the desired characteristics for most outpatient applications (short administration time, minimal training for users). The WRAML2 focuses on memory and learning abilities, and requires intensive training for consistent application by test administrators, but covers many of the components included in the Montreal Cognitive Assessment (MoCA) screening tool.
The MoCA is a brief, cognitive screening test (10 minutes) that can be administered in the clinical setting, covers a broad range of domains essential for adolescent cognitive assessment (executive function, attention, memory, language, and visuospatial), requires minimal user training, and is sensitive to detect mild cognitive deficits; it was originally developed to evaluate patients 18 years and older to rule out mild cognitive impairment (Nasreddine et al., 2005). Subsequently, the MoCA has been applied more widely to hypoxic-ischemic/vascular cognitive impairment, such as stroke (Godefroy et al., 2011), traumatic brain injury (Wong et al., 2013), epilepsy (Phabphal & Kanjanasatien, 2011), and heart failure in adults (Cameron, Worrall-Carter, Page, Stewart, & Ski, 2013). Much of the hypoxic/ischemic white matter injury in CHD patients has been identified by magnetic resonance imaging studies in areas associated with cognition and attention (Rollins et al., 2014). Interestingly, the MoCA has been identified to place greater emphasis on frontal cortex tasks, such as executive function and attention potentially making it more sensitive to non-Alzheimer diseases or conditions (Smith, Gildeh, & Holmes, 2007). However, only a few studies have documented the use of the MoCA in mid to late adolescents (> 15 years of age) (Mittal, Verma, Jain, Khatter, & Juyal, 2012; Phabphal & Kanjanasatien, 2011). Furthermore, many of the tasks on the MoCA are parts of or similar to many different neurocognitive batteries already used and validated for adolescents and young adults with and without CHD. Theoretically, all of these features should make the MoCA screening tool ideal for the evaluation of adolescents and young adults with suspected cognitive deficits. However, there are no reports of validity of MoCA scores in this age range or patient population.
Purpose
The specific aim was to validate MoCA scores in adolescents and young adults with CHD against scores on the WRAML2. We hypothesized that there would be moderate to strong positive relationships in between the MoCA and WRAML2 in the CHD patient population. We also hypothesized that there would be differences in scores between youth with CHD and healthy control.
Methods
Design
This study used a cross-sectional, comparative, and correlational design. Age- and gender-matched adolescents and young adults with surgically palliated CHD and health controls took part.
Sample and Procedures
Participants with CHD were recruited via flyers or provider referrals from University of California, Los Angeles, and Children’s Hospital Los Angeles pediatric cardiology clinics and private practice cardiology groups in Southern California. We included adolescents and young adults with CHD between the ages of 14-21 who have undergone surgical palliation requiring cardiopulmonary bypass (CPB). Congenital heart disease participants were excluded, if they had isolated coarctation of the aorta or patent ductus arteriosus (not requiring CPB), previous head injury (e.g., concussion, stroke), and severe developmental delay precluding active study participation and self-reporting (e.g., cerebral palsy). If eligible, either a same-day clinic or future appointment was made to participate in the study.
Healthy controls were recruited from campus and community flyers, local high schools, or were friends or relatives of the CHD participants. Controls were identified as healthy by self or parent report. Participants were excluded for any chronic medical or psychiatric conditions (e.g., anxiety, depression) or any previous head injury (e.g., concussions). If eligible, controls were matched to a CHD participant for age (plus or minus two years) and gender, and an appointment was made to participation in the study either at their homes, public libraries or research offices.
This study was approved by the Institutional Review Boards at all recruitment sites. Parental permission and assent were obtained for participants under age 18, and informed consent was obtained from participants age 18 and over. Clinical and demographic data were collected from the subjects and/or their medical records. All study procedures were performed with the adolescent and test administrator in a private room. The test administrators were two graduate-trained, research assistants who met the qualifications for administration with interrater agreement of 100% for the MoCA and 98% for the WRAML2. The MoCA was administered first, followed by the WRAML2 testing (all core tests and optional subtests). All participants completed all tests. Total testing time was approximately one hour and 15 minutes with no withdrawals or incomplete tests.
Montreal Cognitive Assessment (MoCA)
The MoCA is a brief cognitive screening tool administered by the examiner using a narrative script. The MoCA takes approximately 10-12 minutes to administer, and participants are asked to draw on the tool for the first three items, with the remaining tasks requiring a verbal response from participants. The MoCA assesses a broad range of cognitive domains: (a) visual-spatial skills, using a clock-drawing task and a three-dimensional cube; (b) executive functions (multiple aspects), using an alternation task (alternating numbers and letters of the alphabet in ascending order) adapted from the Trail-Making B, a phonemic fluency task, and a two-item verbal abstraction task; (c) short-term memory recall task, involving two learning trials of five nouns, and delayed recall after five minutes; (d) attention, concentration, and working memory, using a sustained attention task, a serial subtraction task, and digits forward and backward; (e) language, using a three-item confrontation naming task (asked to name three animals from pictures), repetition of two syntactically complex sentences, and the aforementioned fluency task; and (f) orientation to time and place. The maximum score on the MoCA is 30 points, and scores are corrected for the number of years of education (i.e., if a subject has fewer than 12 years of education, a point is added to the total score; Nasreddine et al., 2005). Scores greater than or equal to 26 are considered normal, and scores less than 26 are considered abnormal, indicating cognitive impairment in adults. This cognitive screener has relatively simple instructions which can be quickly mastered for administration by healthcare personnel. The alpha reliability for the MoCA subtest scores is .83; sensitivity and specificity associated with detection of mild cognitive impairment in adults are 90% and 87%, respectively (Nasreddine et al., 2005).
Prior to use for this study, content validity of the MoCA was assessed and confirmed by a panel of five experts (psychologists, educators, and cardiologists who specialize in pediatrics and/or CHD) to assess the instructions and applicability of the tool for mid to late adolescents. Individual interviews were performed by the investigator on 10 English-speaking adolescents (five CHD/five healthy controls) to assess understanding and clarity of the verbal instructions and visual items (visuospatial/executive writing/drawing tasks, and naming of animals) in the subscales. All adolescents performed the MoCA without hesitation or questions after verbal instructions were given. After the MoCA was administered, 100% agreement was obtained regarding clarity and understanding of the subtest items in all 10 adolescents. The adolescents who took part in the prestudy interviews and test administrator agreements were not part of the study sample.
Wide Range Assessment of Memory and Learning, Second Edition (WRAML2)
Memory was measured using the WRAML2. This administered test takes approximately one hour to complete, and is a highly reliable and valid measure of memory and learning function in subjects from ages five to 90 (Sheslow & Adams, 2003). The WRAML2 is a comprehensive test that measures an overview of memory function which consists of verbal and visual memory, attention/concentration, working memory and memory recognition. The core battery consists of six subtests: story memory, verbal learning, design memory, picture memory, finger window (short-term memory of a visual sequential pattern), numbers/letters (digit-span format using both numbers and letters) that when combined yield a general memory index (GMI) score (expected mean = 100, SD = 15, 1 SD below the expected mean is considered impaired ≤ 85; Sheslow & Adams, 2003). Additional, optional subtests performed were working memory, sentence memory, and memory recognition yielding the general memory recognition index (GMR) score (expected mean = 100, SD = 15). The GMI measures immediate recall and the GMR measures delayed recall. The GMI has been used in previous cognitive studies in CHD (Simmons, Glidden, Sheslow, & Pizarro, 2010). Given the length and complexity of the WRAML2, special training is required for its administration. The alpha reliabilities for scores on the core subtests range from .85 to .94 (GMI alpha reliability = .93; Sheslow & Adams, 2003). External validity was established with moderate to high correlations between GMI scores and scores on other measures of memory and learning (Sheslow & Adams, 2003). The WRAML2 was also correlated with cognitive measures, such as the Full Scale IQ of the Wechsler Adult Intelligence Scale (WAIS-III) and Wechsler Intelligence Scale for Children (WISC-III), which showed a moderate to high correlation with the GMI (.67 and .44, respectively; Sheslow & Adams, 2003). Furthermore, the WMI and Attention Concentration scores of the WRAML2 correlate .67 and .69, respectively, with the WMI of the WAIS-III (Sheslow & Adams, 2003).
Statistical Analysis
Subjects were classified into CHD and healthy control groups. Variables were examined for normality and outliers. The continuous data had nonnormal distributions (based on Shapiro-Wilks tests of normality) and groups were compared using nonparametric statistics consisting of the Mann-Whitney U-test for all continuous variables and χ2 tests for all categorical variables. Characteristics of the sample are presented as means with standard deviations and medians with range for continuous variables. We calculated sensitivity as the proportion of positives (< 26 on the MoCA; indicating impairment) to the positives on the gold standard (GMI ≤ 85; indicating impairment) and the specificity as the proportion of negatives (≥ 26 on the MoCA; normal) to negatives on the gold standard (GMI > 85; normal) of the WRAML2.
A scatterplot was used to illustrate the relationship between scores on the MoCA and GMI scores in the two groups. Positive and negative predictive values were computed on the basis of a 2 × 2 table showing cognitive impairment frequencies based on MoCA and GMI scores. Direct associations between scores on the MoCA and WRAML2 subtests were made using Spearman rank correlation, and the Mann-Whitney U–test was used for between group comparisons. The reliability of the MoCA test scores were estimated by Cronbach’s alpha coefficient in each group. The receiver operating characteristic (ROC) curves and corresponding areas under the curve (AUCs) were computed by group. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were computed for various cutoffs of the MoCA. Another goal for this analysis is to identify the best cutoff point for the CHD group. A good cognitive screening tool should have a sensitivity > 80% and specificity > 60%, with AUCs above 80% showing good diagnostic accuracy or above 90% indicate excellent diagnostic accuracy (Plichta & Kelvin, 2012). All analyses were conducted with SPSS 23 (Somers, NY). Nominal p-values of .05 were reported.
Results
Sample Characteristics
One hundred and fifty-six participants were recruited (80 youth with CHD and 76 healthy controls). The sample demographics, clinical disease severity, and total MoCA and WRAML2 GMI scores are summarized in Table 1. No statistically significant differences in age, gender, ethnicity, and education emerged between groups. The study showed significant differences in cognitive performance based on median MoCA and GMI scores between the CHD and heathy controls (23 vs. 28; p < .001; 85 vs. 108; p < .001), respectively (discriminant validity).
TABLE 1. Characteristics and Cognitive Assessment: Participants with Congenital Heart Disease and Healthy Control Groups.
| Congenital heart disease |
Healthy controls |
||||||||
|---|---|---|---|---|---|---|---|---|---|
| Characteristic | n | (%) | n | (%) | p | ||||
| Gender (male) | 47 | (59) | 44 | (59) | 1.00 | ||||
| Race/ethnicity | |||||||||
| White | 41 | (51) | 38 | (50) | .07 | ||||
| Hispanic | 32 | (40) | 28 | (37) | |||||
| Other | 7 | (9) | 10 | (13) | |||||
| Education | |||||||||
| Grade (9-11) | 54 | (68) | 46 | (60) | .41 | ||||
| High school | 14 | (17) | 14 | (18) | |||||
| College (1-4 years) | 12 | (15) | 16 | (21) | |||||
| MoCA (total score) | |||||||||
| <26 cutoff | 55 | (69) | (10) | (13) | |||||
| <25 cutoff | 51 | (63) | (7) | (9) | |||||
| CHD severitya | |||||||||
| Simple | 3 | (4) | |||||||
| Moderate | 30 | (37) | |||||||
| Complex | 47 | (59) | |||||||
|
|
|
||||||||
| M | (SD) | Mdn | Range | M | (SD) | Mdn | Range | ||
|
|
|||||||||
| Age (years) | 16.8 | (2.1) | 17 | 14-21 | 17.7 | (2.2) | 18 | 14-21 | .70 |
| WRAML2 GMI | 86.2 | (11.7) | 85 | 49-112 | 108.4 | (11.6) | 108 | 85-127 | <.001 |
| MoCA | 23.2 | (3.4) | 23 | 15-30 | 27.8 | (2.0) | 28 | 22-30 | <.001 |
| Visuospatial/executive | 4 | 1-5 | 5 | 3-5 | <.001 | ||||
| Naming | 3 | 2-3 | 3 | 2-3 | .56 | ||||
| Attention | 4 | 1-6 | 6 | 3-6 | <.001 | ||||
| Language | 2 | 0-3 | 3 | 1-3 | <.001 | ||||
| Abstraction | 1 | 0-2 | 2 | 0-2 | <.001 | ||||
| Delayed recall | 3 | 0-5 | 5 | 0-5 | <.001 | ||||
| Orientation | 6 | 5-6 | 6 | 5-6 | .70 | ||||
| Total | 23 | 15-29 | 28 | 22-30 | <.001 | ||||
Note. Congenital heart disease (n = 80); healthy controls (n = 76).CHD = congenital heart disease; GMI = General Memory Index; MoCA = Montreal Cognitive Assessment; WRAML2 = Wide Range Assessment of Memory and Learning, Second Edition.
Severity was based on the Bethesda Conference classification (Warnes et al., 2001).
Montreal Cognitive Assessment Subtests between CHD and Healthy Control Groups
All MoCA subtests were statistically significant except for naming and orientation between groups shown in Table 1. Fifty-five (69%) in the CHD group scored < 26 (impaired) on the MoCA compared to 10 (13%) in the healthy control.
Psychometric Properties of the Montreal Cognitive Assessment
The concurrent validity of the MoCA test score and the GMI was significant (rS = .70; p < .001) using the Spearman rank correlation coefficient. We compared the MoCA to the gold standard GMI subscales and other subtests of the WRAML2, and found modest to high correlations that were statistically significant in the total study cohort and by group (CHD and control) (Table 2). The highest values for each measure in the CHD group are Visuospatial/Executive and Working Memory Index (rS = .34), Naming (no correlation), Attention and Verbal Memory Index (rS = .64), Language and Sentence Memory (rS = .53), Abstraction and Verbal Memory Index (rS = .39), Delayed Recall and Verbal Learning (rS = .43), and Orientation and Numbers/Letters (no correlation). In addition, the scatterplot in Figure 1 shows a strong, positive, rank order correlation between the MoCA and WRAML2 GMI total scores in both groups (rS =.77; p < .001).
TABLE 2. Spearman Rank Correlation Coefficients among MoCA Subtests and WRAML2 Verbal Memory, Visual Memory, Attention/Concentration, and Other Subtests.
| Verbal memory |
Visual memory |
Attention/concentration |
Other |
|||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MoCA/group | StM | VL | VMI | DM | PM | ViMI | FW | NL | ACI | WM | SM | GMI |
| VSE | ||||||||||||
| Total | .41*** | .31*** | .40*** | .29*** | .23*** | .23* | .44*** | .30*** | .43*** | .46*** | .40*** | .48*** |
| CHD | .26* | .03 | .16 | .19 | −.03 | .11 | .34*** | .08 | .30*** | .31*** | .23* | .30*** |
| Controls | .25* | .06 | .39*** | .24* | .20 | .23* | .28* | .20 | .42*** | .46*** | .23* | .48*** |
| Naming | ||||||||||||
| Total | .01 | −.06 | −.04 | −.01 | .09 | .06 | −.03 | .01 | −.02 | .03 | .10 | −.30 |
| CHD | .13 | .16 | .15 | −.10 | .15 | .06 | .18 | .11 | .16 | .13 | .05 | .17 |
| Controls | −.02 | −.20 | −.04 | .08 | .04 | .07 | −.13 | .03 | −.02 | .03 | .20 | −.30 |
| Attention | ||||||||||||
| Total | .42*** | .50*** | .50*** | .22*** | .05 | .16* | .34*** | .38*** | .40*** | .52*** | .45*** | .49*** |
| CHD | .50*** | .60*** | .64*** | .19 | .06 | .07 | .37*** | .30*** | .40*** | .50*** | .34*** | .57*** |
| Controls | .10 | .11 | .50*** | .17 | .06 | .15 | .05 | .23 | .40*** | .51*** | .40*** | .49*** |
| Language | ||||||||||||
| Total | .40*** | .42*** | .42*** | .20* | .06 | .24*** | .30*** | .50*** | .45*** | .60*** | .62*** | .50*** |
| CHD | .40*** | .34*** | .37*** | .10 | .02 | .13 | .21 | .35*** | .30* | .53*** | .51*** | .40*** |
| Controls | −.00 | −.00 | .42*** | .10 | .10 | .20 | −.16 | .40*** | .45*** | .59*** | .44*** | .50*** |
| Abstraction | ||||||||||||
| Total | .44*** | .43*** | .50*** | .20* | .09 | .20* | .33*** | .40*** | .35*** | .43*** | .41*** | .49*** |
| CHD | .35*** | .32*** | .34*** | .13 | .20 | .21 | .20 | .26* | .31* | .28* | .30* | .39*** |
| Controls | .20 | .10 | .49*** | .05 | .10 | −.01 | −.03 | .05 | .35*** | .43*** | .12 | .49*** |
| Delayed recall | ||||||||||||
| Total | .40*** | .50*** | .47*** | .23*** | .10 | .23*** | .34*** | .23*** | .32*** | .45*** | .41*** | .47*** |
| CHD | .30*** | .33*** | .34*** | .12 | −.04 | .05 | .30* | .20 | .25* | .40*** | .43*** | .32*** |
| Controls | .20 | .23* | .47*** | .20 | .21 | .26*** | −.00 | −.23 | .32*** | .45*** | −.15 | .46*** |
| Orientation | ||||||||||||
| Total | .06 | .05 | .04 | −.05 | −.20 | .03 | .07 | .20* | .14 | .13 | .14 | .10 |
| CHD | .03 | −.03 | −.04 | −.14 | −.05 | −.12 | −.03 | .10 | .07 | .04 | .05 | −.02 |
| Controls | .07 | .11 | .04 | .07 | .03 | .07 | .11 | .21 | .14 | .13 | .27* | .10 |
Note. Congenital heart disease (n = 80); healthy controls (n = 76). ACI = Attention/Concentration Index; CHD = congenital heart disease; DM= design memory; FW= finger window; GMI = general memory index; MoCA = Montreal Cognitive Assessment; NL = number/letters; PM= picture memory; SM= sentence memory; StM = story memory; VL= verbal learning; VMI = verbal memory index; ViMI = visual memory index; VSE = Visuospatial/executive; WM= working memory; WRAML2 = Wide Range Assessment of Memory and Learning.
p < .05.
p < .01.
FIGURE 1.
Relationship between Montreal Cognitive Assessment (MoCA) and Wide Range Assessment of Memory and Learning 2, General Memory Index (WRAML2 GMI) scores in youth with congenital heart disease and healthy controls. The Spearman rank order correlation was high in both groups (rS = .77, p < .001).
Cronbach’s alpha between the seven MoCA subtests for the CHD was .80 (range: .74 to .82) and the healthy controls was .72 (range: .68 to .74), suggesting adequate reliabilities in both subsamples. For the clinical utility of the MoCA, on average, adolescents and young adults with CHD took 14 minutes and healthy controls took 10 minutes to complete compared to the WRAML2, which took one hour and 15 minutes for both groups.
Detection of Cognitive Impairment
The validity of the MoCA test scores was examined in comparison to the GMI for the CHD group (Table 3). The MoCA screening cutoff of 26 showed good psychometric properties (sensitivity = .94, specificity = .80, PPV = .70, NPV = .96). However, the cutoff score of 25 showed similar sensitivity (.94), specificity (.80), and PPV (.70) but slightly lower NPV.
TABLE 3. Detection of Cognitive Impairment Using Various Montreal Cognitive Assessment Scores Compared to Wide Range Assessment of Memory and Learning (Second Edition) Scores in the CHD Group.
| Cutpointa | Sensitivity | Specificity | PPV | NPV |
|---|---|---|---|---|
| 15 | 0.03 | 1.00 | 1.00 | 0.70 |
| 16 | 0.06 | 1.00 | 1.00 | 0.71 |
| 17 | 0.15 | 1.00 | 1.00 | 0.72 |
| 18 | 0.18 | 1.00 | 1.00 | 0.73 |
| 19 | 0.25 | 1.00 | 1.00 | 0.75 |
| 20 | 0.31 | 0.99 | 0.94 | 0.76 |
| 21 | 0.39 | 0.98 | 0.90 | 0.78 |
| 22 | 0.48 | 0.95 | 0.82 | 0.80 |
| 23 | 0.71 | 0.88 | 0.74 | 0.88 |
| 24 | 0.85 | 0.84 | 0.71 | 0.93 |
| 25 | 0.94 | 0.80 | 0.70 | 0.95 |
| 26 | 0.94 | 0.80 | 0.70 | 0.96 |
| 27 | 0.94 | 0.51 | 0.55 | 0.97 |
| 28 | 0.99 | 0.38 | 0.42 | 0.98 |
| 29 | 1.00 | 0.18 | 0.37 | 1.00 |
| 30 | 1.00 | 0.14 | 0.25 | 1.00 |
Note. N = 80. The MoCA cutpoint of 26 showed maximal combined sensitivity and specificity against GMI ≤ 85. MoCA = Montreal Cognitive Assessment; NPV = negative predictive value; PPV = positive predictive value.
MoCA scores.
Using the MoCA cutpoint of 26 and WRAML2 cutpoint of 85, sensitivity in the CHD group was .96 (95% CI [0.85, 0.99]) compared to 0.75 in the healthy controls (95% CI [0.19, 0.99]). Specificity in the CHD group was 0.67 (95% CI [0.19, 0.79]) compared to 0.90 (955 CI [0.70, 1.00]) in the healthy controls. In the CHD group, the MoCA had higher sensitivity than specificity whereas the healthy controls had lower sensitivity than specificity. The receiver-operating characteristics (Figure 2) disclose diagnostic accuracy of the MoCA in assessing cognitive deficits with an AUC of 0.84 (95% CI [0.75, 0.93]; p < .001) in the CHD and AUC of 0.84 (95% CI [0.62, 1.00]; p = .02) in the healthy controls.
FIGURE 2.
Receiver-operating characteristic curves disclose diagnostic accuracy of various cut points on the MoCA compared to the GMI cut point of ≤ 85 for youth with CHD (Panel A) and healthy controls (Panel B). In the CHD group, AUC = .84; p <.001 and in the healthy control group, AUC = .84; p = .02).
We also performed a post hoc analysis using G*Power version 3.1.2 in order to obtain estimates on effect size for the relationships between the two measures (Faul, Erdfelder, Lang, & Bucher, 2007). When the total sample size was considered (N =156), a very large effect size was identified (0.73) with a power of 1.00 on a two-tailed Pearson correlation (parametric), as well as on Spearman’s rho (nonparametric) statistical tests. For the individual groups, the CHD group (n = 80) effect size is 0.70 with a power of 1.00 and for the healthy controls (n = 76) effect size was 0.46 with a power of 0.99 for these same statistical tests. These power analyses verify that we had sufficient sample size for this study.
Discussion
Our results are the first to demonstrate content, concurrent, and construct validities of the MoCA as an appropriate screening tool for assessing mild cognitive deficits in middle to late adolescents (≥ 14 years of age) and young adults with CHD. MoCA scores demonstrated strong psychometric properties with good concurrent validity, high sensitivity, specificity, negative and positive predictive values, and validity in detecting cognitive deficits in adolescents and young adults with CHD and healthy controls, and adequate reliability. Despite no other studies in the literature for comparison, the screening cutoffs of 25 and 26 were similar with other studies performed on the MoCA compared to other cognitive screeners used in adult populations with vascular impairment (Godefroy et al., 2011; Wong et al., 2013). In our study, the MoCA’s maximum sensitivity and specificity cutoff was a score of 26 for the CHD group—which is the established instrument cutoff. However, a score of 25 had the same sensitivity and specificity as a score of 26, but had slightly less negative predictive value. Adolescents in this study typically had not completed 12 years of education and received an additional one point added to their total scores as adults per the instrument’s instruction. One other study made the same adjustment in adolescent participant scores with less than 12 years of education (Phabphal & Kanjanasatien, 2011). The similarity between cutoffs of 25 and 26 may be reflective of the greater-than-expected impact of less education, as well as life experiences of adolescents in comparison to the adult subjects usually studied using the MoCA.
Our analysis also revealed a number of modest to strong correlations between the MoCA and WRAML2 GMI and other additional subtests. However, some correlations lacked specificity (e.g., MoCA visuospatial/executive function correlated significantly with all tasks of the GMI). Others have speculated that this is caused by summing individual items (i.e., trail B, cube, and clock) into this subtest that are independently very sensitive (Bezdicek et al., 2013) and was also verified by item analysis in a large cohort (Damian et al., 2011). Conversely, there was very low to no correlations between MoCA naming and orientation subtests compared to the WRAML2 GMI and additional subtests. This could represent that the GMI or other subtests of the WRAML2 do not formally measure the naming task or could represent the “ceiling effect” (e.g. median of MoCA orientation was 6 out of 6 points) which is consistent with other studies on the naming and orientation tasks of MoCA in adult populations (Bezdicek et al., 2013). Cognitive screening studies in children and adolescents show that by early adolescence, there is a trend toward reaching the maximum or upper score limits (Bornholt, Ajersch, Fisher, Markham, & Ouvier, 2010). Thus, more adult cognitive testing screening tools may be suitable for use in middle- to late-adolescent age groups as demonstrated in our study.
Limitations
Limitations to this study included that our sample was very homogeneous in regard to age (range: 14-21) and severity of CHD (primarily moderate to severe CHD). However, there were few participants with simple forms of CHD to assess diagnostic capability based on CHD severity. Some healthy controls were recruited by referral from CHD participants to minimize differences in environmental factors that could impact cognitive measures. We lacked a suitable comparison in WRAML2 for the MoCA subtest naming. The WRAML2 was chosen to match the majority of MoCA subscales and to match with the cognitive resources of the CHD participants in light of mental fatigue and validity of a much longer neurocognitive battery of tests. Lastly, we also cannot exclude the possible effect of attention deficit disorders, anxiety, and depression on cognitive function in participants with CHD.
Conclusion
Our study is the first study to validate the use of the MoCA in the early- to late-adolescent age groups. Our findings show that compared to the WRAML2 GMI score, the MoCA has good sensitivity, specificity, PPV, and NPV at the diagnostic cutoff of 26. However, similar sensitivity, specificity, and PPV was identified at the cutoff of 25 with slightly lower NPV in assessing adolescents without cognitive deficits. This study demonstrates the MoCA as a valid screening tool based on its strong correlations with the WRAML2 to assess cognitive deficits and, if warranted, provides objective data to refer for more formal cognitive testing in adolescents and young adults with CHD. However, future studies on the MoCA in other cognitively-impaired adolescent populations is imperative to provide further validity information.
Acknowledgments
The authors acknowledge this study was supported by grant 1R01NR013930 from the National Institutes of Health.
Footnotes
The authors have no conflicts of interest to report.
Contributor Information
Nancy A. Pike, School of Nursing, University of California, Los Angeles; Division of Cardiothoracic Surgery, Children’s Hospital Los Angeles.
Marie K. Poulsen, Division of General Pediatrics, Children’s Hospital Los Angeles.
Mary A. Woo, School of Nursing, University of California, Los Angeles.
References
- Ballweg JA, Wernovsky G, Gaynor JW. Neurodevelopmental outcomes following congenital heart surgery. Pediatric Cardiology. 2007;28:126–133. doi: 10.1007/s00246-006-1450-9. doi:10.1007/s00246-006-1450-9. [DOI] [PubMed] [Google Scholar]
- Bellinger DC, Rivkin MJ, DeMaso D, Robertson RL, Stopp C, Dunbar-Masterson C, Newburger JW. Adolescent with tetralogy of Fallot: Neuropsychological assessment and structural brain imaging. Cardiology in the Young. 2015;25:338–347. doi: 10.1017/S1047951114000031. doi:10.1017/S1047951114000031. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Bellinger DC, Wypij D, Rivkin MJ, DeMaso DR, Robertson RL, Dunbar-Masterson C, Newburger JW. Adolescents with d-transposition of the great arteries corrected with the arterial switch procedure: Neuropsychological assessment and structural brain imaging. Circulation. 2011;124:1361–1369. doi: 10.1161/CIRCULATIONAHA.111.026963. doi:10.1161/CIRCULATIONAHA.111.026963. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Bezdicek O, Majerova V, Novak M, Nikolai T, Ruzicka E, Roth J. Validity of the Montreal Cognitive Assessment in the detection of cognitive dysfunction in Huntington’s disease. Applied Neuropsychology: Adult. 2013;20:33–40. doi: 10.1080/09084282.2012.670158. doi:10.1080/09084282.2012.670158. [DOI] [PubMed] [Google Scholar]
- Bornholt LJ, Ajersch S, Fisher IH, Markham RH, Ouvier RA. Cognitive screening for children and adolescents: General limits or ceiling effects? Journal of Child Neurology. 2010;25:567–571. doi: 10.1177/0883073809352686. doi:10.1177/0883073809352686. [DOI] [PubMed] [Google Scholar]
- Cameron J, Worrall-Carter L, Page K, Stewart S, Ski CF. Screening for mild cognitive impairment in patients with heart failure: Montreal Cognitive Assessment versus Mini Mental State Exam. European Journal of Cardiovascular Nursing. 2013;12:252–260. doi: 10.1177/1474515111435606. doi:10.1177/1474515111435606. [DOI] [PubMed] [Google Scholar]
- Cassidy AR, White MT, DeMaso DR, Newburger JW, Bellinger DC. Executive function in children and adolescents with critical cyanotic congenital heart disease. Journal of the International Neuropsychological Society. 2015;21:34–49. doi: 10.1017/S1355617714001027. doi:10.1017/S1355617714001027. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Damian AM, Jacobson SA, Hentz JG, Belden CM, Shill HA, Sabbagh MN, Adler CH. The Montreal Cognitive Assessment and the Mini-Mental State Examination as screening instruments for cognitive impairment: Item analyses and threshold scores. Dementia and Geriatric Cognitive Disorders. 2011;31:126–131. doi: 10.1159/000323867. doi:10.1159/000323867. [DOI] [PubMed] [Google Scholar]
- Dominguez TE, Wernovsky G, Gaynor JW. Cause and prevention of central nervous system injury in neonates undergoing cardiac surgery. Seminars in Thoracic and Cardiovascular Surgery. 2007;19:269–277. doi: 10.1053/j.semtcvs.2007.07.005. doi:10.1053/j.semtcvs.2007.07.005. [DOI] [PubMed] [Google Scholar]
- Faul F, Erdfelder E, Lang A-G, Buchner A. G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods. 2007;39:175–191. doi: 10.3758/bf03193146. doi:10.3758/BF03193146. [DOI] [PubMed] [Google Scholar]
- Gaynor JW, Stopp C, Wypij D, Andropoulos DB, Atallah J, Atz AM, Goldberg CS, International Cardiac Collaborative on Neurodevelopment (ICCON) Investigators Neurodevelopmental outcomes after cardiac surgery in infancy. Pediatrics. 2015;135:816–825. doi: 10.1542/peds.2014-3825. doi:10.1542/peds.2014-3825. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Godefroy O, Fickl A, Roussel M, Auribault C, Bugnicourt JM, Lamy C, Petitnicolas G. Is the Montreal Cognitive Assessment superior to the Mini-Mental State Examination to detect poststroke cognitive impairment? A study with neuropsychological evaluation. Stroke. 2011;42:1712–1716. doi: 10.1161/STROKEAHA.110.606277. doi:10.1161/STROKEAHA.110.606277. [DOI] [PubMed] [Google Scholar]
- Kahr PC, Radke RM, Orwat S, Baumgartner H, Diller G-P. Analysis of association between congenital heart defect complexity and health-related quality of life using a meta-analytic strategy. International Journal of Cardiology. 2015;199:197–203. doi: 10.1016/j.ijcard.2015.07.045. doi:10.1016/j.ijcard.2015.07.045. [DOI] [PubMed] [Google Scholar]
- Markowitz SD, Ichord RN, Wernovsky G, Gaynor JW, Nicolson SC. Surrogate markers for neurological outcome in children after deep hypothermic circulatory arrest. Seminars in Cardiothoracic and Vascular Anesthesia. 2007;11:59–65. doi: 10.1177/1089253206297481. doi:10.1177/1089253206297481. [DOI] [PubMed] [Google Scholar]
- Marino BS, Lipkin PH, Newburger JW, Peacock G, Gerdes M, Gaynor JW, Mahle WT. Neurodevelopmental outcomes in children with congenital heart disease: Evaluation and management: A scientific statement from the American Heart Association. Circulation. 2012;126:1143–1172. doi: 10.1161/CIR.0b013e318265ee8a. doi:10.1161/CIR.0b013e318265ee8a. [DOI] [PubMed] [Google Scholar]
- McCabe N, Dunbar SB, Butler J, Higgins M, Book W, Reilly C. Antecedents of self-care in adults with congenital heart defects. International Journal of Cardiology. 2015;201:610–615. doi: 10.1016/j.ijcard.2015.08.125. doi:10.1016/j.ijcard.2015.08.125. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Mittal S, Verma P, Jain N, Khatter S, Juyal A. Gender based variation in cognitive functions in adolescent subjects. Annals of Neurosciences. 2012;19:165–168. doi: 10.5214/ans.0972.7531.190406. doi:10.5214/ans.0972.7531.190406. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Murphy LK, Compas BE, Reeslund KL, Gindville MC, Mah ML, Markham LW, Jordan LC. Cognitive and attentional functioning in adolescents and young adults with tetralogy of Fallot and d-transposition of the great arteries. Child Neuropsychology. 2015 doi: 10.1080/09297049.2015.1087488. Advance online publication. doi:10.1080/09297049.2015.1087488. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Nasreddine ZS, Phillips NA, Bédirian V, Charbonneau S, Whitehead V, Collin I, Chertkow H. The Montreal Cognitive Assessment, MoCA: A brief screening tool for mild cognitive impairment. Journal of the American Geriatric Society. 2005;53:695–699. doi: 10.1111/j.1532-5415.2005.53221.x. doi:10.1111/j.1532-5415.2005.53221.x. [DOI] [PubMed] [Google Scholar]
- Niwa K, Tateno S, Tatebe S, Fujita K, Sugita K, Terai M, Takahashi O. Social concern and independence in adults with congenital heart disease. Journal of Cardiology. 2002;39:259–266. [PubMed] [Google Scholar]
- Phabphal K, Kanjanasatien J. Montreal Cognitive Assessment in cryptogenic epilepsy patients with normal Mini-Mental State Examination scores. Epileptic Disorders. 2011;13:375–381. doi: 10.1684/epd.2011.0469. doi:10.1684/epd.2011.0469. [DOI] [PubMed] [Google Scholar]
- Pike NA, Evangelista LS, Doering LV, Eastwood J-A, Lewis AB, Child JS. Quality of life, health status, and depression: Comparison between adolescents and adults after the Fontan procedure with healthy counterparts. Journal of Cardiovascular Nursing. 2012;27:539–546. doi: 10.1097/JCN.0b013e31822ce5f6. doi:10.1097/JCN.0b013e31822ce5f6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Plichta SB, Kelvin E. Munro’s statistical methods for health care research. 6th ed. Lippincott Williams & Wilkins; Philadelphia, PA: 2012. [Google Scholar]
- Rollins CK, Watson CG, Asaro LA, Wypij D, Vajapeyam S, Bellinger DC, Rivkin MJ. White matter microstructure and cognition in adolescents with congenital heart disease. Journal of Pediatrics. 2014;165:936–944. doi: 10.1016/j.jpeds.2014.07.028. doi:10.1016/j.jpeds.2014.07.028. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Sheslow D, Adams W. Wide range assessment of memory and learning administration and technical manual. 2nd ed. Wide Range Inc.; Wilmington DE: 2003. [Google Scholar]
- Shillingford AJ, Glanzman MM, Ittenbach RF, Clancy RR, Gaynor JW, Wernovsky G. Inattention, hyperactivity, and school performance in a population of school-age children with complex congenital heart disease. Pediatrics. 2008;121:e759–e767. doi: 10.1542/peds.2007-1066. doi:10.1542/peds.2007-1066. [DOI] [PubMed] [Google Scholar]
- Simons JS, Glidden R, Sheslow D, Pizarro C. Intermediate neurodevelopmental outcome after repair of ventricular septal defect. Annals of Thoracic Surgery. 2010;90:1586–1591. doi: 10.1016/j.athoracsur.2010.06.082. doi:10.1016/j.athoracsur.2010.06.082. [DOI] [PubMed] [Google Scholar]
- Smith T, Gildeh N, Holmes C. The Montreal Cognitive Assessment: Validity and utility in a memory clinic setting. Canadian Journal of Psychiatry. 2007;52:329–332. doi: 10.1177/070674370705200508. doi:10.1177/070674370705200508. [DOI] [PubMed] [Google Scholar]
- von Rhein M, Kugler J, Liamlahi R, Knirsch W, Latal B, Kaufmann L. Persistence of visuo-constructional and executive deficits in adolescents after open-heart surgery. Research in Developmental Disabilities. 2015;36:303–310. doi: 10.1016/j.ridd.2014.10.027. doi:10.1016/j.ridd.2014.10.027. [DOI] [PubMed] [Google Scholar]
- Warnes CA, Liberthson R, Danielson GK, Dore A, Harris L, Hoffman JIE, Webb GD. Task force 1: the changing profile of congenital heart disease in adult life. Journal of the American College of Cardiology. 2001;37:1170–1175. doi: 10.1016/s0735-1097(01)01272-4. [DOI] [PubMed] [Google Scholar]
- Wernovsky G. Current insights regarding neurological and developmental abnormalities in children and young adults with complex congenital cardiac disease. Cardiology in the Young. 2006;16:92–104. doi: 10.1017/S1047951105002398. doi:10.1017/S1047951105002398. [DOI] [PubMed] [Google Scholar]
- Wong GKC, Ngai K, Lam SW, Wong A, Mok V, Poon WS. Validity of the Montreal Cognitive Assessment for traumatic brain injury patients with intracranial haemorrhage. Brain Injury. 2013;27:394–398. doi: 10.3109/02699052.2012.750746. doi:10.3109/02699052.2012.750746. [DOI] [PubMed] [Google Scholar]
- Wray J, Sensky T. Congenital heart disease and cardiac surgery in childhood: Effects on cognitive function and academic ability. Heart. 2001;85:687–691. doi: 10.1136/heart.85.6.687. doi:10.1136/heart.85.6.687. [DOI] [PMC free article] [PubMed] [Google Scholar]