Reliability and Validity of the Executive Interview (EXIT) and Quick EXIT among Community Dwelling Older Adults

Abstract

Objectives

To investigate the psychometric properties of the Executive Interview (EXIT) and Quick EXIT in community dwelling older adults.

Design

Secondary analysis of data obtained as part of a longitudinal study of cognitive function in late life depression. Setting: A university hospital.

Participants

Community dwelling adults (n=422), aged 59 years and older, with current or recent history of non-psychotic unipolar major depression, and never-depressed comparison subjects.

Measurements

The EXIT and other measures of executive control functions (ECF), non-executive cognitive domains and global cognitive function. We calculated Quick EXIT scores from the EXIT.

Results

The EXIT demonstrated high inter-rater reliability (Intraclass correlation coefficient = .978, F(7, 21) = 174.85, p < .001), while both the EXIT and Quick EXIT demonstrated moderate internal consistency (α= 0.66 and α = 0.68, respectively). Both tests also demonstrated acceptable convergent validity against several standard tests of ECF (r_s −.399 to .322, except for the Trail Making Test B, where r_s was .057 to .063) as well as against measures of multifactorial cognitive function (r_s −.432 to .491). However, both tests demonstrated inconsistent discriminant validity against a variety of standard non-ECF tests (r_s −.013 to .376).

Conclusions

Both the EXIT and the Quick EXIT have adequate reliability and appear to require ECF in this population. However, both the EXIT and the Quick EXIT also reflect non-ECF domains. The EXIT and Quick EXIT should be considered to be measures of global cognitive function rather than pure ECF measures. Given similar reliability and validity, the Quick EXIT should be preferred clinically as it is briefer and less burdensome than the full EXIT.

Impairments in executive control functions (ECFs), sometimes simply called “executive functions,” are common and clinically significant in older adults.(1–4) Impairment in ECF is associated with poor performance of important tasks necessary for independent, community-based living, such as dressing, grooming, managing finances, and performing other home- and occupation-related tasks. Indeed, impairment in ECF has been shown to predict the level of care required by older adults.(5–8) Furthermore, deficits in ECF are independently associated with impaired balance and mobility in older adults with stroke (9) and with decreased gait speed (10, 11) and accidental falls (12) in healthy older adults, even after accounting for medical comorbidities and functional differences, suggesting that intact ECF is strongly associated with both basic and complex functional skills.

ECF is an umbrella term encompassing numerous individual constructs, and there is no consensus on specific components.(13) Often-studied components of ECF include cognitive fluency (rapidly generating many solutions to a task); set shifting or mental flexibility (ability to switch back and forth between types of stimuli or responses); abstract reasoning; response inhibition (ability to suppress an overlearned or nearly automatic response in favor of producing a more effortful response); and task planning and sequencing (the ability to initiate and follow complex behavior patterns). Some investigators and authors also consider ECF to include working memory (the ability to hold information in one’s mind while performing mental manipulations), various behavioral attributes (such as apathy or agitation), and primitive reflexes such as grasp and snout reflexes.(14) The components of ECF are complex and interrelated; hence, evaluating ECF requires a lengthy battery of neuropsychological tests which can be a significant burden for patients or research participants.

Royall and colleagues (6) developed a single assessment tool, the Executive Interview (EXIT), as an alternative to a traditional ECF battery. The EXIT may be used in any clinical setting, can be administered and scored by trained personnel of any discipline, and requires only 15 minutes to complete, (15), making it a practical, ‘bedside’ alternative to traditional ECF tests. During initial validation testing with 40 residents of a retirement community representing a continuum of care from independent living through intermediate care and dementia care, Royall et al.(6) reported that the EXIT showed high internal consistency (Cronbach’s alpha = 0.87) and high inter-rater reliability (Pearson’s r = 0.90).

However, despite its strengths, the EXIT may have limited utility in some clinical populations. For example, while an administration time of 15 minutes is an improvement over lengthy neuropsychological batteries, even 15 minutes may be burdensome for acutely ill or easily fatigued patients. In addition, the behavioral requirements of certain items could perplex some individuals, leading to scores that may be confounded by either social desirability response bias or a misunderstanding of the examiner’s expectations, rather than indicating actual ECF impairment. For example, one item comprises the examiner suddenly and without explanation clapping his/her hands. Cognitively intact persons may be surprised by this stimulus, which may seem incongruous in a testing situation, or may be uncertain about the examiner’s intent in performing this behavior, and may respond to this stimulus by hesitating or appearing uncertain or confused. However, these types of responses are to be scored as indicating executive impairment. The Quick EXIT (16) is a short form of the EXIT developed to be less burdensome and perplexing to subjects and to have improved face validity and content validity. In a sample of 147 individuals with acquired brain injury, Larson and colleagues found that internal consistency was acceptable (α = .88) and construct validity was similar to that of the original EXIT.

Initial work establishing the EXIT’s reliability and validity was conducted in the small sample of 40 older adults described above.(6) Neither the EXIT nor the Quick EXIT have been fully validated in a large population of older adults, or against a wide range of neuropsychological tests of both ECF and other, non-ECF domains of cognitive function. Therefore, we conducted a secondary analysis involving a large sample of community dwelling older adults in order to examine the psychometric properties of the EXIT and the Quick EXIT.

Methods

Participants

Individuals were participants in a federally-funded longitudinal study of cognitive function in late-life depression [PHS K01 MH01684, R01 072947] (17, 18) conducted within the Advanced Center for Intervention and Services Research Center for the Study of Late-Life Mood Disorders at the University of Pittsburgh School of Medicine between 1996 and 2009. The protocol was approved by the Institutional Review Board of the University of Pittsburgh, and all participants provided written informed consent. Recruitment and eligibility criteria have been described in detail elsewhere.(18, 19) For this study, we analyzed data from 422 community dwelling adults, aged 59 years and older, both with current and recent history of non-psychotic unipolar major depression and never-depressed comparison subjects, using data from their baseline assessment. Participants with medical conditions that could directly affect cognitive abilities, such as traumatic brain injury, multiple sclerosis, or dementia, were excluded.(19)

Procedure

Participants were administered either a full neuropsychological test battery or smaller subset of this battery (depending on when they enrolled) that is standard for all participants in the Center’s studies. Component tests of this battery are described below (see Measures). Five neuropsychological examiners, under the supervision of a qualified, experienced neuropsychologist (MAB), administered all tests, including the EXIT.

After completing the neuropsychological test battery, an 8-subject subset of the sample participated in an EXIT inter-rater reliability study. One of the five examiners administered the EXIT to each of the 8 subjects while being videotaped. The remaining 4 examiners independently viewed the videotaped sessions and computed EXIT raw scores for each subject.

Measures

EXIT

The EXIT (6) is a 25-item screening tool that yields a single score reflecting a broad array of executive functions. Each item’s possible score ranges from 0 to a maximum of 2 points; total scores range from 0 to 50, with higher scores indicating greater ECF impairment. The items test number/letter sequencing; word and design fluency; sentence-repetition; thematic perception; memory with distraction; interference inhibition; grasp and snout reflexes; social habits; motor perseveration; finger-nose repetition; echopraxia; complex hand sequences; complex commands; counting and serial-order reversal; and automatic, utilization, and imitation behavior.

Quick EXIT

The Quick EXIT (16) is an abridged, 14-item version of the original EXIT. It was developed by omitting 11 EXIT items that fit the scale poorly, based on a Rasch analysis of item difficulty and fit. Items omitted include those testing primitive reflexes, social habit, and automatic, utilization, and imitation behaviors. It is scored identically to the EXIT, with a range of 0 to 28, with higher scores also indicating greater ECF impairment. For this analysis, we derived the Quick EXIT score from participants’ original EXIT item scores.

Convergent Validity Measures

We examined the following commonly used tests of ECF, all of which have strong, established psychometric properties in older adults: the Stroop Color-Word Interference Test,(20) the Trail Making Test,(21) (for which we isolated executive ability by calculating the ratio of seconds per connection for Part B to seconds per connection for Part A), the Wisconsin Card Sorting Test (22) perseverative errors score, the Initiation/Perseveration subscale of the Dementia Rating Scale,(23) and the Clock Drawing Test.(24) See Table 1 for a description of these instruments, and for the median and range of these tests in our sample.

Table 1.

Tests used to analyze convergent and discriminant validity.

	Instrument	Domain Measured	N	Median (IQR)
Executive Functions	EXIT (Total raw score; max 50)		422	8.0 (5.0–11.0)
	Quick EXIT (Total raw score; max28)		422	4.0 (2.0–6.0)
	Wisconsin Card Sort Test (Total perseverative errors)	Set Maintenance and Set Shifting	325	10 (5.5–14.5)
	Trail Making Test-B (Ratio of TMT-B seconds/connection to TMT-A-seconds/connection)	Divided Attention	319	2.5 (1.9–3.1)
	Stroop Color-Word Test (Ratio of Color-Word seconds/item to Color Naming seconds/item)	Response Inhibition	311	2.7 (2.2–3.2)
	Clock Drawing Test (Total raw score; max10)	Planning and Sequencing	346	9 (8.5–9.5)
	Dementia Rating Scale Initiation / Perseveration Subscale (Total raw score, max 37)	Initiation and Perseveration	422	37 (36.5–37.0)
Non-Executive Function	Speed and Capacity of Language Processing Spot-the-Word Errors (Total errors; max 60)	Verbal Memory (Recognition)	318	9 (5.4–12.6)
	Boston Naming Test (Total raw score; max 60)	Language (Visual Naming Ability)	341	56 (53–59)
	Trail Making Test A (Seconds/connection)	Psychomotor Speed	316	1.6 (1.2–2.0)
	Simple Drawings (Total raw score; max 18)	Visuospatial Impairment	400	16 (14.5–17.5)
	California Verbal Learning Test Discriminability Index (Percent of targets and distractors correctly discriminated)	Verbal Memory-Recognition	359	93 (88.5–97.5)
	Dementia Rating Scale Attention Subscale (Total raw score; max 37)	Complex Attention	422	36 (35–37)
	Finger Tapping Test (Mean number of taps in 10 seconds)	Motor Speed	297	38.6 (32.1–45.1)
Multifactorial Cognitive Function	Digit Symbol Subtest (Raw score in 90 seconds)	Emphasis on Attention, Visual Scanning, Memory	343	40.8 (33.1–48.6)
	Mini Mental State Examination (Total score; max 30)	Emphasis on Attention, Memory, Language	422	29 (28–30)

IQR = Interquartile Range

Discriminant Validity Measures

We included neuropsychological tests putatively without a significant ECF component, in order to evaluate discriminant validity. These tests included the Trail Making Test Part A (attention and processing speed),(21) the Boston Naming Test (language),(25) the Speed and Capacity of Language Processing Spot the Word task,(26) (vocabulary), the Finger Tapping Test,(21) (fine motor speed), the Attention subscale of the Dementia Rating Scale (basic attention),(23) the California Verbal Learning Test (27) discriminability index (verbal recognition memory), and the Simple Drawings Test (28) (visuospatial and construction ability). We also included two tests of multifactorial cognitive function; that is, tests whose successful completion requires the use of multiple cognitive functions. The multifactorial cognitive tests included the Mini Mental State Examination (MMSE), (29) a test that is heavily dependent upon orientation, memory, language, and attention span,(29, 30) and the Digit Symbol Subtest of the Wechsler Adult Intelligence Scale, 3^rd ed. (WAIS-III) (31), which requires motor persistence, sustained attention, response speed, and visuomotor coordination.(30) All of these tests have demonstrated reliability and validity in older adults. Please see Table 1.

Data Analysis

SPSS software version 21.0 (32) was used for all analyses. We analyzed descriptive data for the entire sample on key demographic and clinical characteristics using percentages for categorical variables, and medians with interquartile ranges (IQRs, expressed as the range between the 25^th and 75^th percentiles) for continuous variables.

We examined reliability and validity using nonparametric statistics due to the skewed distribution of neuropsychological test scores in our sample, which was expected given our focus on a sample of community dwelling older adults. We computed internal consistency and inter-rater reliability for both the EXIT and Quick EXIT. Some authors note that Cronbach’s alpha may underestimate the internal consistency of ordinal scales with fewer than 5 levels of response, (33) so we used a nonparametric alternative in which we calculated the mean Spearman’s rank order correlations between the EXIT items, then using those nonparametric correlations to calculate a nonparametric version of Cronbach’s alpha. Inter-rater reliability was analyzed using the intraclass correlation coefficient (ICC), calculated to determine consistency among raters using a two-way random effects model, to allow generalization to all possible subjects and all possible raters.

We assessed convergent and discriminant validity by estimating the magnitude of the relationship between the criterion measures described in Table 1, and both EXIT and Quick EXIT total scores, using Spearman’s correlation coefficients (r_s) We used the 95% confidence intervals to denote the likely population value of each validity coefficient.(34)

Results

The median age of participants was 73 years (IQR = 68–78); the sample was nearly 70% female, with a median 13 years of education (IQR = 12–16) and with a median score of 5 (IQR 3–8) on the 17-item Hamilton Rating Scale for Depression at the time of assessment. The largely Caucasian sample (89%) reflects the demographic characteristics of western Pennsylvania, the geographical area from which the sample was drawn.

Internal consistency for both the EXIT and Quick EXIT was moderate, α= 0.66 and α = 0.68, respectively. Inter-rater reliability of the EXIT among 4 raters was high; the ICC was .978, F(7, 21) = 174.85, p < .001.

Convergent and discriminant validity coefficients are presented in Table 2. The EXIT was moderately correlated with most tests of ECF, including the Wisconsin Card Sorting Test, the Stroop Color-Word Test, the Dementia Rating Scale Initiation/Perseveration Subscale, and the Clock Drawing Test, but was not correlated with the Trail Making Test-B. Discriminant validity tests showed the expected weak correlation between EXIT total scores and some of the non-ECF measures (Boston Naming Test, Trail Making Test-A), but unexpectedly demonstrated moderate correlations between the EXIT and other non-ECF measures, including the Simple Drawings test, the California Verbal Learning Test discriminability index, the Dementia Rating Scale Attention Subscale, Spot the Word Errors, and the Finger Tapping Test. Regarding tests of multifactorial cognitive functioning, the EXIT was moderately correlated with both the WAIS-III Digit Symbol Subtest and the MMSE.

Table 2.

Validity coefficients for EXIT and Quick EXIT vs. criterion neuropsychological tests.

	Test	EXIT rs (95% C.I.)	Quick EXIT rs (95% C.I.)
Executive Functions (Convergent Validity)	Wisconsin Card Sort Test (N = 325)	.343 (.244, .435)	.351 (.252, .442)
	Trail Making Test B (N = 319)	.057 (−.053, .165)	.063 (−.047, .171)
	Stroop Color-Word Test (N = 311)	.322 (.219, .419)	.338 (.236, .432)
	Clock Drawing Test (N = 346)	−.389 (−.474, −.296)	−.375 (−.462, −.281)
	Dementia Rating Scale Initiation / Perseveration Subscale (N = 422)	−.377 (−.456, −.293)	−.399 (−.476, −.316)
Non-Executive Functions (Discriminant Validity)	Speed and Capacity of Language Processing Spot- the-Word Errors (N = 318)	.326 (.225, .420)	.376 (.278, .493)
	Boston Naming Test (N = 341)	.081 (−.025, .185)	.044 (−.062, .149)
	Trail Making Test A (N = 316)	−.013 (−.096, .122)	.037 (−.072, .145)
	Simple Drawings (N = 400)	−.142 (−.236, −.045)	−.143 (−.237, −.046)
	California Verbal Learning Test Discriminability Index (N = 359)	−.357 (−.444, −.264)	−.369 (−.455, −.277)
	Dementia Rating Scale Attention Subscale (N = 422)	−.347 (−.428, −.261)	−.365 (−.444, −.280)
	Finger Tapping Test (N = 297)	−.337 (−.434, −.233)	−.338 (−.435, −.234)
Mulitfactorial Cognition (Convergent Validity)	Digit Symbol Subtest (N = 343)	−.432 (−.514, −.342)	−.491 (−.564, −.407)
	Mini Mental State Examination (N = 422)	−.440 (−.513, −.360)	−.465 (−.536, −.387)

r_s: Spearman’s rank order correlation

The Quick EXIT demonstrated similar convergent validity to the EXIT. In terms of discriminant validity, the Quick EXIT demonstrated the same pattern of correlations as seen between the EXIT and tests of non-ECF domains and multifactorial cognitive function.

Conclusions

In our sample of community dwelling older adults, the EXIT demonstrated high inter-rater reliability, while both the EXIT and Quick EXIT demonstrated moderate internal consistency. Both tests also demonstrated acceptable convergent validity against standard tests of ECF. However, both tests demonstrated relatively poor discriminant validity, as both tests demonstrated moderately high correlations with some measures that reflect non-ECF domains.

We demonstrated moderate internal consistency of the EXIT (α = 0.66) in our large sample of community dwelling older adults. The EXIT purports to test a variety of the component domains of ECF; hence, our results are not unexpected, and may accurately reflect the multi-dimensional nature of ECF. However, our results for the EXIT differ from those reported by other authors, including Royall’s and Larson’s groups (α = 0.87 and α = 0.86, respectively). Similarly, our finding regarding internal consistency of the Quick EXIT (α = 0.68) also differs from that reported by Larson’s group (α = 0.88). These differences may reflect differences in the sample characteristics. Our sample excluded persons with clinically definable brain pathology and therefore, likely possessed a narrower range of cognitive function, and lack of diversity of types of cognitive impairment relative to the participants in the other studies. Royall’s sample was selected to have a broad range of cognitive function (no impairment to severely impaired) and Larson’s sample included persons with acquired brain injury. Nevertheless, the internal consistency demonstrated by the EXIT and Quick EXIT in our sample is acceptable.(35, 36) We demonstrated high inter-rater reliability of the EXIT although our result was slightly lower than reported by Royall. Again, differences in the sample characteristics, with its resultant narrower range of scores in our sample of community-dwelling older adults, may have affected our results. Repeating this study in a sample that includes persons exhibiting a full range of cognitive ability, representing both persons with and without neurological diseases known to affect cognitive function (e.g. stroke, traumatic brain injury, Parkinson’s disease, multiple sclerosis) could clarify whether the EXIT or the Quick EXIT are equally useful for screening for executive impairment among persons both with and without brain pathology, and across a wide range of cognitive functioning.

Our analysis suggests that both the EXIT and the Quick EXIT are acceptable measures of ECF, based on their significant correlations with a variety of other accepted tests of ECF and multifactorial cognitive function as well as similar internal consistency. Our results regarding the EXIT are consistent with Royall’s report regarding convergent validity in that they found that the EXIT correlated strongly with similar tests of ECF. However, the ability of the EXIT and the Quick EXIT to distinguish ECF impairment from impairment in other cognitive domains in our sample was variable, at best. Discriminant validity analyses showed the expected weak (non-significant) correlation between performance on the EXIT and some of the non-ECF measures, but unexpectedly demonstrated a moderate, significant correlation between the EXIT and other non-ECF measures. This finding is consistent with other studies.(6, 16) Royall reported that EXIT scores correlated with ECF tests (Trail Making Test-B, Wisconsin Card Sorting Test) and tests that we considered to be non-ECF tests (Trail Making Test-A, sustained attention/tracking). Similarly, Larson reported that the EXIT and Quick EXIT correlated with ECF tests (Trail Making Test-B) and tests that we considered to be non-ECF tests (Trail Making Test-A, Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) subscales including attention, language, visuo-construction, immediate memory and delayed memory).

The association of the EXIT with the ‘non-ECF’ tests may reflect the innate dependence on executive functions of non-ECF tasks. For example, Larson et al. (16) found that the EXIT correlated with the RBANS immediate and delayed memory indices. As pointed out by the authors, this is not wholly surprising as these memory indices rely on retrieval ability (an executive function). However, in our study, we selected measures for discriminant validity analyses that are minimally reliant on ECFs. We used a recognition memory task that tests the person’s ability to retain (a pure memory function) but not retrieve, information. However, even when using relatively ‘pure’ non-ECF tasks, we still found significant correlations with the EXIT. Furthermore, non-ECF skills are needed to complete ECF tasks. For example, as pointed out by Larson et al., the EXIT requires strong expressive language skills and hence a significant correlation of the EXIT with language tests is not completely unexpected. We believe that this relationship explains our finding of a correlation between the EXIT and tests of fine motor speed and attention, which measure very basic cognitive functions. Further, our results echo those of Koltai et al., (37) who found similar correlations between EXIT and a variety of ECF and non-ECF cognitive tests. They suggest that poor scores on the EXIT indicate presence of a cognitive deficit with an executive component, but that the EXIT is not likely to be a specific, reliable measure of ECF alone.

Our study has several limitations. First, as noted above, our participants comprised a community sample initially chosen to reflect a range of cognitive functioning. However, a ceiling effect can be seen in our sample’s overall general cognitive function (MMSE) scores, suggesting that the abilities and skills needed to live in the community may have effectively excluded subjects exhibiting frank neurocognitive impairment. This ceiling effect may have served to reduce the level of internal consistency we observed and may have underestimated the true size of correlations with tests of other domains. We must also consider that participants in our sample may have ECF impairments due to subcortical brain dysfunction as seen in subclinical cerebrovascular disease (e.g., small vessel ischemic brain changes which are long- term consequences of common diseases such as hypertension and diabetes) and/or late-life major depression. As such, it is possible that the types of cognitive impairment seen in our sample were restricted primarily to those deficits in domains associated with subcortical structures, namely ECF, attention, and processing speed. Further research is needed to evaluate whether the EXIT or Quick EXIT can reliably identify ECF impairment in other populations, for example, individuals with a major neurocognitive disorder. A relative strength of this study is that for the discriminant validity analysis we chose tasks most likely to be independent of ECF. However, we acknowledge that there are no ‘pure’ cognitive tests; that is, performance on any given test depends on performance ability in other domains. Another strength of our study is that test scores came from each participant’s initial testing session; hence avoiding practice effects confounding test performance.

Our analysis suggests that both the EXIT and the Quick EXIT are able to detect ECF impairment in a population of community dwelling older adults without frank neurocognitive impairment. However, both the EXIT and the Quick EXIT correlated with tests of non-ECF, suggesting that they have limited utility in distinguishing ECF impairment from other types of cognitive impairment. Practically, this may have two implications. First, it may not be possible to capture ‘pure’ ECF especially using bedside measures due to the interdependency of cognitive functions on test performance. Second, the EXIT and Quick EXIT could be considered measures of multifactorial cognitive function with a large ECF component, rather than pure ECF measures. As with other multifactorial cognitive tests such as the Digit Symbol Subtest and MMSE, the EXIT and the Quick EXIT may not offer the degree of precision needed to distinguish ECF impairment from general cognitive impairment in research protocols. Yet, because the EXIT and Quick EXIT include items specific to ECF, they may be advantageous over other multifactorial tests that do not emphasize ECF.

Clinically, these bedside measures may be useful for providing a snapshot of cognitive function upon which to base further diagnostic workup and can inform care planning. For this purpose, the Quick EXIT has several advantages over the EXIT. First, at half the length of the EXIT, the Quick EXIT is minimally burdensome to both the examiner and examinee. Second, the Quick EXIT eliminates the behaviorally confusing items present in the EXIT. Finally, the Quick EXIT eliminates some EXIT items (e.g., snout reflex) that may be late signs of cognitive impairment, thus not particularly useful for care planning purposes. Given the similar reliability and validity between the two tests, the Quick EXIT should be the preferred bedside measure for clinical populations.