Cognitive Correlates of Functional Abilities in Individuals with Mild Cognitive Impairment: Comparison of Questionnaire, Direct Observation and Performance-based Measures

Abstract

The relationship between and the cognitive correlates of several proxy measures of functional status were studied in a population with mild cognitive impairment (MCI). Participants were 51 individuals diagnosed with MCI and 51 cognitively healthy older adults (OA). Participants completed performance-based functional status tests, standardized neuropsychological tests, and performed eight activities of daily living (e.g., watered plants, filled medication dispenser) while under direct observation in a campus apartment. An informant interview about everyday functioning was also conducted. Compared to the OA control group, the MCI group performed more poorly on all proxy measures of everyday functioning. The informant-report of instrumental activities of daily living (IADL) did not correlate with the two performance-based measures; however, both the informant-report IADL and the performance-based everyday problem-solving test correlated with the direct observation measure. After controlling for age and education, cognitive predictors did not explain a significant amount of variance in the performance-based measures; however, performance on a delayed memory task was a unique predictor for the informant-report IADL, and processing speed predicted unique variance for the direct observation score. These findings indicate that differing methods for evaluating functional status are not assessing completely overlapping aspects of everyday functioning in the MCI population.

It is now well documented that individuals diagnosed with mild cognitive impairment (MCI) have subtle deficits in everyday activities that extend beyond those of cognitively healthy older adults (e.g., De Vriendt, Gorus, Cornelis, Velghe, Petrovic & Mets, 2012; Jefferson et al., 2008; Schmitter-Edgecombe, McAlister & Weakley, 2012; Yeh, Lin, Chen, Lin, Chen, & Wang, 2011). Recent research suggests that MCI patients who exhibit greater impairment completing instrumental activities of daily living (IADLs) are more at risk of progression to dementia and have a lower chance of returning to a previous higher level of function (e.g., Anstey et al., 2013; Farias, Mungas, Hinton, & Haan, 2011; Fauth, Schwartz, Tschanz, Østbye, Corcoran, & Norton, 2013; Luck, Luppa, Angermeyer, Villringer, König, & Riedel-Heller, 2011). In addition, greater functional dependence has been associated with poorer quality of life in individuals diagnosed with MCI (Ettema, Dröes, de Lange, Ooms, Mellenbergh, & Ribbe, 2005; Teng, Tassniyom, & Lu, 2012) and greater burden in MCI caregivers (Gallagher et al., 2011). Given the significant implications of poor functional status on independent living, quality of life, and conversion to dementia in individuals diagnosed with MCI, it is important to investigate the ecological validity of neuropsychological tests to accurately predict everyday functioning in the MCI population.

Common proxy measures used in research studies for assessment of real-world functioning include self-report, informant-report, and performance-based measures. Each method has distinct pros and cons. For example, while both self- and informant-report measures are inexpensive, easy to administer, and provide a reasonably accurate representation of everyday performance (given the opportunity for multiple observations in real-world situations), they are subject to reporter bias (e.g., Dassel & Schmitt, 2008; Richardson, Nadler, & Malloy, 1995). Similarly, although performance-based measures are objective and have reliable coding systems, they represent a single evaluation point with typical everyday environmental cues removed. In addition, they can fluctuate with motivation, cognition and behavior (Marson & Hebert, 2006; Myers, Holliday, Harvey, & Hutchinson, 1993; Zimmerman & Magaziner, 1994) and take time and can be expensive to administer.

Several studies that directly compared questionnaire and performance-based methods of functional assessment found that these measures do not always correlate highly with each other, and further, they can provide different estimates of an individual’s ability to perform everyday activities (e.g., Burton, Strauss, Bunce, Hunter, & Hultsch, 2009; Finlayson, Havens, Holm, & Van Denend et al., 2003; Jefferson et al., 2008; Schmitter-Edgecombe et al., 2011; Tabert et al., 2002). For example, in a sample of cognitively healthy controls and individuals with MCI, Burton et al. (2009) found low to moderate correlations (−.23 to .36) between a performance-based measure of functional abilities (i.e., Everyday Problems Test [EPT]; Willis & Marsiske, 1993) and several self- and informant-report measures. The lack of agreement between proxy measures of functional status, and the fact that there is no “gold standard” for measuring everyday functioning, further complicates efforts to evaluate the relationship between cognitive processes and everyday functioning.

It has been argued that naturalistic observation is the most valid determination of functional status (Marcotte, Scott, Kamat, & Heaton, 2010). Direct observation of behavior in an everyday environment allows for assessment of behavioral subtleties and long-term changes that may otherwise be overlooked in a proxy measure. In a prior study, our laboratory collected behavioral observation data of healthy older adults completing eight, highly-scripted everyday activities (e.g., locate and fill a medication dispenser; prepare a cup of noodle soup) within a campus apartment (Schmitter-Edgecombe et al., 2011). We found that although there was no significant relationship between a self-report measure of IADLs and a performance-based everyday problem-solving test (i.e., EPT), both measures were unique predictors of the direct observation score. It was reasoned that the questionnaire and performance-based measures of functional status might be measuring different aspects of everyday performance. More specifically, the questionnaire measure may be tapping into knowledge gained from multiple experiences completing everyday activities, while the performance-based measure may be assessing the ability to use and apply everyday problem-solving skills. Studies have also suggested that different executive/cognitive processes may be employed by questionnaire and performance-based measures, further supporting some dissociation (e.g., Cahn-Weiner, Malloy, Boyle, Marran, & Salloway, 2002; Vaughan & Giovanello, 2010).

To further understanding of the associations between proxy measures of functional status in a cognitively compromised older adult population, we examined relationships between informant-report, performance-based measures, and a direct behavioral observation measure of everyday functioning in individuals diagnosed with MCI. No study that we are aware of as directly compared these multiple measures assessing functional status in an MCI population. We used an informant-report measure, as prior research has shown that informant-report generally shows better relationships with objective cognitive decline than self-report (e.g., Miller, Brown, Mitchell, & Williamson, 2013; Mitchell et al., 2011; Tsang, Diamond, Mowszowski, Lewis, & Naismith, 2012), especially in populations that may lack adequate awareness of functional deficits (Tabert et al., 2002). The performance-based tests used in this study included: (1) a paper-and-pencil task that assessed everyday problem-solving using cognitively challenging real-world problems (i.e., EPT; Willis & Marsiske, 1993), and (2) a behavioral simulation task that required individuals to complete everyday tasks within a laboratory environment [e.g. write a check; Revised Observed Tasks of Daily Living (OTDL-R), Diehl, Marsiske, Horgas, Rosenberg, Saczynski, & Willis, 2005]. In addition, participants completed eight scripted everyday activities in a campus apartment while under direct observation. Although the campus apartment provided a unique opportunity to collect behavioral observation data within a naturalistic environment, repeated measurements of the same everyday activities were not obtained, nor were participants tested in their own homes.

As a second goal, we sought to further investigate the ecological validity of neuropsychological tests for predicting everyday functioning in an MCI population. Research suggests that many cognitive factors contribute to functional impairment in older adult populations, including global cognitive functioning (Farias, Harrell, Neumann, & Houtz, 2003; Inzarti & Basile, 2003), memory (Farias, Mungas, Reed, Harvey, Cahn-Weiner, & DeCarli, 2006; McCue, Rogers, & Goldstein, 1990), processing speed (Tuokko, Morris, & Ebert, 2005; Teng, Becker, Woo, Knopman, Cummings, & Lu, 2010), visuoperceptual abilities (Glosser, Gallo, Duda, de Vries, Clark, & Grossman, 2002; Jefferson, Barakat, Giovannetti, Paul, & Glosser, 2006), and executive functioning (Bell-McGinty, Podell, Franzen, Baird, & Williams, 2002; Cahn-Weiner et al., 2000; Lewis & Miller, 2007). Within the older adult population, executive functioning (e.g., Mariani et al., 2008; Rapp & Reischies, 2005; Cahn-Weiner et al., 2000) and memory (e.g., Farias et al., 2006; Jefferson et al., 2008; Koehler et al., 2011; Teng et al., 2010) have been identified as the two cognitive domains most consistently related to everyday functioning. Recent work suggests that although executive abilities may be closely related to complex everyday task performance (e.g., multi-tasking) in healthy older adults (e.g., McAlister & Schmitter-Edgecombe, 2013), memory may play a more significant role in the MCI population (e.g., Schmitter-Edgecombe et al., 2012).

In a prior study conducted with healthy older adults (Schmitter-Edgecombe et al., 2011), our laboratory found that cognitive predictors representing five cognitive domains (i.e., general cognition, memory, speeded processing, executive functioning, and visuoperception) did not explain significant variance in a self-report measure of IADLs, a performance-based everyday problem-solving measure (i.e., EPT), or a direct observation measure of functional status. However, when cognitive functioning is intact, variability in cognitive functioning may carry less weight in predicting functional independence (Bombin et al., 2012). Therefore, it is important to understand whether cognitive variables can better predict functional performance deficits in a more cognitively compromised population, as well as to identify the domains of cognitive abilities that best predict everyday functioning. Answers to these questions have implications for the development of interventions to aid with everyday functioning and for understanding the ecological validity of neuropsychological tests for predicting real-world performances.

Consistent with prior research, we hypothesized that individuals diagnosed with MCI would exhibit poorer performance than healthy older adult controls (OAs) across all proxy measures of functional status, indicating difficulties with complex IADLs. Based on previous findings (e.g., Jefferson et al., 2008; Schmitter-Edgecombe et al., 2011), we did not expect strong correlations between the informant-report and performance-based measures, but we did expect both to correlate with the direct observation measure. Given prior research with MCI populations (e.g., Farias et al., 2006; Koehler et al., 2011), we also hypothesized that, after controlling for non-cognitive risk factors, memory would emerge as the strongest cognitive predictor of everyday functioning across proxy measures in individuals with MCI.

Method

Participants and Procedure

Participants were 51 individuals diagnosed with MCI and 51 age- and education-matched community-dwelling healthy older adults (OAs). This study was initiated as the latter part of a larger study investigating the relationship between cognition and everyday functioning in healthy OAs (Schmitter-Edgecombe et al., 2011). Data collection occurred between March, 2009 and July, 2011. Participants were recruited for a memory in older adulthood study through advertisements, community health and wellness fairs, physician referrals, referrals from local agencies working primarily with older adults, and from past studies in our laboratory. Table 1 shows the flow of individuals through the study. Initial screening of participants was conducted over the phone and included a medical interview to rule out medical exclusionary criteria (i.e., history of brain surgery, head trauma with permanent brain lesion, psychoactive substance abuse within the past year, stroke, or a known medical, neurological or psychiatric cause of cognitive dysfunction). To rule out cognitive impairment suggestive of moderate to severe dementia and hence an inability to complete the assessment, the initial screening also included the Telephone Interview of Cognitive Status (TICS; Brandt & Folstein, 2003; Ferrucci et al., 1998) and the Clinical Dementia Rating (CDR; Hughes, Berg, Danzinger, Coben, & Martin, 1982; Morris, 1993), which was conducted with the participant and an informant by a certified examiner.

Table 1.

Flow of Individuals through Study and Inclusionary and Exclusionary Criteria for the Mild Cognitive Impairment (MCI) and Healthy Older Adult Control Groups.

Assessed for eligibility with phone interview ( n = 378)

Excluded prior to testing (n = 69)

Did not meet telephone inclusion criteria (n = 54) Declined to participate (n = 7) Other reasons (e.g., scheduling difficulties) (n = 8)

Completed testing (n = 309)

Excluded following testing (n = 90)

Did not complete both testing sessions (n = 12) Met criteria for dementia (n = 36) Medical records revealed other medical exclusionary criteria (n = 9) High probability of depression (n = 4) Could not confidently be classified as either healthy OA or MCI (n = 29)

Classified as MCI (n = 51)

Inclusion Criteria for MCI Group

Self- or informant-report of subjective memory impairment for at least 6 months. Objective evidence of impairment in single or multiple cognitive domains (memory, executive, speeded processing, and/or language; see Table 2); scores falling at least 1.5 standard deviations below age-matched norms or in comparison to prior testing data. Nonfulfillment of the Diagnostic and Statistical Manual of Mental Disorders (DSM–IV) dementia criteria (American Psychiatric Association, 2000). Preserved general cognitive functions as confirmed by a score of ≥ 27 on the Telephone Interview of Cognitive Status (TICS), which is equivalent to the normality cutoff score of 24 on the Mini Mental Status Exam (Measso et al., 1993; Ferrucci et al., 1998). No significant impact of cognitive deficits on daily activity completion, as confirmed by a total Clinical Dementia Rating (CDR) score of no greater than 0.5, which is consistent with minimal change in habits. Absence of severe depression; scored < 10 on the Geriatric Depression Scale (GDS-15; Sheikh & Yesavage, 1986), which is almost always indicative of depression.

Classified as Healthy Older Adult (n = 168)

Current sample (n = 51; best demographic matches to MCI participants)

Inclusion Criteria for Healthy Older Adult Controls

No report of cognitive changes. Within normal limits on the TICS, taking age and education into consideration (≥27). Scored a 0 on the CDR. Scored < 10 on the GDS-15.

Participants who met initial screening criteria (i.e., no medical rule-outs, TICS score > 20, and CDR < 2) were invited to complete a battery of standardized and experimental neuropsychological tests in a laboratory. This evaluation was followed, typically one week later, by assessment of participants’ ability to complete complex activities of daily living (e.g., operate a DVD, water plants) in an apartment located on the Washington State University (WSU) campus. Each testing session lasted approximately 3 hours and was usually completed by different examiners. Scoring of all data occurred after the second testing session.

Interview, neuropsychological testing, and collateral medical information (e.g., results of laboratory and brain imaging data when available) were carefully evaluated by two experienced neuropsychologists to determine whether each participant met clinical criteria for MCI. The functional outcome measures evaluated in this study were not used in clinical diagnosis. Inclusion criteria for MCI were consistent with the diagnostic criteria defined by Petersen and colleagues (Petersen et al., 2001; Petersen & Morris, 2005) and are detailed in Table 1. The majority of MCI participants met criteria for amnestic MCI (n = 45). Participants diagnosed with both single-domain MCI (n = 19) and multi-domain MCI (n = 32; attention and speeded processing, memory, language, and/or executive functioning deficits) are represented in this sample.

Each MCI participant was closely matched with a healthy older adult participant in terms of age and education (see Table 4). There was no significant difference in the gender distribution of the two groups, χ²(1) = 1.00, p = .32 (percent female: MCI = 52.9%, OAs = 62.7%). The OA control participants reported no history of cognitive change, had a CDR score of 0, a TICS score ≥ 27, and a GDS-15 < 10. Because this study was conducted as part of a larger study (Schmitter-Edgecombe et al., 2011), the control group represented a subsample of the 168 healthy OAs who best demographically matched the MCI participants.

Table 4.

Demographic Data and Mean Summary Data for Healthy Older Adult and Mild Cognitive Impairment Groups

	Group
	Healthy OA (n=51)		MCI (n=51)
Variable or Test	M	SD	M	SD	t-test	Cohen’s d
Demographics
Age	70.94	8.10	70.98	8.42	.02	.01
Education (years)	15.39	2.70	15.16	3.12	−.41	.08
GDS-15	1.44‡	1.66	2.63†	2.26	2.96*	.60
Global cognitive status
Shipley total score	35.78	2.69	33.26‡	3.98	−3.74**	.74
TICS total score	34.00‡	2.69	32.54§	3.13	−.25*	.66
Attention/Processing Speed
SDMT Oral total	55.84‡	10.37	42.51	11.01	−6.26**	1.25
SDMT Written total	48.52‡	9.03	38.59	9.84	−5.28**	1.05
Verbal Memory
MAS list learning	60.61	5.60	48.71	10.08	−7.37**	1.50
MAS Delayed Recall†	11.35	.84	8.24	3.65	−5.94**	1.17
Executive Skills
D-KEFS Letter Fluency	42.76	9.65	34.22	14.50	−3.50**	.69
CLOX 1	13.00	1.70	12.04	2.29	−2.38*	.48
Visuoperception
CLOX 2 (copy)	13.82	1.21	13.24	1.57	−2.12*	.41

Note. Unless otherwise indicated, mean scores are raw scores. Norm sources for the cognitive tests are in parentheses following the test. GDS-15 = Geriatric Depression Scale; TICS = Telephone Interview for Cognitive Status (Brandt & Folstein, 2003); SDMT = Symbol Digit Modalities Test (Smith, 1991); MAS = Memory Assessment Scale (Williams, 1991); D-KEFS = Delis-Kaplan Executive Function System (Delis, Kaplan, & Kramer, 2001); CLOX = Clock Drawing Executive Test (Royall, Codes, & Polk, 1998).

^*p < .05;

^**p < .001.

^†n = **;

^‡n = 50;

^§n = 48

In return for their time, all participants were given a report documenting their performance on the neuropsychological tests, as well as pre-paid parking passes. Participants who traveled to the laboratory from outside Whitman or Latah counties were also provided travel reimbursement. This protocol was reviewed and approved by the Institutional Review Board at WSU.

Measures

Non-Cognitive Risk Factors

Non-cognitive risk factors included participants’ age, gender, education level, and endorsement of depressive symptoms. The GDS-15 was used as the measure of depressive symptomology (total number of endorsed symptoms).

Cognitive Variables

The cognitive predictor variables were derived from performances on standardized neuropsychological tests during a laboratory assessment. A brief description of each task, as well as the score used for each cognitive predictor, can be found in Table 2. The cognitive predictors represent five different cognitive domains that were found to predict functional status in prior studies (e.g., Farias et al., 2006; Mariani et al., 2008), including global cognitive status, processing speed, memory, visuoperceptual abilities, and executive functioning. Although the chosen cognitive predictors (i.e., neuropsychological tests) are not “domain pure”, they are most commonly associated with the identified cognitive domain. In addition, for comparison purposes, the cognitive predictors were identical to those used in a prior study with healthy OAs that examined cognitive correlates and functional abilities from middle-age through the old-old (Schmitter-Edgecombe et al.; 2011).

Table 2.

Description of Neuropsychological Tests.

Neuropsychological Tests
Test Description	Cognitive Variable for Analysis
Telephone Interview for Cognitive Status (TICS; Brandt & Folstein, 2003): Brief cognitive screening of basic cognition, including attention, orientation, memory, and conceptualization.	The total score was used as a measure of global cognitive functioning.
Symbol Digits Modalities Test (SDMT; Smith, 1991): This measure of processing speed requires use of a “code” of numbers and symbols where participants provide the corresponding numbers to given symbols (either orally or written) as quickly as possible.	The total score (number of correct responses) was used as the measure of processing speed. The oral version was used to reduce health-related effects (e.g., arthritis) on timed performance.
Memory Assessment Scale (MAS) List Learning subtest (Williams, 1991): This test of list learning and memory requires participants to recall a 12-item word list that is dictated by the examiner. Participants repeat this process up to 6 times to learn the list. Long delayed recall for the list is conducted approximately 20-minutes after the learning trial.	The long-delay recall score (i.e. total number of words recalled after 20 minutes) was used as the memory measure.
CLOX 1&2 (Royall et al., 1998): Participants draw a clock with the hands set to a given time (CLOX 1) and then draw a copy of the same clock (CLOX 2).	Total score (based on Royall et al., 1998 criteria) on CLOX 2 (copy version) was used for the measure of visuoperceptual abilities.
D-KEFS Letter Fluency subtest (Delis, Kaplan & Kramer, 2001): Participants are given a letter of the alphabet (F, A, and S) and they must provide as many words that begin with that letter in 60 seconds.	The combined total of words produced for letters F, A, and S was used as the measure of executive functioning.

Functional Status Measures

The IADL domains assessed by the informant-report, performance-based, and direct observation measures were largely consistent (e.g., medication management, meal preparation). For the purposes of this study, we defined these proxy measures of real-world functioning as “functional status measures.”

Knowledgeable Informant Report about Instrumental Activities of Daily Living (IADLs)

A knowledgeable informant completed a 50-question interview about the participant’s ability to complete everyday activities (Schmitter-Edgecombe et al., 2012). The questions captured the following ten IADL domains: using the phone, traveling, shopping, meal preparation, household tasks, conversation, organization, social functioning, medication management, and financial management. Each IADL question was rated using a Likert scale, ranging from ‘1’ (independent, as well as ever, no aid) to ‘8’ (not able to complete activity anymore). Answer choices for “no basis for judgment” or “does not need to complete the activity” were also options. A total score was derived as the average response of the answered questions. Forty-six informants of MCI participants and 38 informants of OA controls completed the interview (5 MCI participants and 13 OAs did not have informant-report data for evaluation); 61% of the informants were spouses/partners, 26% were children, 7% were friends, 6% were niece/nephew, sibling, or grandchild.

The Revised Observed Test of Daily Living

The OTDL-R (Diehl et al., 2005) is a performance-based measure of everyday activities. Participants are provided with tangible items (e.g., medication bottles, utility bill and a blank check) and presented with an everyday scenario (e.g., preparing medications, paying a bill). Participants then carry out the necessary steps to complete the task, as well as answer questions about the items provided. The OTDL-R is comprised of nine IADL tasks, categorized into three IADL domains: medication management, using the telephone and managing finances. These tasks are further delineated into 28 total steps for the nine tasks (maximum score = 28).

The Everyday Problems Test

The EPT (Willis & Marsiske, 1993) is a paper-and pencil task of everyday knowledge and competency. On this task, participants solve problems that simulate daily experiences (e.g., navigating a page in the phonebook, reading a recipe). Six multiple-choice questions were presented for each domain. The participant uses the provided stimuli to answer the questions. The EPT is comprised of problems from seven IADL domains; however, for the purpose of this study four domains (i.e., shopping, transportation, household, and meal preparation) were assessed, as the remaining three domains were measured by the OTDL-R. Correct responses were summed for a total EPT score (maximum score = 24).

Direct Observation of Everyday Activity Completion

While under the direct observation of two experimenters, participants completed eight scripted IADLs in an apartment on the WSU campus. The apartment consisted of two floors: the bottom floor was used for activity completion and consisted of a kitchen, dining room, and living room. Experimenters watched participants through live-feed video from the upstairs of the apartment and could communicate instructions through an intercom system. Participants were asked to retrieve the appropriate materials for each task (e.g., from marked cupboards or cabinets) and return the items to their original locations after task completion. The eight activities included: sweeping kitchen and dusting living and dining room, filling a medication dispenser, writing a birthday card with birthday check and addressing envelope, operating a DVD, watering plants, answering and conversing on phone about DVD, preparing a cup of noodle soup, and selecting an outfit for a job interview.

Similar to typical neuropsychological testing procedures, prior to completing each activity, participants were given brief verbal instructions about the upcoming task, as well as the locations of needed materials. The description and required steps for the sweeping and dusting task are provided in Appendix A (full task descriptions for all eight tasks can be found in Schmitter-Edgecombe et al., 2011). The sequence and accuracy of the steps to complete the task were coded by the experimenters during observation. Activities were coded for the following six error types:

Appendix A.

Description of sweeping and dusting activity with steps for accurate task completion.

Description of Activity

Activity Completion Steps

Sweep and Dust: The participant sweeps the kitchen floor and dusts the dining and the living room using the supplies located in the kitchen closet.

Participant retrieves broom from supply closet Participant retrieves duster from supply closet Participant retrieves dust pan and brush from closet Participant sweeps kitchen floor Participant uses dust pan and brush Participant dusts living room Participant dusts dining room Participant returns broom to supply closet . Participant returns duster to supply closet returns dust pan and brush to supply closet

1. Critical Omission: Coded when a step or subtask necessary for accurate activity completion was not performed (e.g., failed to put check into envelope in birthday card task).

2. Critical Substitution: Coded when an alternate object, or a correct object but an incorrect gesture, was used and disrupted accurate activity completion (e.g., dusted kitchen instead of dining room in sweeping and dusting task).

3. Non-Critical Omission: Coded when a step or subtask was not performed but the activity was still completed accurately (e.g., failed to turn off television at end of operating a DVD task).

4. Non-Critical Substitution: Coded when an alternate object, or a correct object but an incorrect gesture, was used but the activity was still completed accurately (e.g., used a container other than watering can to water plants).

5. Irrelevant Action: Coded when an action unrelated to the activity and unnecessary for activity completion was performed (e.g., retrieved knitting supplies in outfit selection task).

6. Inefficient Action: Coded when an action that slowed down, or compromised the efficiency of task completion was performed (e.g., made multiple trips to the dining room table in cooking task).

An overall score for each task was derived using the scoring criteria in Table 3. The eight individual task scores were then summed to produce the direct observation score, which ranged from 8 (best) to 40 (worst).

Table 3.

Coding Schema Used to Derive the Direct Observation Score

Direct Observation Score 1 = task completed without any errors 2 = task completed but with no more than 2 total of the following errors: non-critical omissions, non-critical substitutions, irrelevant actions, inefficient actions 3 = task completed but with more than 2 total of the following errors: non-critical omissions, non-critical substitutions, irrelevant actions, inefficient actions 4 = task incomplete, coded when a critical omission or substitution occurs; more than 50% of the task must be completed 5 = task incomplete, coded when a critical omission or substitution occurs; less than 50% of the task completed

Coding Reliability

The data was coded by two independent raters who were blind to participants’ cognitive status. The raters had access to the videotapes of each participant completing the eight activities. For each activity, a list of potential errors was generated and when new errors occurred they were added to the list under the appropriate error type. When discrepancies arose, the scorers discussed the error in question and resolved the discrepancy. Agreement was 93.75% for critical errors, 94.22% for noncritical errors, and 96.10% for extraneous errors (i.e., irrelevant + inefficient errors). Overall direct observation total score agreement was 96.02%.

Results

Analyses

All variables were evaluated for normality before conducting parametric statistics. Independent t-tests were used to compare the MCI and control groups on predictor and functional status variables. Effect sizes (Cohen’s d) were calculated to indicate the strength of group differences (see Tables 4 and 5). Correlational analyses were used to examine the relationships between the predictor and outcome variables. Given our interest in identifying potential overlap in functional outcome measures and controlling for non-cognitive variables that might be important in the hierarchical regression analyses, we choose to use a more liberal p-value of p < .05 as significant even with the multiple correlations. However, in all Tables multiple levels of significance for the p-values are indicated. Hierarchical regression analyses were then used to examine how well the five cognitive test variables predicted performance on each of the functional status measures. Given the small sample size (N = 51), we reduced the number of non-cognitive risk factors in the first step of the regression models by identifying those factors that correlated with the functional status measures. The cognitive predictors were then entered simultaneously in the second step of the model.

Table 5.

Mean Summary Data for the Healthy Older Adult and Mild Cognitive Impairment Groups

	Group
	Healthy OA (n=51)		MCI (n=51)
Functional Status Measure	M	SD	M	SD	t-test	Cohen’s d
Informant-report IADLa	1.22†	.31	1.68‡	.90	3.22**	.68
EPT	21.20§	2.16	19.11‡	3.63	−3.39**	.70
OTDL-R	22.92	2.75	20.20	3.29	−4.54**	.90
Direct Observationa	14.20	3.05	17.96	6.55	3.72**	.74

Notes. Mean scores are raw scores. IADL = Instrumental Activities of Daily Living; EPT = Everyday Problem-solving; OTDL-R = Revised Observed Test of Daily Living.

^**p < .01.

^†n = 36;

^‡n = 45;

^§n = 49;

^alower scores indicate better performance.

Demographic and Neuropsychological Data

Demographic and neuropsychological testing data for the MCI and OA control group is presented in Table 4. Independent t-tests revealed that the MCI group performed more poorly than the healthy OA group on tests assessing global cognitive status, attention and speeded processing, verbal memory, executive functioning, and visuoperception. Although the overall rate of reported depressive symptoms was low, the MCI group reported significantly more symptoms of depression than the healthy OA controls.

Functional Status Measures

Independent t-tests were used to compare MCI and control group performances across the four measures of functional status. As seen in Table 5, significant group differences were found for all functional status measures, ps < .01. In addition, group differences for each functional status measure remained significant when the number of depressive symptoms (i.e., GDS-15 score) was used as a covariate in ANCOVA analyses; Fs > 4.10, ps < .05. These data suggest that, relative to healthy OA controls, the MCI group experienced greater difficulty with complex everyday activities. Because this study sought to examine relationships between different measures of functional status and their cognitive correlates in a MCI population, the remaining analyses were conducted with the MCI sample only. Cognitive correlates for a larger sample of healthy OAs ranging from middle-aged to old-old was reported in an earlier study; 57% of the OA sample in this study were also part of the earlier sample (see Schmitter-Edgecombe et al., 2011).

Correlations among Functional Status Measures

Table 6 shows the correlations among the four proxy measures of functional status for the MCI group (age held constant). The informant-report of IADLs did not correlate significantly with either performance-based measure of functional status (OTDL-R and EPT; rs = − .26, ps > .05). Significant correlations emerged between the direct observation score and both the performance-based EPT (r = −.42, p < .01) and the informant-report IADL (r = −.37, p < .05), with poorer direct observation scores associated with poorer EPT scores and informant report of greater everyday IADL difficulties. The correlation between the performance-based OTDL-R and the direct observation score did not reach significance (r = −.23, p = .10). A significant correlation emerged between the two performance-based measures (r = .30, p < .05).

Table 6.

Correlation Matrix for the Four Functional Status Measures for MCI group (after controlling for Age)

	Informant IADL	EPT	OTDL-R	Direct Observation
Informant IADLa	1.00	−.26	−.26	.37*
EPT		1.00	.30*	−.42**
OTDL-R			1.00	−.23
Direct Observationa				1.00

Notes: IADL = Instrumental Activities of Daily Living; EPT = Everyday Problems Test; OTDL-R = Revised Observed Test of Daily Living.

^*p < .05;

^**p < .01;

^alower scores indicate better performance.

Regression Analyses: Cognitive Determinants

Hierarchical regression analyses were conducted to identify cognitive determinants of everyday functional status. To reduce the number of predictor variables included in the regression analyses, we first examined correlations between the four measures of functional status and the non-cognitive risk factors (i.e., age, gender, education, and depression). Correlations among the predictor and criterion variables are presented in Table 7. Because age and education correlated with some of the functional outcome measures, they were entered as non-cognitive risk factors in the first step of the regression model. The five cognitive predictors were entered simultaneously into the second step of the model to determine unique predictive value. There was no significant multi-collinearity among the five cognitive predictor variables or the non-cognitive risk factors (Variance Inflation Factors < 1.93).

Table 7.

Correlations between the Functional Status Measures and Non-Cognitive Risk Factors and Neuropsychological Testing Data for the MCI group.

	Functional Status Measures
Variables	Informant IADL	OTDL-R	EPT	Direct Observation
Age	.31	−.38*	−.38*	.53**
Gender	−.13	−.06	.28	−.07
Education	−.01	.31*	.31*	.03
GDS-15: depression	−.19	.11	.29	−.07
TICS: Global Cognitive Status	−.23	.26	.34*	−.40*
SDMT oral: Processing Speed	−.37*	.33*	.51**	−.71**
MAS delayed recall: Memory	−.49**	.31*	.23	−.29*
D-KEFS LF: Executive Functioning	.07	.01	.23	−.15
CLOX copy: Visuoperception	−.16	.21	.05	−.21

Notes. Total correct raw score was used for all neuropsychological measures. IADL = Instrumental Activities of Daily Living; EPT = Everyday Problems Test; OTDL-R = Revised Observed Test of Daily Living; GDS-15 = Geriatric Depression Scale; TICS = Telephone Interview for Cognitive Status; SDMT = Symbol Digit Modalities Test; MAS = Memory Assessment Scale; D-KEFS Letter Fluency subtest.

^*p < .05;

^**p < .005.

Analysis of the regression models (see Table 8) revealed that the cognitive predictors (variance accounted for represented by ΔR²) accounted for significant variance, above and beyond variance explained by age and education, for the informant-based measure [ΔR²= .28, ΔF(5, 35) = 3.21, p < .05; total R² = .39] and the direct observation score [ΔR²= .33, ΔF(5, 40) = 6.47, p < .001; total R² = .59]. The MAS delayed recall score emerged as a unique predictor for the informant-based measure, B = −.40, t = 2.70, p < .05, while the SDMT-oral score emerged as a unique predictor for the direct observation score, B = −.56, t = 3.98, p < .001. In contrast, the cognitive predictors did not account for significant variance, above and beyond age and education, for the performance-based OTDL-R [ΔR²= .12, ΔF(5, 40) = 1.45, p = .23; total R² = .33] and EPT [ΔR²= .16, ΔF(5, 35) = 1.72, p = .15; total R² = .37] measures. Education was a unique predictor for both the OTDL-R, B = .28, t = 2.11, p < .05, and the EPT, B = .33, t = 2.31, p < .05. There were no significant cognitive predictors for the OTDL-R, ts < 1.16, ps > .25, while the SDMT-oral score emerged as a unique predictor for the EPT, B = −.38, t = 2.04, p < .05. Measures of global cognition, executive function, and visuoperception did not significantly predict performance on any of the functional status measures for the MCI participants¹.

Table 8.

Summary of Hierarchical Regression Analyses for Variables Predicting the Everyday Functional Status Measures

	Functional Status Measures
Variables	Informant IADL	OTDL-R	EPT	Direct Observation
Model 1
Age	.32*	−.33*	−.27	.51**
Education	.08	.29*	.33*	.08
R2	.11	.21	.21	.26
F for R2	2.31	8.92**	5.22*	7.70**
Model 2
Age	.16	−.19	−.03	.21
Education	.13	.28*	.33*	.04
TICS: Global Cognitive Status	−.19	.12	.11	−.09
SDMT oral: Processing Speed	−.20	.20	.38*	−.56**
MAS delayed recall: Memory	−.40*	.15	.06	−.09
D-KEFS LF: Executive Functioning	.29	−.17	−.03	.19
Clox 2, copy: Visuoperception	−.00	.12	.05	−.15
R2 change	.28	.12	.16	.33
F for change in R2	3.21*	1.45	1.72	6.47**

Notes. Standardized Coefficients Beta reported in Table. IADL = Instrumental Activities of Daily Living; EPT = Everyday Problems Test; OTDL-R = Revised Observed Test of Daily Living; GDS-15 = Geriatric Depression Scale; TICS = Telephone Interview for Cognitive Status; SDMT = Symbol Digit Modalities Test; MAS = Memory Assessment Scale; D-KEFS Letter Fluency subtest.

^*p < .05;

^**p < .001.

Regression Analyses: Functional Status Measures

An additional regression was conducted to determine whether informant-report and performance-based measures accounted for unique variance in the direct observation score. Age and education were entered in the first step of the hierarchical regression, followed by the functional status measures (informant-report IADL, OTDL-R, EPT) in the second step. This model revealed that the proxy measures of everyday functioning accounted for significant variance above and beyond variance accounted for by age and education [ΔR²= .27, ΔF(3, 39) = 3.95, p < .05, total R² = .58]. In addition, both age, B = .28, t = 2.35, p < .05, and the EPT, B = −.48, t = −3.65, p < .01, were unique predictors of performance on the direct observation measure.

Discussion

Neuropsychological assessment data are routinely used by clinicians to predict real-world functioning. Prior studies, however, suggest that different methods of assessing everyday functional status can lead to differing conclusions (e.g., Jefferson et al., 2008; Loewenstein et al., 2001). In this study, we compared four proxy measures of real world functioning and examined their cognitive correlates in an MCI population. These methods included informant-report, performance-based problem-solving and behavioral simulations measures, and direct observation of performance in a real-world setting.

The MCI group performed more poorly than the healthy OA control group on all four proxy measures of functional status. These findings are consistent with recent research (e.g., De Vriendt et al., 2012; Yeh et al., 2011) indicating that individuals diagnosed with MCI experience more difficulties in everyday functioning compared to cognitively healthy OAs. These findings extend prior research by illustrating that the MCI participants’ everyday difficulties were evident on a performance-based measure of everyday problem-solving (i.e., EPT) and a behavioral simulation measure (i.e., OTDL-R). Furthermore, the MCI participants’ everyday difficulties were apparent to their knowledgeable informants, and were observable when they completed scripted everyday tasks under direct observation in a campus apartment.

Correlations among the functional status measures revealed that the MCI participants who performed more poorly on the performance-based behavioral simulation measure (i.e., OTDL-R) also demonstrated poorer everyday problem-solving abilities on the EPT. Of note, although the two performance-based measures significantly correlated with each other, neither of these measures correlated significantly with the informant-report of IADLs. This is consistent with previous findings of limited correlations between questionnaire and performance-based measures of functional status (e.g., Siedel, in press; Kempen, Steverink, Ormel, & Deeg, 1996; Reuben, Valle, Hays, & Siu, 1995; Schmitter-Edgecombe et al., 2011). In a recent study (Schmitter-Edgecombe et al., 2011), it was suggested that questionnaire and performance-based measures of everyday functioning may tap into different aspects of everyday functioning; that is, knowledge gained from multiple experiences in everyday situations and the individual’s application of everyday problem-solving skills, respectively. In addition, both the EPT and informant-report IADL correlated significantly with the direct-observation measure, but not with each other, which further supports a dissociation between these two types of functional status measures.

The cognitive correlates data also indicates that these different proxy methods for measuring everyday functioning are not necessarily overlapping. For example, after holding constant age and education, the five cognitive predictors did not account for significant additional variance for either performance-based measure (i.e., EPT or OTDL-R). One explanation for this finding is that the performance-based EPT and OTDL-R are indeed capturing functional skills that differ from the specific domains of cognition commonly assessed in neuropsychological evaluations (e.g., memory, executive functioning). In support of the supposition, the EPT was found to be a unique predictor of the direct observation measure in the current MCI sample; this was also demonstrated in a prior study with a healthy OA population (Schmitter-Edgecombe et al., 2011). In contrast, in both the current study and the prior study with healthy OAs (Schmitter-Edgecombe et al., 2011), the OTDL-R did not correlate with, or uniquely predict, the direct observation score. In fact, in our prior study with healthy OAs, performance on the OTDL-R was more closely related to cognitive correlates (i.e., SDMT – oral) compared to performance on other measures of functional status. Consistent with the prior study, it appears that the EPT may be a better performance-based proxy measure for real-world functioning than the OTDL-R.

After holding constant age and education level, cognitive predictors accounted for a significant amount of additional variance for both the informant-report IADL and the direct observation measure (i.e., 28% and 33%, respectively). Interestingly, the memory measure (i.e., MAS delayed list recall) emerged as a unique predictor for the informant-report IADL measure, whereas the processing speed measure (SDMT-oral) was a unique predictor for the direct observation score. Several prior studies have found that memory impairment may play a significant role in everyday IADL difficulties in the MCI population (e.g., Brown, Devanand, Liu, & Caccappolo, 2011; Farias et al., 2006; Greenaway, Duncan, Hanna, & Smith, 2012; Schmitter-Edgecombe et al., 2012). One possibility is that difficulties in everyday functioning as a result of memory impairment may be more likely to be noticed by informants relative to difficulties with slower speed of processing. This may explain the stronger relationship between the informant IADL measure and memory functioning. In contrast, the reason for the strong relationship between the direct observation score and the measure of speeded processing is not intuitively clear, as no timed component factors into the direct observation score. Speeded processing (as measured by the Digit Symbol subtest of the WAIS-R and by the Trail Making Test, Part A) has emerged as a significant predictor of everyday functioning in prior studies (e.g., Brown et al., 2011; Marshall, Rentz, Frey, Locascio, Johnson, & Sperling, 2011; Teng et al., 2010; Tuokko et al., 2005). In a recent study examining cognitive changes over 4 years in a MCI population, age, a global measure of cognitive status (i.e., MMSE), and an informant-report measure of functional impairment were the best predictors of incident dementia (Aretouli, Tsilidis, & Brandt, 2013). The authors suggested that progression from MCI to dementia may be due to the general degeneration of cognitive mechanisms. Thus, in the present study, it may be that general decline in cognitive mechanisms and brain function was being captured by the SDMT-oral, which is a sensitive measure of speeded processing and disrupted brain function. In support of this idea, it is interesting to note that the SDMT-oral measure was the only cognitive predictor that correlated significantly with all four proxy measures of functional status.

Separate regression models showed that, after holding constant age and education, both the functional status (27%) and cognitive (33%) predictors explained a significant amount of variance in the direct observation score for the MCI participants. This result contrasts with the findings from our prior study with a healthy OA population (Schmitter-Edgecombe et al., 2011). In this prior study, the functional predictors (21%), but not the cognitive predictors (7%), accounted for a significant amount of the variance in the direct observation score. This discrepancy suggests that when trying to assess for subtle changes in IADL performance (i.e., those reflective of the normal aging process), proxy measures of functional status (e.g., EPT) may be better predictors of real-world functioning relative to standardized neuropsychological measures of specific cognitive domains. However, when cognitive difficulties become more pronounced, as in the MCI population, standardized measures of cognitive status and proxy measures of functional status can predict difficulties with activities of daily living.

Regarding study limitations, the older adults in our sample were predominantly Caucasian, highly educated, and reported low rates of depressive symptomology, which may limit generalizability of findings and contrasts with other clinical and community-based samples that have been reported in the literature. In addition, the majority of participants met criteria for amnestic MCI, and the findings might differ in a primarily non-amnestic MCI sample or if different criteria were applied to diagnose the MCI population. Results of the regression analyses were also limited by the study sample size, the specific battery of neuropsychological tests administered, and the neuropsychological measures used to represent the cognitive constructs. In addition, although participants in this study were generally in good physical health, non-cognitive physical limitations that might limit everyday functioning (e.g., mobility issues) should be better assessed in future studies. Furthermore, although the eight scripted activities represented common everyday tasks, participants were not tested in their own homes. The campus apartment used in this study included a basic floor plan (i.e., kitchen, living room, dining room) common to most Western cultures. We believe the findings of this study to be generalizable to similar everyday environments, but may not generalize across different cultures or to individuals who do not complete the examined activities in their everyday environment. Finally, future studies could vary the complexity level of the everyday functional tasks or create everyday tasks that are similar in complexity level across domains. This may increase understanding of the specific everyday task domains that are failed by individuals with MCI and improve understanding of the relationship with cognitive correlates. Differences in the complexity level of the tasks across measures may also be one reason for inconsistencies between the performance-based measures and the direct-observation scores that generally assessed similar activities.

In summary, our findings indicate that proxy measures of functional status do not measure identical aspects of everyday functioning. While proxy measures of functional status (e.g., EPT, self-report IADL) may be better predictors of real-world functioning in the healthy OA population (Schmitter-Edgecombe et al., 2011), in the MCI population, measures of cognitive and functional status appear similarly useful for predicting everyday functioning. Future research should focus on development of a gold standard clinical measure of functional status and understanding the influence of cognitive and non-cognitive abilities on everyday tasks. Such advancements would allow clinicians to draw more informed conclusions about everyday functioning from assessment data.