Skip to main content
NCBI home page
Archives of Clinical Neuropsychology logoLink to Archives of Clinical Neuropsychology
. 2023 Jul 24;39(1):51–64. doi: 10.1093/arclin/acad052

A Clinic-Based Measure of Everyday Planning Ability: The Overnight Trip Task

Katelyn D Brown 1, Maureen Schmitter-Edgecombe 2,✉
PMCID: PMC10802221  PMID: 37489707

Abstract

Objective

Traditional measures of cognitive abilities most used by neuropsychologists are generally low in ecological validity and only capture a small proportion of the variance of a person’s true functional capacity. This study evaluates the association between clinic-based performance and parallel real-world completion of an everyday planning test, the Overnight Trip Task (OTT).

Method

A cross-sectional cohort of 65 community-dwelling older adults completed a battery of cognitive assessments and two formats of the OTT: the paper-and-pencil clinic OTT, which was completed remotely through video conferencing (Zoom), and the parallel real-world OTT (RW-OTT), which was completed at home between sessions. Both formats required participants to plan for what they would pack and prepare for a hypothetical overnight trip based on a provided story that included rules and embedded contingencies.

Results

The clinic OTT demonstrated a small-to-moderate relationship with the RW-OTT (r = 0.35) and no relationship with measures of learning, long delay recall, an executive function composite, and a measure of everyday planning. Hierarchical regressions indicated that the clinic OTT demonstrated incremental validity above an executive function composite measure and global cognition when predicting self-reported everyday functioning and RW-OTT performance.

Conclusions

The clinic OTT showed only modest association with the RW-OTT and discriminate but not convergent validity was demonstrated. The clinic OTT showed incremental validity when predicting self-reported everyday functioning and RW-OTT performance above more traditional measures. Before the clinic OTT could be considered a reliable and valid clinic-based measure for predicting real-world behavior, additional research would be needed.

Keywords: Executive functions, Naturalistic assessment, Functional status, Ecological validity, Aging

Introduction

Executive functions (EF) are typically classified as higher order cognitive abilities comprising planning/problem solving abilities, organizational abilities, inhibitory control, cognitive flexibility, reasoning abilities, and working memory (Diamond, 2013). These functions are important for successful completion of everyday activities that are novel, open-ended, and/or ill-structured (Shallice and Burgess 1991). Assessing for mild to moderate EF deficits that can affect everyday functional abilities with traditional clinical methods (e.g., paper-and-pencil tasks) can be difficult. This is because most clinical measures were designed for the localization and diagnosis of brain pathology (Burgess et al., 2006; Chan et al., 2008) and not for predicting behaviors in real-world functioning (Chaytor & Schmitter-Edgecombe, 2003; Rabin et al., 2007). A critique of clinical neuropsychology is that data acquired from “office-based tests” of EF that have not demonstrated ecological validity for predicting behaviors in real-world settings are often relied upon by neuropsychologists too heavily when predicting real-world ability (Manchester et al., 2004). One challenge is creating an ecologically valid measure that can be easily and practically administered across treatment settings, yet still capture real-world performance of the same task. The study purpose is to evaluate whether a more open-ended, clinic-based paper and pencil measure of a real-world EF activity (i.e., packing for a trip) can adequately capture the real-world performance of this same everyday activity.

Most traditional EF tasks require straightforward responses to single events, neglecting the often-complex nature of everyday life activities that can require multiple steps, prioritization, forecasting, prospective memory, and so on (Chan et al., 2008). Furthermore, the premise of most traditional EF tests (e.g., Trail Making Test) do not trigger a prominent “real-world” context, which may aid performance, nor are the tasks common exercises performed in daily life. Researchers have employed several methodologies to develop more ecologically valid EF tasks, most involve improving verisimilitude or the degree to which the demands of a task more accurately reflect actual demands encountered in the real-world environment (Franzen & Wilhelm, 1996).

Shallice and Burgess (1991) developed the Multiple Errands Test (MET), one of the first function-led assessments of EF administered in a “real-world” setting. For the MET, patients are given a list of objectives and rules for completing errands in a real shopping complex and rated as they carry out the task objectives. The MET has successfully detected deficits in EF over and above traditional neuropsychological tests (Shallice & Burgess, 1991), exposed behavioral deficits contributing to poorer everyday functioning (Alderman et al., 2003), and predicted executive problems in a traumatic brain injury population (Cuberos-Urbano et al., 2013). However, the MET can take an extensive amount of time to administer, experimental control is sacrificed (Logie et al., 2011), and it requires traveling to a location and procuring a consent from the location decreasing its practicality as a clinical measure (but see newer adaptations such as the hospital version, Dawson et al., 2009). Simulated environments (e.g., kitchen, grocery store) have also been used to improve assessment of real-world EF (Chevignard et al., 2008; Robertson et al., 2018). Although tasks performed in simulated environments have greater experimental control than the MET, and have been associated with real-world functions (Schmitter-Edgecombe et al., 2012), they do require access to a simulated environment and patients confined to bed in a hospital or with significant mobility issues would not be able to participate.

The Night Out Task (NOT; Schmitter-Edgecombe et al., 2021) is an example of a complex, EF task administered in a laboratory setting using props (e.g., plastic cupboards). The NOT requires participants to prepare for a hypothetical night out by completing eight subtasks while multitasking. The NOT was found to predict real-world functioning, including over and above traditional performance-based measures (Schmitter-Edgecombe et al., 2021). Other naturalistic tasks completed with props (e.g., Naturalistic Action Test; Schwartz et al., 2003) have been found to differentiate between healthy and cognitively impaired older adults (Giovannetti et al., 2008; Lawrence et al., 2013). Although such tasks increase the portability of administration, they may still be a cumbersome format to administer in a clinical environment, require props and space to administer, and could not be administered bedside.

Function-based virtual reality tasks are another means being explored to enhance the ecological validity of EF tests (Parsons et al., 2017). As examples, the Virtual Errands Test (VET; McGeorge et al., 2001) requires participants to complete various work-related tasks in a virtual office environment, whereas the breakfast task (Craik & Bialystok, 2006; Kerr, 1991) requires participants to “start” and “stop” cooking times for various food items in a specific order based off of rules. Poorer performance by older as compared with younger adults (Craik & Bialystok, 2006) on the breakfast task was interpreted to illustrate a decline in multitasking and executive control abilities with age. Although virtual environments offer experimental control, real-time analyses of performance, and ease of administration (Parsons, 2011), virtual formats can become outdated quickly due to rapid changes in operating systems, making them costly.

Most clinicians will not be able to leave the office to examine individuals in their real-world environments or have access to simulated real-world environments in their work space. Furthermore, virtual reality programs are currently not a feasible option due to program maintenance requirements. Increasing the ecological validity of tests of EF so that they are more predictive of real-world ability is an important goal for the field of clinical neuropsychology. However, such tasks must be practical and feasible to administer in a clinical setting.

A number of easy-to-administer clinic-based EF tasks that approximate real-world activities have been developed to examine everyday planning abilities: processes involved in the formulation, evaluation, and selection of actions necessary for reaching a goal. For example, Wilson et al. (1996) created the Behavioral Assessment of the Dysexecutive Syndrome (BADS), a test battery of a variety of face valid measures of EF. Among the various measures of the BADS, the Zoo Map subtest is a purported measure of planning/problem-solving ability as it examines a participant’s ability to identify and organize the steps needed to reach a goal state (Lezak et al., 2004) by drawing a route through a maze-like zoo while following certain rules and visiting specified locations. Similarly, The UCSD Performance-based Skill Assessment-2 (UPSA-2; Patterson et al., 2001) contains a comprehension and planning subtest that requires participants to read a mock newspaper article, recall information about the article, and determine seven items they would need if they were to spend the day at the water park. Although these EF measures have been found to capture planning ability (Oosterman et al., 2013), there is little in the literature regarding the tests ability to predict real-world performance. That is, do these tasks truly capture real-world behavior, like how someone would plan and navigate through a real zoo if they were provided a set of rules to follow and locations to visit?

Mumford et al. (2001) defined planning behavior as the development of a “mental simulation of actions and their outcomes.” Although the number of cognitive processes and activities involved in planning are dependent on the problem or task complexity, planning behavior typically involves analysis of complex and dynamic environments in which an individual will act, the identification of resources and requirements, and forecasting of contingencies (Mumford et al., 2001). Consistent with this definition and based on commonly observed planning behaviors reported in the research literature (e.g., selectively attending to plan relevant information), the Overnight Trip Task (OTT) was developed as a function-lead measure of everyday planning ability.

The OTT requires participants to identify resources needed for a hypothetical trip, forecast what contingencies may influence needed resources, and imposes rules or requirements onto the framework of their plan. Two formats were created, a paper-and-pencil “clinic” OTT and a “real-world” RW-OTT. The clinic OTT was designed for practical, feasible, and portable administration in a variety of clinical settings (e.g., clinic, bedside). The RW-OTT requires participants to carry out the task (i.e., pack and prepare for the trip) using the same parameters within their home environment. If the OTT's clinic format is a successful proxy for real-world behavior, it is expected that performance on the clinic OTT will be moderately to highly associated with the RW-OTT. To test the convergent and discriminant validity of the clinic OTT, we hypothesized that the clinic OTT will have moderate to high correlations with another clinic-based proxy measure of everyday planning ability (i.e., UPSA planning), moderate correlations with an EF composite, and low to no correlations with measures of memory. Finally, we examined whether the clinic OTT would be a significant predictor of a self-report measure of everyday functioning and RW-OTT performance over and above demographics, general cognitive status, and traditional neuropsychological tests of EF.

Materials and Methods

Participants

A cross-sectional cohort of 67 community-dwelling older adults was recruited through health and wellness fairs, physician referrals, and advertisements. Two participants declined to complete the clinic OTT. The final sample consisted of 65 older adult participants with a mean age in years of 70.92 (SD = 8.07; range = 52–88), a mean education in years of 16.66 (SD = 2.58; range = 12–20), and a mean estimated verbal intellectual ability level falling above the average range (standard score = 13.13; SD = 1.32; range = 9–15; North American Adult Reading Test, (Blair & Spreen, 1989)). The participants were predominantly women (75%), White (92%), and not Hispanic or Latino (92%). Participants were initially screened via the Telephone Interview for Cognitive Status (TICS; Brandt et al., 1998), a brief cognitive screening measure that assesses orientation, attention, short-term memory, mathematical skills, praxis, and language. If a participant had a TICS score in the dementia range (<26; M = 35.31; SD = 2.43; range = 27–40) or was unable to give informed consent, he was excluded from study participation. Additionally, participants who were under 50 years of age or were not fluent in English were excluded. As measured by the 15-item Geriatric Depression Scale (Scogin et al., 2000), none of the study participants reported symptoms of severe depression (<12; M = 1.89, SD = 2.45; range = 0–10; 89% normal score; 8% mild symptom of depression; 3% moderate symptoms of depression). Because testing took place over video conferencing (Zoom), participants were required to have access to the internet. However, if they did not have a device with a web camera, they were provided with one.

Participants received a$40 honorarium for participating in the study and a report summarizing their performance on the standardized neuropsychological tests. The study was reviewed and approved by the Washington State University Institutional Review Board and participants signed a written consent form.

Measures

Overnight Trip Task

Clinic OTT. The clinic OTT is a naturalistic paper and pencil-based assessment that can be administered in a clinical or office-based environment. Participants are provided with a story that contains information about a hypothetical overnight trip (see Fig. 1). They are required to create a plan of what they need to pack for a successful trip. Within the story are rules that the participant must follow, irrelevant information that the participant must ignore, and information that should influence how the participant packs and prepares for the trip due to embedded contingencies. See Table 1 for the primary task variable and scoring criteria.

Fig. 1.

Fig. 1

Open in a new tab

Overnight Trip Task Story and Task Directions.

Table 1.

Overnight Trip Task (OTT) Primary Variables and Scoring

Primary OTT Variable Definition
Planning Score Planning = Contingency Items + Rules Followed
Contingency Item: Number of resources required based on contingencies generated from an analysis of environmental context. This value is out of nine points.
Rules Followed: Number of rules correctly followed. This value is out of eight points.
Contingency Items Rules
 1. Sleepwear  1. Does not pack/record over 20 items*
 2. Light Jacket/Sweater  2. Packed/recorded extra bag for car trip items
 3. Rain Protection  3. Has blue clothing/accessory item
 4. Item to help keep car clean  4. Has some form of sun protection
 5. Money/wallet  5. Brought an extra pair of socks
 6. Toothbrush  6. Brought tennis shoes/hiking shoes
 7. Toothpaste  7. Snack item for Jack is fruit
 8. Soap/Bodywash  8. Brought form of entertainment for car ride
 9. Shampoo
Open in a new tab

Note. *Participants that indicate they plan to wear an item intended for the trip instead of packing it (i.e., blue accessory or light jacket) get credited for the item but the item does not factor into the 20-item limit.

Real-World Overnight Trip Task (RW-OTT). The RW-OTT requires the participant to actually pack for an overnight trip, using the same contextual information and parameters as the OTT (i.e., participants are given the same story card, only they are instructed to pack/prepare for the trip). The RW-OTT follows identical scoring criteria (Table 1).

Neuropsychological tests

In addition to the TICS global cognition measure administered during screening, the following cognitive and performance-based measures were administered to examine discriminant and convergent validity, and to serve as predictor variables and a proxy measure of real-world functional status for the regression analysis.

Rey Auditory Verbal Learning Test (RAVLT; Rey, 1964). The RAVLT is a well-validated (Van den Burg & Kingma, 1999) verbal learning and memory test. Participants are required to learn a list of 15 orally presented words over five trials and then recall the list after a 20-min time delay. For this study, we used total words recalled across the five trials and total number of words recalled after the delay to examine discriminant validity.

Delis–Kaplan Executive Function System-Verbal Fluency subtasks (Letter Fluency and Category Switching; Delis et al., 2001). The DKEFS is a well-validated (Strauss et al., 2006) battery of tests that measures various executive functioning abilities, including cognitive flexibility, inhibition, and fluency. The letter fluency subtask requires participants to generate as many words as they can in 60 s that begin with a specified letter. The category switching subtest requires participants to generate as many words as they can in 60 s while switching back and forth between two categories. As EF has been defined as a multidimensional construct that comprises a multitude of abilities (Diamond, 2013; Gold, 2012), a clinical EF composite was created by computing z-scores for the number of correct words generated for each subtask and then averaging the z-scores. Collapsing across measures tapping different components of executive abilities should provide more representative measures and is a common methodology for research studies centered on detecting changes in EF (Iverson et al., 2020).

UCSD Performance-based Skill Assessment-2 – Organization/Planning subtest (UPSA planning; Patterson et al., 2001). The UPSA planning subtest is a measure of everyday planning ability and was used as a second measure of EF more closely associated with planning abilities. It requires a participant to read a mock advertisement for a water theme park and to recall various information from the advertisement and report items they would bring with them if they were to spend the day there. The measure is derived from the total correct answers and high values indicate better performance. Score can be in the range 0–14.

Instrumental Activities of Daily Living-Compensation Scale (IADL-C; Schmitter-Edgecombe, et al., 2014). The IADL-C is a self-report proxy measure of functional ability that assess everyday activities that people engage in and the degree to which a person is able to complete those activities, with or without compensatory strategies. The IADL-C was developed using factor analysis and Rasch item analysis and has demonstrated good internal consistency, and convergent and discriminant validity (Schmitter-Edgecombe, et al., 2014). An average score is derived from all items rated a 1 (independent, as well as ever, no aid) through 8 (always need help); ratings of 9 (does not need to complete activity) are not included in the average. A total raw score can then be derived by multiplying the average score by the number of items in the questionnaire; scores can range from 27 to 216.

Procedure

Participants who met initial screening criteria engaged in two testing sessions that were approximately 2 to 3 weeks apart and each session lasted approximately 3 hr. Both sessions took place through video conferencing using the Zoom application. Data were collected between July, 2020 and May, 2021, when social distance practices and data collection restrictions were in place as a result of the COVID-19 pandemic. Before the testing sessions, participants received a testing binder that contained their materials and instructions on how to structure their testing area. During testing, participants were informed not to remove materials from their testing binders until instructed to do so. Moreover, once participants completed an item, they were instructed to place the item into a return envelope. Participants were then told that they were no longer allowed to interact with items once they were placed in the return envelope. During both sessions, participants completed standardized neuropsychological tests and experimental tasks in the same order and were provided with one formal rest break and other rest breaks as needed.

During the first session, immediately following the break, participants were administered the OTT. For the OTT, participants were asked to create a physical plan after reading through a story that provided context for preparing for an imaginary overnight trip. Participants had up to 15 min to create their plan and they were able to reference the contextual story and instructions as many times as needed. Following completion of the plan, participants were asked to place their plan and the contextual story into their return envelope. At the end of the first testing session, participants retrieved another stimulus card that contained the same contextual story and were asked to actually pack and prepare for this hypothetical trip before the second testing session (RW-OTT). They were also provided with clear instruction to not attempt to recreate their original plan but instead to treat this as a brand-new task. Participants then had the opportunity to pack and prepare for this trip however they typically would in their everyday life over a 2-to-3-week period to approximate their “real-world” behavior outside of an evaluative setting. Directly after the participant finished packing for the trip, they were instructed to record the amount of time it took to complete the task.

Approximately 2 to 3 business days before the second testing session, participants received a phone call to remind them to have their bag packed and ready for the second testing session. During the second testing session, participants were asked to unpack the items that they prepared for the hypothetical overnight trip. Participants were then asked several follow-up questions, including (1) how performance on the RW-OTT differed from how they would normally prepare/pack for an overnight trip, and (2) which version of the task they felt they performed better on and why. Responses were then reviewed and classified based on common themes. Items from the clinic OTT constructed plans and the RW-OTT packed bags were scored by two research assistants following a detailed scoring methodology (task scoring criteria available upon request).

The OTT task and the scoring criteria were developed by K.B. and M.S.E. based upon iterative pilot work conducted with 15 participants over the age of 65 years. Based on performances and feedback provided by the participants, task instructions and scoring criteria were altered or eliminated to reduce the impact of unclear task instructions or vague participant responses.

Statistical analysis

Analyses were completed using SPSS Version 27. Intra class coefficient (ICC) estimates and their 95% confidence intervals were calculated based on a mean-rating (k = 2), absolute agreement, two-way mixed-effects model to examine interrater reliability of the planning score for the clinic OTT. A two-tailed Pearson correlation was then conducted to examine the strength of the association between the planning score of the clinic OTT and RW-OTT. Pearson coefficients were also used to examine convergent and discriminant validity of the clinic-OTT with the EF composite measure, the performance-based measure of everyday planning (UPSA planning subtest), and two measures of verbal memory (learning and delayed recall). Unless otherwise reported, sample sizes for all analyses were 65.

To examine incremental validity of the clinic-OTT to predict real-world behaviors above typical neurocognitive predictors of EF, two separate hierarchical multiple regression analyses were performed. The outcome measures were self-reported everyday functioning (i.e., IADL-C) and the RW-OTT. Global cognition and the EF composite were entered in block 2 and the clinic OTT was then entered as a predictor in block 2. If demographic variables (age, sex, education) or the UPSA planning were related to the outcome variables, they were also included in the regression analysis. With a sample size of 65, the multiple regression analysis with four predictors can detect a medium effect size (f2 = 0.20) with 80% power and 5% significance. The data that support the findings of this study are available from the corresponding author, upon reasonable request.

Results

Inter-rater reliability

ICC for the clinic OTT and the RW-OTT were 0.98 (CI = 0.95–0.99) and 0.94 (CI = 0.81–0.98), respectively. These values represent excellent inter-rater reliability (Koo & Li, 2016).

Establishing ecological validity for clinic OTT

See Table 2 for descriptive statistics of all study variables. To establish the ecological validity of the clinic OTT, a Pearson correlation was conducted between the planning scores of the two formats. The clinic OTT demonstrated a small to moderate relationship with the RW-OTT (r = 0.35, p = 0.005). Participants’ response to a question about whether their performance on the RW-OTT aligned with how they would typically prepare and pack for an overnight trip indicated that execution of the RW-OTT generally captured their real-world behavior. Specifically, 63.1% of participants reported that task completion did not differ from their typical preparation and packing behavior. The majority of the 36.9% of participants that reported a difference indicated that the difference was due to the item count limitation, but otherwise, consistent with their typical, real-world behavior.

Table 2.

Descriptive Statistics for Main Study Variables

N = 65
Primary Variables M Range Skewness/Kurtosis
Overnight Trip Task
 “Clinic” planning 11.22 7–15 −0.04/−0.64
 “Real-World” planning 12.38 7–16 −0.64/0.15
Executive Functions
Traditional measures
 EF Composite (DKEFS; VF) 0.00 −3.08 – 3.89 0.10/−0.38
Performance-based measure
 UPSA planning subtest 12.54 9–14 −0.64/−0.25
Memory (RAVLT)
 Learning 49.18 26–66 −0.17/−0.31
 Delayed recall 10.63 1–15 −0.64/0.10
Everyday Functioning
 Self-report measurea (IADL-C) 37.26 27–63 1.23/1.26
Open in a new tab

Note. aLower values indicate more independence/higher level of functioning and denote a sample size of 64. For all other values, higher values indicate better performance/higher level of functioning. EF composite = average z-score for Delis–Kaplan Executive Function System-verbal fluency and category switching accuracy measures (  Delis et al., 2001), RAVLT = Rey Auditory Verbal Learning Test (  Rey, 1964), IADL-C = Instrumental Activities of Daily Living- Compensatory Strategy scale (Schmitter-Edgecombe et al., 2014), UPSA = UCSD Performance-based Skill Assessment (  Patterson et al., 2001).

Participant perceptions of clinic OTT and RW-OTT performances

Participants were also asked which OTT format they thought they performed better on and why. Only two (3.1%) participants endorsed performing better on the clinic OTT than the RW-OTT, whereas 16 (25%) participants endorsed performing similar across formats. Most participants (71.9%; n = 46) endorsed performing better on the RW-OTT, with the majority of these participants (n = 22) providing an explanation related to time to explain why they thought they performed better on the RW-OTT, for example, greater perceived pressure to finish the clinic OTT quickly, more time to think about task during the RW-OTT, and more time to check over work during RW-OTT. Interestingly, only three participants used the full allotted time for the clinic OTT (15 min), with 50.8% spending 7.5 min or less on the task. The average self-reported completion time for the RW-OTT was 40.5 min (SD = 31.6 min; three participants left this item blank). More time spent on the clinic OTT was associated with better task performance (r = 0.48, p < 0.001). More self-reported time spent on the RW-OTT also had a relationship with better task performance (r = 0.29, p = 0.02; n = 62).

The second most common reason reported for better performance on the RW-OTT (n = 17) included explanations related to the ability to see or interact with items required for the task. The remaining sample reported the following reasons: the clinic OTT “felt like it was pretend” and so they did not “try as hard” (n = 2), task familiarity from the first format aided performance for the second format (n = 4), and unsure (n = 5). A Paired Samples t-test revealed that on average, participants did perform 1-point higher on the RW-OTT than the clinic OTT (t = −3.98, p < 0.001). However, there was not a significant relationship between participants’ perceptions of which task they performed better on and their actual performances (i.e., better OTT, same, better RW-OTT), X2(4, n = 64) = 7.02, p = 0.13.

Convergent and discriminant validity for clinic OTT

Cohen (1988) benchmarks were used in determining strength of relationships for convergent and discriminant validity. Correlation ranges from 0.0 to 0.2 were used to represent discriminant validity. For convergent validity, ranges of 0.3 and higher were used when assessing the clinic OTT’s relationship with the composite measure of EF and the planning test (see Table 3). Consistent with expectations, the clinic OTT demonstrated low correlation values with the measures of verbal learning (r = −0.15, p = 0.24) and memory (r = −0.22, p = 0.07). However, convergent validity could not be established for the clinic OTT as it did not significantly associate with the composite EF measure (r = 0.22, p = 0.09) or the UPSA planning subtest (r = 0.12, p = 0.33).

Table 3.

Correlations between study variables and the clinic Overnight Trip Task (OTT) and real-world OTT testing formats

OTT Testing Formats
Clinic OTT Real-World OTT
Demographics/Screening Variables
 Age -0.15 -0.04
 Sexa -0.03 -0.28*
 Education -0.04 -0.02
 Global Cognition 0.06 0.26*
 Depressive symptoms (GDS)^ 0.17 0.17
Executive Functions
 EF composite 0.22 0.07
 UPSA Planning 0.12 0.27*
Memory
 Learning -0.15 -0.08
 Delayed recall -0.22 -0.08
Open in a new tab

Note. *p ≤ 0.05, aSpearman correlation conducted. GDS 15-item = Geriatric Depression Scale; EF composite = Delis–Kaplan Executive Function System-verbal fluency and category switching accuracy measures; UPSA planning = UCSD Performance-based Skill Assessment-Planning subtest. ^n = 64.

Table 3 also provides correlations between the clinic and RW-OTT measures and age, sex, education, global cognition, and depression. The only significant correlations to emerge were those between the RW-OTT and sex (r = −0.28, p = 0.02) and the global cognition screening measure (r = 0.26, p = 04), with the female gender and higher cognition associated with better RW-OTT performance.

Incremental validity for clinic OTT

Self-reported Everyday Function. Due to the lack of significant relationships between demographic variables (i.e., age, sex, and education) and the outcome (IADL-C) or predictor variables (clinic OTT and EF composite) and to increase power for the multiple regression, demographics were excluded from the subsequent analyses. A hierarchical multiple regression analysis was conducted to evaluate the incremental validity of the clinic OTT to predict everyday functioning as assessed by a self-report proxy measure (i.e., IADL-C) after controlling for global cognition and the EF composite (Table 4). As one participant was missing the IADL-C, sample size for the regression analysis was 64. Global cognition, as measured by the TICS screening score, and the EF composite entered in block 1 accounted for a nonsignificant 5% of the variance in everyday functioning, R2 = 0.05, F(2, 61) = 1.50, p = 0.23, with no significant predictors emerging. When the clinic OTT was added in block 2 it accounted for an additional significant 7% of the variance in self-reported everyday functioning, ΔR2 = 0.07, ΔF = 4.48, p = 0.04. Although the clinic OTT was a significant predictor (t = 2.12, p = 0.04) in the final model, the final model just failed to reach significance (F(3, 60) = 2.55, p = 0.06), and the EF composite (t = 0.34, p = 0.74) and global cognition (t = −1.75, p = 0.08) were nonsignificant predictors.

Table 4.

Hierarchical Regression Analysis for Establishing Incremental Validity of the Clinic Overnight Trip Task (OTT) with a Self-report Measure

Everyday Functioning: IADL-C
Model Statistics
Model 1 R2 F p
0.05 1.50 0.23
Predictor Statistics
Predictor Β t p
 Global cognition −0.82 −1.70 0.09
 EF composite 0.58 0.79 0.43
Model Statistics
Model 2 R2 F p
ΔR2 = 0.07, p = 0.04 0.11 2.55 0.06
Predictor Statistics
Predictors Β t p
 Global cognition −0.82 −1.75 0.08
 EF composite 0.25 0.34 0.74
 Clinic OTT 1.12 2.12 0.04
Open in a new tab

Note. Dependent variable = Instrumental Activities of Daily Living- Compensatory Strategy scale (IADL-C). Global cognition = Telephone Interview for Cognitive Status; EF composite = Delis–Kaplan Executive Function System-verbal fluency and category switching accuracy measures.

RW-OTT. An additional hierarchical multiple regression analysis was conducted using RW-OTT as an outcome measure of real-world planning behavior (Table 5). We included in the regression analysis both demographic (i.e., sex) and cognitive measures (i.e., UPSA planning) that correlated with the RW-OTT in addition to global cognition and the EF composite (see Table 3). For the regression analysis, global cognition and the EF composite were entered into block 1 along with biological sex and UPSA planning. The initial model accounted for a significant 19% of the variance in real-world planning and packing behavior, R2 = 0.19, F(4, 60) = 3.58, p = 0.01, with both sex (t = −2.37, p = 0.02) and UPSA planning (t = −2.31, p = 0.02) emerging as significant predictors. When the clinic OTT was introduced, an additional significant 12% of the variance was accounted for, ΔR2 = 0.12, F(5, 59) = 10.42, p = 0.002. The clinic OTT, t = 3.23, p = 0.002, UPSA planning score, t = 2.27, p < 0.03, and sex, t = −2.80, p = 0.007, all were significant predictors of the RW-OTT.

Table 5.

Hierarchical Regression Analysis for Establishing Incremental Validity of the Clinic Overnight Trip Task (OTT) with the Real-world OTT

Everyday Functioning: RW-OTT
Model Statistics
Model 1 R2 F p
0.12 2.80 0.05
Predictor Statistics
Predictor Β t p
 Global cognition 0.14 1.33 0.19
 EF composite −0.10 −0.64 0.53
 Sex −1.33 −2.37 0.02
 UPSA planning 0.43 2.31 0.02
Model Statistics
Model 2 R2 F p
ΔR2 = 0.07, p = 0.04 0.19 8.29 0.001
Predictor Statistics
Predictors Β t p
 Global cognition 0.14 1.45 0.15
 EF composite −0.20 −1.33 0.19
 Sex −1.47 −2.80 0.007
 UPSA planning 0.39 2.27 0.03
 Clinic OTT 0.34 3.23 0.002
Open in a new tab

Note. Dependent variable = RW-OTT. Global cognition = Telephone Interview for Cognitive Status; EF composite = Delis–Kaplan Executive Function System-verbal fluency and category switching accuracy measures; UPSA planning = UCSD Performance-based Skill Assessment-Planning subtest. For sex; females coded as 0 and males coded as 1.

Discussion

Clinical neuropsychologists are frequently tasked with assessing the everyday functional capabilities of patients. This study serves as preliminary analyses for a recently developed function-led clinic OTT measure of everyday planning ability. To demonstrate the clinic OTT’s ecological validity beyond face validity, an analogous real-world format of the task (RW-OTT) was created. Most participants reported that their packing and planning behavior prompted by the RW-OTT resembled how they would typically prepare for a simple overnight trip, indicating that the RW-OTT was a good representation of this behavior. Although we hypothesized that the clinic OTT would demonstrate a moderate to high correlation with the RW-OTT, the clinic OTT only showed a small to moderate relationship with the RW-OTT. One possible explanation for this lower association may be the inability to interact with or see objects while completing the clinic OTT. In fact, of those that reported that they believed they performed better on the RW-OTT, most attributed their better performance to having more time on the task or due to seeing or interacting with the items they were packing (more on this described in a later section).

Additionally, female sex was associated with higher RW-OTT performance but not with clinic OTT performance. This suggests that other factors may be important to consider when comparing parallel clinic and real-world instruments. One such factor is experience, which could play a greater role in performance on the RW-OTT as compared with the clinic OTT, and for this cohort, it may be that women have more experience with packing. The novelty of the clinic OTT may have helped to eliminate the sex discrepancy. Future work will need to investigate this explanation and other factors that could modify the relationship between a laboratory task and a parallel real-world task (e.g., environmental support; personality traits).

The small-to-moderate relationship between the clinic- and RW-OTT found in this study falls slightly below previous findings for a recently developed virtual reality performance-based measure of functional ability, the Virtual Kitchen Challenge (VKC; Giovannetti et al., 2019). The VKC is a touch screen, computer-based task that requires participants to prepare breakfast or lunch by interacting with various objects on screen (e.g., a coffee cup, toaster oven, lunch box, etc.). Giovannetti et al. (2019) found Spearman rank order coefficients that ranged from 0.40 to 0.50 when comparing performance errors on the VKC with performance errors on “Real Kitchen”, a parallel task that requires participants to use real kitchen props. It is possible that higher associations between the VKC and “Real Kitchen” may be due to the ability to see and interact with objects across both tasks.

The majority of participants self-reported better performance on the RW-OTT compared with the clinic OTT. The most commonly described reason for the performance discrepancy was related to time. Either participants reported that they felt pressured to complete the clinic OTT as quickly as they were able, or they felt they had more time to reflect on and review the task during the RW-OTT. Spending more time on the clinic OTT was related to better task performance, however, only two participants spent the full allotted time and almost half of the sample only worked for less than half of the allotted time on their plans. Self-reported time spent on the RW-OTT was also related to better task performance. Although participants did perform better on the RW-OTT in comparison with the clinic OTT, the discrepancy on average was negligible (1-point difference). It is possible that the presence of an evaluator during the clinic OTT but not the RW-OTT may have contributed to a perceived time pressure, negatively affecting clinic OTT or allowing participants to perform better on the RW-OTT. To reduce the perceived time pressure and better reflect real-world packing behavior, the clinic OTT could be administered without an evaluator present or participants encouraged to work on the task for a specified minimum number of minutes.

Another common explanation among participants for better RW-OTT performance was the ability to see or interact with items required for the task. A common critique of office-based assessments is they lack “real-world” context, which may affect performance by obscuring true functional ability (Burgess et al., 2006). Incorporating some type of visual component to the clinic OTT may further enhance the task and evoke a better contextual sense to support performance. For example, a stimuli sheet containing pictures of typical and atypical items that someone might pack for an overnight trip could be provided in addition to the contextual story. Alternatively, four participants reported that task familiarity from the clinic OTT during the first testing session aided their performance on the RW-OTT. Due to an inability to counterbalance for this study, performance improvement for the RW-OTT could reflect a practice effect.

The clinic OTT demonstrated sufficient discriminant validity when compared with the two clinical measures of memory but did not demonstrate convergent validity in comparison with the clinical EF composite or the UPSA performance-based planning subtest. The lack of association may reflect the fact that EF tests used in this study are construct-driven tasks designed to measure discrete processes (i.e., set shifting and fluency), whereas the clinic OTT is designed to be a function-led measure and elicit use of a multitude of cognitive abilities (i.e., planning, forecasting, selective attention, etc.). Other clinic-based measures of executive functioning that more closely assess planning abilities (e.g., zoo map test, Tower of London) may have shown better associations, but could not be administered due to the remote format and await future investigation. The lack of association between the clinic OTT and UPSA planning subtest was unexpected, especially given that the UPSA demonstrated a significant association with the RW-OTT.

Incremental validity was established for the clinic OTT as the clinic OTT accounted for significant additional variance when predicting a self-report measure of everyday functioning (i.e., IADL-C) over and above global cognition and executive functioning. Furthermore, the clinic OTT was the only significant predictor of self-reported everyday functioning. Additional evidence of incremental validity came from the finding that after controlling for global cognition, sex, EF, and the UPSA, the clinic OTT accounted for significant variance in predicting RW-OTT. Moreover, both the clinic OTT and UPSA scores were significant predictors of RW-OTT along with sex. These findings suggest that the clinic OTT and UPSA planning measures are capturing some nonoverlapping aspects of real-world planning behavior. The UPSA planning subtest differs from the clinic OTT in that it contains a component that requires selective attention and short-term recall abilities. More specifically, after reading a mock advertisement about a water theme park, participants must remember specific information such as what time the park opens and what is the best way to get to the park. Although memory abilities were not significantly associated with OTT performances, the ability of the clinic OTT to predict RW-OTT performance might be improved by asking participants about key information from the contextual story (similar to UPSA) to improve attention to relevant details that could influence the quality of created plans. That is, the clinic-OTT might show even greater association with actual RW-OTT performance if participants are directed toward important contextual details during plan formulation as participants have more time to reflect on these contextual details when completing the RW-OTT.

Future studies should continue to work to develop and evaluate better clinic-based measures with greater ecological validity. In addition to demonstrating that the developed tasks can predict everyday functioning, the measures must also be demonstrated to be psychometrically sound (e.g., reliable, valid, normed) across populations and add value to current available assessment approaches. One particular challenge related to demonstrating the ecological validity of many measures is the selection of proxy measures for assessing everyday functioning as there is currently no gold standard. Although a variety of methodologies exist, including self- and informant-report and performance-based measures, each method has its limitations (Dassel & Schmitt, 2008; Marcotte et al., 2010). Poor agreement between methods for capturing functional ability is a common theme in the literature (e.g., Burton et al., 2009; Finlayson et al., 2003; Jefferson et al., 2008; Schmitter-Edgecombe et al., 2011; Tabert et al., 2002), suggesting that these methods may be capturing different aspects of everyday functioning. Furthermore, similar to EF, functional ability is not likely a unitary construct but rather comprises a multitude of cognitive systems (Gold, 2012) that in turn may be influenced by an amalgamation of cognitive abilities. Future work may benefit from development of a comprehensive composite score to capture more facets of everyday functioning. Alternatively, the problems demonstrated in the literature may serve as evidence to indicate that traditional measures are not the best method for measuring functional ability. Instead of continuing to reify the use of these proxy measures of everyday functioning, future work should seek to develop modernized methodologies for assessing real-world functioning. For example, incorporating wearable and sensor-based technologies that can collect real-time data and record daily behavioral routines may provide a more complete picture of everyday functioning.

Limitations

Due to the remote administration format, a selection bias may have occurred such that those more comfortable with technology may have self-selected to participate in this study. This self-selection may have led to a higher functioning sample (e.g., more highly educated, less cognitive or functional difficulties) than what is typical of community-dwelling older adults, reducing generalization to other older adult populations and variability in outcome measures. The study was also limited by the cognitive measures that could be selected due to the video conferencing administration format. The clinic OTT was also administered while participants were in their own homes rather than a laboratory, which may have allowed participants to better visualize what items they would pack for the hypothetical trip. However, participants were not allowed to leave their testing area or the view of the video camera while they completed the clinic OTT and the majority of participants had a limited view of their whole home (e.g., testing areas were located in office spaces or at a dining room tables).

The generalizability of some task requirements of the OTT poses another limitation. For example, individuals from different cultural or ethnic backgrounds may have different grooming or hygiene practices (i.e., a participant may not be inclined to nor need to wash their hair for an overnight trip if their hair is in a protective style and so may not include shampoo in their plan). A future iteration of the OTT should focus on the development of task requirements that are typically followed by or familiar across cultures. Future studies should also focus on recruitment of more diverse populations, as generalizability of the current findings is limited. In addition, future studies should counterbalance formats. Due to study design restrictions, we were unable to counterbalance the two formats of the OTT such that participants may have performed better on the RW-OTT due to reduced novelty and practice with the task.

Conclusion

In summation, the clinic OTT is a portable, easily administered, function-led measure of everyday planning ability. This study found that the clinic OTT associated low to moderately with a parallel RW-OTT. However, this association was lower than predicted and female sex was associated with better performance on the RW-OTT but not the clinic OTT, suggesting that other factors may need to be considered when comparing parallel clinic and real-world tasks. The OTT showed discriminant and incremental validity, but there was no support for convergent validity. This may partly reflect the inability to administer other EF planning measures (e.g., zoo map; Tower test) in this study due to the remote data collection format. Before the clinic OTT could be considered a reliable and valid clinic-based measure for predicting behavior in the real-world setting, additional research would be needed, including psychometric work and research demonstrating the utility of the clinic OTT in clinical practice. Future studies should continue to strive to develop proxy outcome measures of everyday functioning and take advantage of modernized technologies that more completely and accurately represent true real-world functioning. This is important because decisions are often made that can affect a patient’s future based on neurocognitive test findings, such as whether they can live independently or return to work. It is imperative that clinical neuropsychologists have the most appropriate and practical tools to address this goal.

Funding

This work was supported in part by funding from the National Institute of Biomedical Imaging and Bioengineering (R01 EB009675 to M.S-E.), the National Institute on Aging (R35 AG071451 to M.S-E), a Marchionne Graduate Research Fellowship, an H.L. Eastlick Distinguished Professorship, and Howard Hosick Student Research funds.

Conflict of Interest

None declared.

Author Contributions

Katelyn Brown (Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Writing – original draft) and Maureen Schmitter-Edgecombe (Conceptualization, Funding acquisition, Methodology, Resources, Supervision, Writing – review & editing).

Acknowledgements

We would like to thank members of the Neuropsychology and Aging laboratory for their support with testing participants and scoring the clinical data. We would also like to thank Justin Frow and Sarah Norman for coordinating data collection, and Lourdes Romero Zuniga and Lindsey Simmes for assistance with scoring the OTT protocols.

Contributor Information

Katelyn D Brown, Department of Psychology, Washington State University, Pullman, WA, USA.

Maureen Schmitter-Edgecombe, Department of Psychology, Washington State University, Pullman, WA, USA.

References

  1. Alderman, N., Burgess, P. W., Knight, C., & Henman, C. (2003). Ecological validity of a simplified version of the multiple errands shopping test. Journal of the International Neuropsychological Society: JINS, 9(1), 31–44. https://doi.org/ 10.1017/s1355617703910046. [DOI] [PubMed] [Google Scholar]
  2. Blair, J. R., & Spreen, O. (1989). Predicting premorbid IQ: A revision of the national adult reading test. The Clinical Neuropsychologist, 3(2), 129–136. https://doi.org/ 10.1080/13854048908403285. [DOI] [Google Scholar]
  3. Brandt, J., Spencer, M., Folstein, M. (1998). The Telephone Interview for Cognitive Status. Neuropsychiatry, Neuropsychology & Behavioral Neurology, 1, p 111–117. [Google Scholar]
  4. Burgess, P. W., Alderman, N., Forbes, C., Costello, A., Coates, L. M., Dawson, D. R., et al. (2006). The case for the development and use of "ecologically valid" measures of executive function in experimental and clinical neuropsychology. Journal of the International Neuropsychological Society: JINS, 12(2), 194–209. https://doi.org/ 10.1017/S1355617706060310. [DOI] [PubMed] [Google Scholar]
  5. Burton, C. L., Strauss, E., Bunce, D., Hunter, M. A., & Hultsch, D. F. (2009). Functional abilities in older adults with mild cognitive impairment. Gerontology, 55(5), 570–581. https://doi.org/ 10.1159/000228918. [DOI] [PubMed] [Google Scholar]
  6. Chan, R. C., Shum, D., Toulopoulou, T., & Chen, E. Y. (2008). Assessment of executive functions: Review of instruments and identification of critical issues. Archives of clinical neuropsychology: the official journal of the National Academy of Neuropsychologists, 23(2), 201–216. https://doi.org/ 10.1016/j.acn.2007.08.010. [DOI] [PubMed] [Google Scholar]
  7. Chaytor, N., & Schmitter-Edgecombe, M. (2003). The ecological validity of neuropsychological tests: A review of the literature on everyday cognitive skills. Neuropsychology Review, 13(4), 181–197. https://doi.org/ 10.1023/b:nerv.0000009483.91468.fb. [DOI] [PubMed] [Google Scholar]
  8. Chevignard, M. P., Taillefer, C., Picq, C., Poncet, F., Noulhiane, M., & Pradat-Diehl, P. (2008). Ecological assessment of the dysexecutive syndrome using execution of a cooking task. Neuropsychological Rehabilitation, 18(4), 461–485. https://doi.org/ 10.1080/09602010701643472. [DOI] [PubMed] [Google Scholar]
  9. Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates, Publishers. [Google Scholar]
  10. Craik, F. I., & Bialystok, E. (2006). Planning and task management in older adults: Cooking breakfast. Memory & Cognition, 34(6), 1236–1249. https://doi.org/ 10.3758/bf03193268. [DOI] [PubMed] [Google Scholar]
  11. Cuberos-Urbano, G., Caracuel, A., Vilar-López, R., Valls-Serrano, C., Bateman, A., & Verdejo-García, A. (2013). Ecological validity of the multiple errands test using predictive models of dysexecutive problems in everyday life. Journal of Clinical and Experimental Neuropsychology, 35(3), 329–336. https://doi.org/ 10.1080/13803395.2013.776011. [DOI] [PubMed] [Google Scholar]
  12. Dassel, K. B., & Schmitt, F. A. (2008). The impact of caregiver executive skills on reports of patient functioning. The Gerontologist, 48(6), 781–792. https://doi.org/ 10.1093/geront/48.6.781. [DOI] [PubMed] [Google Scholar]
  13. Dawson, D. R., Anderson, N. D., Burgess, P., Cooper, E., Krpan, K. M., & Stuss, D. T. (2009). Further development of the multiple errands test: Standardized scoring, reliability, and ecological validity for the Baycrest version. Archives of Physical Medicine and Rehabilitation, 90(11), S41–S51. https://doi.org/ 10.1016/j.apmr.2009.07.012. [DOI] [PubMed] [Google Scholar]
  14. Delis, D., Kaplan, E., & Kramer, J. (2001). Delis-Kaplan executive function scale. San Antonio, TX: The Psychological Corporation. [Google Scholar]
  15. Diamond, A. (2013). Executive functions. Annual Review of Psychology, 64(1), 135–168. https://doi.org/ 10.1146/annurev-psych-113011-143750. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Finlayson, M., Havens, B., Holm, M., & Van Denend, T. (2003). Integrating a performance-based observation measure of functional status into a population-based longitudinal study of aging. Canadian Journal on Aging / La Revue Canadienne Du Vieillissement, 22(2), 185–195. https://doi.org/ 10.1017/S0714980800004505. [DOI] [Google Scholar]
  17. Franzen, M. D., & Wilhelm, K. L. (1996). Conceptual foundations of ecological validity in neuropsychological assessment. In Sbordone, R. J., & Long, C. J. (Eds.), Ecological validity of neuropsychological testing (pp. 91–112). Delray Beach, FL: Gr Press/St Lucie Press, Inc. [Google Scholar]
  18. Giovannetti, T., Bettcher, B. M., Brennan, L., Libron, D. J., Kessler, R. K., & Duey, K. (2008). Coffee with jelly or unbuttered toast: Commissions and omissions are dissociable aspects of everyday action impairment in Alzheimer's disease. Neuropsychology, 22(2), 235–245. https://doi.org/ 10.1037/0894-4105.22.2.235. [DOI] [PubMed] [Google Scholar]
  19. Giovannetti, T., Yamaguchi, T., Roll, E., Harada, T., Rycroft, S. S., Divers, R., et al. (2019). The virtual kitchen challenge: Preliminary data from a novel virtual reality test of mild difficulties in everyday functioning. Neuropsychology, Development, and Cognition. Section B, Aging, Neuropsychology and Cognition, 26(6), 823–841. https://doi.org/ 10.1080/13825585.2018.1536774. [DOI] [PubMed] [Google Scholar]
  20. Gold, D. A. (2012). An examination of instrumental activities of daily living assessment in older adults and mild cognitive impairment. Journal of Clinical and Experimental Neuropsychology, 34(1), 11–34. https://doi.org/ 10.1080/13803395.2011.614598. [DOI] [PubMed] [Google Scholar]
  21. Iverson, G. L., Karr, J. E., Terry, D. P., Garcia-Barrera, M. A., Holdnack, J. A., Ivins, B. J., et al. (2020). Developing an executive functioning composite score for research and clinical trials. Archives of clinical neuropsychology: the official journal of the National Academy of Neuropsychologists, 35(3), 312–325. https://doi.org/ 10.1093/arclin/acz070. [DOI] [PubMed] [Google Scholar]
  22. Jefferson, A. L., Byerly, L. K., Vanderhill, S., Lambe, S., Wong, S., Ozonoff, A., et al. (2008). Characterization of activities of daily living in individuals with mild cognitive impairment. The American journal of geriatric psychiatry: official journal of the American Association for Geriatric Psychiatry, 16(5), 375–383. https://doi.org/ 10.1097/JGP.0b013e318162f197. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Kerr, S. (1991). Prospective memory, aging, and lapses of intention. Unpublished MA thesis, University of Toronto.
  24. Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting Intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/ 10.1016/j.jcm.2016.02.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Lawrence, I., Giovannetti, T., Seligman, S., Libon, D., & Sestito, J. (2013). P2–311: Performance‐based assessment of everyday action in MCI. Alzheimer's & Dementia., 9(4S_Part_11), P472–P473. https://doi.org/ 10.1016/j.jalz.2013.05.959. [DOI] [Google Scholar]
  26. Lezak, M. D., Howieson, D. B., & Loring, D. W. (2004). Neuropsychological assessment (4th ed.). New York, NY: Oxford University Press. [Google Scholar]
  27. Logie, R. H., Trawley, S., & Law, A. (2011). Multitasking: Multiple, domain-specific cognitive functions in a virtual environment. Memory & Cognition, 39(8), 1561–1574. https://doi.org/ 10.3758/s13421-011-0120-1. [DOI] [PubMed] [Google Scholar]
  28. Manchester, D., Priestley, N., & Jackson, H. (2004). The assessment of executive functions: Coming out of the office. Brain Injury, 18(11), 1067–1081. https://doi.org/ 10.1080/02699050410001672387. [DOI] [PubMed] [Google Scholar]
  29. Marcotte, T. D., Scott, J. C., Kamat, R., & Heaton, R. K. (2010). Neuropsychology and the prediction of everyday functioning. In Marcotte, T. D., & Grant, I. (Eds.), Neuropsychology of everyday functioning (pp. 5–38). New York: The Guilford Press. [Google Scholar]
  30. McGeorge, P., Phillips, L. H., Crawford, J. R., Garden, S. E., Sala, S. D., Milne, A. B., et al. (2001). Using virtual environments in the assessment of executive dysfunction. Presence: Teleoperators and Virtual Environments, 10(4), 375–383. https://doi.org/ 10.1162/1054746011470235. [DOI] [Google Scholar]
  31. Mumford, M. D., Schultz, R. A., & Van Doorn, J. R. (2001). Performance in planning: Processes, requirements, and errors. Review of General Psychology, 5(3), 213–240. https://doi.org/ 10.1037/1089-2680.5.3.213. [DOI] [Google Scholar]
  32. Oosterman, J. M., Wijers, M., & Kessels, R. P. (2013). Planning or something else? Examining neuropsychological predictors of zoo map performance. Applied Neuropsychology. Adult, 20(2), 103–109. https://doi.org/ 10.1080/09084282.2012.670150. [DOI] [PubMed] [Google Scholar]
  33. Parsons, T. D. (2011). Neuropsychological assessment using virtual environments: Enhanced assessment technology for improved ecological validity. In Yoshida, H., Jain, A., Ichalkaranje, A., Jain, L.C., & Ichalkarange, N.(Eds.), Advanced computational intelligence paradigms in heathcare 6. Virtual reality in psychotherapy, rehabilitation, and assessment (pp. 271–289). Berlin Heidelberg: Springer. [Google Scholar]
  34. Parsons, T. D., Carlew, A. R., Magtoto, J., & Stonecipher, K. (2017). The potential of function-led virtual environments for ecologically valid measures of executive function in experimental and clinical neuropsychology. Neuropsychological Rehabilitation, 27(5), 777–807. https://doi.org/ 10.1080/09602011.2015.1109524. [DOI] [PubMed] [Google Scholar]
  35. Patterson, T. L., Goldman, S., McKibbin, C. L., Hughs, T., & Jeste, D. V. (2001). UCSD performance-based skills assessment: Development of a new measure of everyday functioning for severely mentally ill adults. Schizophrenia Bulletin, 27(2), 235–245. https://doi.org/ 10.1093/oxfordjournals.schbul.a006870. [DOI] [PubMed] [Google Scholar]
  36. Rabin, L. A., Burton, L. A., & Barr, W. B. (2007). Utilization rates of ecologically oriented instruments among clinical neuropsychologists. The Clinical Neuropsychologist, 21(5), 727–743. https://doi.org/ 10.1080/13854040600888776. [DOI] [PubMed] [Google Scholar]
  37. Rey, A. (1964). L’examen clinique enpsychologie. Paris: Presses Universitairesde France. [Google Scholar]
  38. Robertson, K., Schmitter-Edgecombe, M., Weeks, D., & Pimentel, J. (2018). Naturalistic assessment using a simulated environment: Cognitive correlates and relationship to functional status in individuals with neurologic conditions. Archives of clinical neuropsychology: the official journal of the National Academy of Neuropsychologists, 33(8), 1024–1039. https://doi.org/ 10.1093/arclin/acx136. [DOI] [PubMed] [Google Scholar]
  39. Schmitter-Edgecombe, M., Cunningham, R., McAlister, C., Arrotta, K., & Weakley, A. (2021). The night out task and scoring application: An ill-structured, open-ended clinic-based test representing cognitive capacities used in everyday situations. Archives of clinical neuropsychology: the official journal of the National Academy of Neuropsychologists, 36(4), 537–553. https://doi.org/ 10.1093/arclin/acaa080. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Schmitter-Edgecombe, M., McAlister, C., & Weakley, A. (2012). Naturalistic assessment of everyday functioning in individuals with mild cognitive impairment: The day-out task. Neuropsychology, 26(5), 631–641. https://doi.org/ 10.1037/a0029352. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Schmitter-Edgecombe, M., Parsey, C., & Cook, D. J. (2011). Cognitive correlates of functional performance in older adults: Comparison of self-report, direct observation, and performance-based measures. Journal of the International Neuropsychological Society: JINS, 17(05), 853–864. https://doi.org/ 10.1017/S1355617711000865. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Schmitter-Edgecombe, M., Parsey, C., & Lamb, R. (2014). Development and psychometric properties of the instrumental activities of daily living: compensation scale. Archives of clinical neuropsychology : the official journal of the National Academy of Neuropsychologists, 29(8), 776–792. 10.1093/arclin/acu053. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Schwartz, M. F., Buxbaum, L. J., Ferraro, M., Veramonti, T., & Segal, M. (2003). The naturalistic action test. Bury St. Edmunds, England: Thames Valley Test Company. [Google Scholar]
  44. Scogin, F., Rohen, N., & Bailey, E. (2000). Geriatric Depression Scale. In Marusih, M. E. (Ed.), Handbook of psychological assessment in primary care settings (pp. 491–508). Hillsdale, NJ: Lawrence Erlbaum Associates Publishers. [Google Scholar]
  45. Shallice, T., & Burgess, P. W. (1991). Deficits in strategy application following frontal lobe damage in man. Brain: A Journal of Neurology, 114(2), 727–741. https://doi.org/ 10.1093/brain/114.2.727. [DOI] [PubMed] [Google Scholar]
  46. Strauss, E., Sherman, E. M. S., & Spreen, O. (2006). A Compendium of Neuropsychological Tests: Administration, Norms and Commentary. New York: Oxford University Press. [Google Scholar]
  47. Tabert, M. H., Albert, S. M., Borukhova-Milov, L., Camacho, Y., Pelton, G., Liu, X., et al. (2002). Functional deficits in patients with mild cognitive impairment: Prediction of AD. Neurology, 58(5), 758–764. https://doi.org/ 10.1212/wnl.58.5.758. [DOI] [PubMed] [Google Scholar]
  48. van den Burg, W., & Kingma, A. (1999). Performance of 225 Dutch school children on Rey's Auditory Verbal Learning Test (AVLT): parallel test-retest reliabilities with an interval of 3 months and normative data. Archives of clinical neuropsychology: the official journal of the National Academy of Neuropsychologists, 14(6), 545–559. 10.1016/s0887-6177(98)00042-0. [DOI] [PubMed] [Google Scholar]
  49. Wilson, B. A., Alderman, N., Burgess, P., Emslie, H., & Evans, J. J. (1996). Behavioral assessment of the dysexcutive syndrome (BADS). Bury St. Edmunds, England: Thames Valley Test Company. [Google Scholar]

Articles from Archives of Clinical Neuropsychology are provided here courtesy of Oxford University Press

RESOURCES