Abstract
Purpose
To assess the feasibility of using a widely validated, web-based neurocognitive test battery (Cambridge Brain Sciences, CBS) in a cohort of critical illness survivors.
Methods
We conducted a prospective observational study in two intensive care units (ICUs) at two tertiary care hospitals. Twenty non-delirious ICU patients who were mechanically ventilated for a minimum of 24 hours underwent cognitive testing using the CBS battery. The CBS consists of 12 cognitive tests that assess a broad range of cognitive abilities that can be categorized into three cognitive domains: reasoning skills, short-term memory, and verbal processing. Patients underwent cognitive assessment while still in the ICU (n = 13) or shortly after discharge to ward (n = 7). Cognitive impairment on each test was defined as a raw score that was 1.5 or more standard deviations below age- and sex-matched norms from healthy controls.
Results
We found that all patients were impaired on at least two tests and 18 patients were impaired on at least three tests. ICU patients had poorer performance on all three cognitive domains relative to healthy controls. We identified testing related fatigue due to battery length as a feasibility issue of the CBS test battery.
Conclusions
Use of a web-based patient-administered cognitive test battery is feasible and can be used in large-scale studies to identify domain-specific cognitive impairment in critical illness survivors and the temporal course of recovery over time.
Introduction
Long-term cognitive impairment is a common complication in critical illness survivors [1–3] but its natural history and the temporal profile of cognitive recovery in individual patients remain unknown. Understanding these features is critical for identifying optimal therapeutic windows and selecting patient subgroups most likely to benefit from targeted interventions.
Several studies have examined cognitive function in the ICU patients at various time points ranging from assessments in hospital at time of or shortly after ICU discharge [4–8] to long-term follow-up of up to 13 years after discharge [9]. Limitations of existing neurocognitive assessment tools have restricted cognitive testing in prior studies to a few discrete time points [1–9]. Many neurocognitive tools require specially trained staff for test administration, take a long time to administer, and necessitate patients to attend testing sessions in person, which often excludes those who have limited mobility, are institutionalized, live far away from the testing centre, or are unwilling to return to the hospital where they were admitted due to associated traumatic memories. Attrition rates for studies involving long-term follow-up of ICU patients vary from 60 to 79% at 3 months and 69 to 95% at 12 months [1, 6, 10–13]. High attrition rates in some studies that require ICU patients to return to clinic for follow-up testing may affect the generalizability of study findings. A web-based neurocognitive assessment tool that can be self-administered remotely by patients may lead to higher follow-up rates and enable large-scale natural history studies of long-term cognitive impairment in critical illness survivors.
We assessed the feasibility of using a widely validated, web-based neurocognitive test battery previously used in large cohort studies, Cambridge Brain Sciences (CBS) [14–16], as well as in other clinical populations [17–22], in a cohort of critical illness survivors.
Materials and methods
Study design and setting
We conducted a prospective observational study of patients admitted to two adult tertiary care centre intensive care units (ICUs) in London, Canada. The study was approved by our local research ethics board (WesternREM, study identification number 108156).
Patients
We included adult patients (18 to 80 years of age) who were mechanically ventilated for a minimum of 24 hours. The latter criterion excluded routine post-operative patients and non-critically ill patients requiring a brief period of mechanical ventilation (e.g. for acute intoxication).
Patients were excluded if they had a pre-existing diagnosis of dementia, new or pre-existing diagnosis of neurological disease known to affect cognitive function (e.g. stroke, head trauma, intracranial hemorrhage, traumatic brain injury, or intracranial malignancy), impaired vision or significant upper extremity weakness precluding use of a computer, inability to communicate in English or were unable to or declined to provide informed consent, or active delirium as determined by the Intensive Care Delirium Screening Checklist (ICDSC) [23].
Measurements
Cognitive assessment
CBS is a web-based neurocognitive assessment battery consisting of 12 tests that tap into a broad range of cognitive processes. These include: Feature Match, Odd One Out, Polygons, Rotations, Spatial Planning, Monkey Ladder, Paired Associates, Spatial Span, Spatial Search, Digit Span, Double Trouble, and Grammatical reasoning. Each of the 12 tests taps into three cognitive domains–reasoning skills, short-term memory, and verbal ability—to varying degrees. Previously, it was shown that these three factors–reasoning skills, short-term memory, and verbal ability–explain a large proportion of variability in performance across the tests [15]. The CBS can be self-administered by individuals without the need for a neuropsychologist or other trained personnel. The battery has been widely used as a measure of cognitive function in individuals across the world [14–16], as well as those with neuroanatomical lesions [17–18] and Parkinson’s disease [19–20].
Informant Questionnaire on Cognitive Decline in Elderly
To screen for previous history of cognitive deficits, we asked a relative or friend who knows the patient well to complete the Informant Questionnaire on Cognitive Decline in Elderly (IQCODE). The IQCODE has been shown to have high reliability in measuring cognitive decline and correlates well with other neurocognitive tests [24]. Family assessors are presented with 16 specific tasks (e.g., remembering where things are usually kept) and asked to rate the patient’s current status on that task relative to 10 years ago on a scale of 1 (much improved) to 5 (much worst) with a score of 3 indicating “no change”. A cut-off score of 3.44 was used as a positive screen for pre-existing dementia [9].
Feasibility
Our aim was to assess the feasibility of using a web-based, neurocognitive battery to assess cognition in ICU survivors at the time of ICU discharge. Our primary measure of feasibility was the number of patients who completed the entire 12-test cognitive battery. In addition, researchers who recruited patients and were present while patients underwent cognitive testing were asked to make notes about any issues that arose throughout the administration of the cognitive battery and any concerns that were raised by patients throughout testing so that these challenges could be addressed in future studies using the CBS battery.
Study protocol
One investigator (KH or SM) or a research assistant screened patients for eligibility using medical charts and discussion with bedside nurses. We approached eligible patients and obtained written informed consent prior to commencing the study. Patients were deemed to be adequately alert and capable of providing consent if they were able to participate in the consent discussion.
We recorded demographic and clinical variables including age, sex, admission diagnosis, hospital and ICU admission dates, and Nine Equivalents of Nursing Manpower Use Score (NEMS) [25] on the day of testing from each patient’s chart or electronic medical record.
Patients completed cognitive assessment using a laptop computer with an attached computer mouse in their hospital bed in the ICU or shortly after discharge to the ward. Investigators assisted patients by creating a personalized login and password on the CBS study webpage. A standardized set of written and pictorial instructions and a short instructional video preceded each cognitive test. Patients were provided with as much time as they needed to review the instructions prior to beginning each test. Patients completed each of the tests in sequence until the entire battery of 12 tests was completed or they were unable to continue due to testing related fatigue.
We asked a relative or friend who knew the patient well to complete the IQCODE. Family members responded to the IQCODE questions by telephone if they were not available in person.
Data analysis
Demographic and clinical variables were reported using descriptive statistics, expressed as frequency and percentage for categorical variables and mean and standard deviation (SD) or median and interquartile range (IQR) for continuous variables as applicable.
Patients were defined as having cognitive impairment on a given test if their raw score was ≥ 1.5 SDs below age- and sex-matched controls derived from the CBS normative database [15]. We then compared patients’ cognitive performance with available data from healthy age- and sex-matched control data by converting raw scores into z-scores.
To determine patients’ scores on each of the three cognitive domains (reasoning skills, short-term memory, and verbal processing), the z-score for each individual test was multiplied by a value that reflected the contribution of that test to each cognitive domain (i.e., factor loading) as established by Hampshire and colleagues [15]. Patients’ overall score on each cognitive domain was therefore the sum of the weighted (factor loaded) scores for that domain across all 12 tests. Data imputation was only used to compare domain scores for patients who completed all 12 tests. The scores are designed such that the healthy population mean on each cognitive domain is 0 and the SD is 1.0.
Finally, we defined an abbreviated CBS battery that included six of the 12 CBS tests which most strongly reflect one of the three cognitive domains based on previously published data (Reasoning Skills: Odd One Out and Rotations; Short-term Memory: Paired Associates and Monkey Ladder; and Verbal Processing: Digit Span and Verbal Reasoning) [15]. We replaced the scores for the omitted tests with their expected values given the six observed test scores and the known correlation structure among the tests in the population. The correlation structure between the 12 tests in the CBS battery was derived from a sample of 44,600 [15]. This method has been shown to be most accurate when calculating principle component analysis scores in the presence of missing data [26]. We then calculated a z-score for each patient on each of the three cognitive domains based on the abbreviated CBS and compared this score to the z-scores calculated based on the complete CBS battery.
Results
Of 45 patients approached, 25 declined to participate in this study, most commonly due to self-reported inability to participate or limited availability due to required clinical care activities (e.g., diagnostic tests). Twenty patients (7 females) were included in the analysis (Fig 1). Patient demographic and clinical characteristics are presented in Table 1. We reached family members or friends of 15 patients for completion of the IQCODE. None of these patients were found to have pre-morbid dementia as assessed by the IQCODE.
Fig 1. Recruitment flow diagram.
Table 1. Patient demographic and clinical characteristics.
| CHARACTERISTIC | STATISTIC | |
|---|---|---|
| Age, median (IQR) | 58.5 (41–66) | |
| Sex | ||
| Males | 13 | |
| Females | 7 | |
| Admission Diagnosis | ||
| Respiratory, n | 3 | |
| Cardiac, n | 2 | |
| Sepsis, n | 2 | |
| Cardiac arrest, n | 4 | |
| Neurologic, n | 1 | |
| Trauma, n | 2 | |
| Surgical, n | 3 | |
| Other, n | 3 | |
| Duration of Mechanical Ventilation, median (IQR) | 2.5 (1–3) | |
| ICU length of stay (d), median (IQR) | 5 (4–9) | |
| Nine equivalents of nursing manpower use score, median (IQR) | 18 (18–22) | |
IQR = Interquartile range.
Thirteen patients were tested in the ICU and seven were tested within one to four days of transfer to the ward. Patients were tested after a median of 4 days (IQR 3.75) following ICU admission. Three patients completed testing on two separate days at the patient’s request. Seventeen of 20 patients completed the full 12-test CBS battery (the other three patients completed four, six and seven tests). Of the three patients who did not complete the full battery, two were tested on the ward and one was tested while still in the ICU. The mean duration for completion of the 12-test CBS battery was 45.5 minutes (SD 11.1) excluding the three patients who did not complete the entire battery and one patient whose testing who had a long break between tests.
All patients were impaired on at least two tests, and 18 were impaired on at least three tests relative to healthy controls. Among the 17 of 20 patients who completed the full CBS battery, patients were impaired on a median of eight tests (IQR 4.5–10, range 2–11).
Individual patient performances as well as cohort means on each of the 12 CBS tests relative to normative data are presented in Fig 2. Patients had poorer performance on all CBS tests compared to healthy controls (Table 2). Patients also had poorer performance on all three cognitive domains compared to healthy controls (Fig 3 and Table 2). Our anonymized data set is provided in S1 Table.
Fig 2. Patient performance on the 12-test Cambridge Brain Sciences neurocognitive battery.
Individual patient (circles) and cohort (solid lines) test performance presented as z-scores corrected for age and sex.
Table 2. Cognitive performance of critical care survivors relative to healthy control population.
| COGNITIVE TEST/ DOMAIN | MEAN Z-SCORE (SD) | STATISTICS |
|---|---|---|
| CBS COGNITIVE TEST | ||
| Feature Match | -2.21 (1.16) | t(17) = -8.10, p < 0.0001 |
| Odd One Out | -2.10 (1.83) | t(19) = -5.14, p < 0.0001 |
| Polygons | -1.35 (0.67) | t(16) = -8.28, p < 0.0001 |
| Rotations | -1.58 (0.84) | t(18) = -8.22, p < 0.0001 |
| Spatial Planning | -1.12 (0.92) | t(16) = -5.04, p < 0.0001 |
| Monkey Ladder | -2.21 (1.93) | t(19) = -5.13, p < 0.0001 |
| Paired Associates | -1.42 (1.04) | t(16) = -5.64, p < 0.0001 |
| Spatial Span | -1.97 (1.56) | t(18) = -5.51, p < 0.0001 |
| Spatial Search | -1.40 (0.92) | t(16) = -6.30, p < 0.0001 |
| Digit Span | -1.85 (0.91) | t(16) = -8.40, p < 0.0001 |
| Double Trouble | -1.43 (0.62) | t(19) = -10.40, p < 0.0001 |
| Grammatical Reasoning | -2.46 (0.65) | t(19) = -16.89, p < 0.0001 |
| COGNITIVE DOMAIN | ||
| Reasoning Skills | -2.13 (1.45) | t(16) = -6.05, p < 0.0001; |
| Short-term Memory | -1.92 (1.41) | t(16) = -5.60, p < 0.0001 |
| Verbal Processing | -1.66 (1.21) | t(16) = -5.65, p < 0.0001 |
Fig 3. Patient performance on the three Cambridge Brain Sciences cognitive domains: reasoning skills, short-term memory, and verbal processing.
Individual patient (circles) and cohort (solid lines) test performance on each cognitive domain presented as z-scores corrected for age and sex. By design, healthy norm mean = 0 and standard deviation = 1.
We identified two feasibility issues with the web-based CBS platform: First, patient-reported testing related fatigue due to battery length (with three of 20 patients requesting that the testing be stopped before completion of all 12 tests); and second, one patient reported inexperience with use of a computer mouse and was instructed on its use by the investigator.
To address the issue of test fatigue related to battery length, we compared the performance of an abbreviated version of the CBS battery containing 6 tests with the full 12-test battery and found that the abbreviated battery performed well in predicting performance on each of the three cognitive domains (Fig 4).
Fig 4. Scatterplots including a linear fit of the abbreviated Cambridge Brain Sciences Battery against the full battery on the three CBS cognitive domains.
(A) Reasoning (r = 0.91), (B) Memory (r = 0.80), and (C) Verbal Processing (r = 0.94). Solid black line indicates x = y.
Discussion
Our results demonstrate that a web-based patient-administered neurocognitive test battery can identify domain-specific cognitive impairment in critical illness survivors. Unlike paper-based cognitive assessment tools, the web-based battery can be self-administered by patients from any location (ward, rehabilitation facility, home) after minimal instructions on logging in and starting the tests, obviating the need for patients to travel to hospital to be tested by specially trained staff. The self-administering nature of web-based battery may enable large-scale studies similar to those completed in other populations [14–16], and provides opportunity for repeated remote monitoring of cognitive recovery in patients who are unable to return to clinic. Given high prevalence of functional disability, post-traumatic stress disorder, and challenging travel logistics, remote web-based testing of cognition can provide a more patient-centred approach to monitoring cognition in ICU survivors.
The majority of previous studies have used traditional, paper-and-pencil testing to assess cognitive function in critical illness survivors [1–9]. Screening tools such as the Mini-Mental State Examination (MMSE) [27], have been designed to screen for overt dementia, but lack comprehensiveness and have poor sensitivity for more subtle, yet consequential cognitive deficits in critical illness survivors [28].
More comprehensive cognitive batteries such as the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) [29] and Wechsler Adult Intelligence Scale-Fourth Edition (WAIS-IV) [30], have well-established psychometric properties (RBANS [31,32]; WAIS-IV [33,34]) have been widely used in various patient populations (RBANS [31,32,35,36]; WAIS-IV [37,38]), but are not optimal for some natural history studies of critical illness survivors due to the need for in-person testing by trained personnel. Our results demonstrate that web-based cognitive testing is both feasible and able to detect subtle changes in cognitive domains similar to traditional paper-and-pencil tests.
There are several advantages to the CBS battery that make it an attractive tool for assessing cognitive dysfunction in ICU survivors. Similar to traditional cognitive assessment tools, the CBS battery used in this study is comprehensive, with 12 individual tests assessing cognitive function across three distinct cognitive domains. It has been used to study cognition in several large patient cohorts [14–16] and has an established large normative database to enable sex- and age-matched comparisons of patients’ performance to healthy controls. Furthermore, the CBS battery can be administered in the functional magnetic resonance imaging scanner, enabling mapping of cognitive results to neuroimaging data [14,39].
Optimization of the CBS battery for ICU needs by addressing feasibility issues identified in this study would enable its use for remote monitoring of cognitive recovery in individual patients, and provide an objective outcome metric for tracking patient’s cognitive health and gauging effectiveness of candidate preventative, therapeutic, and rehabilitative interventions. Given that repeated testing can lead to practice effects, the CBS battery has been optimized for repeated measures studies by incorporating proprietary algorithms that generate a unique set of problems for each assessment, facilitating longitudinal studies with minimal ‘practise effects’. However, whether repeated testing leads to practice effects in ICU survivors requires further dedicated studies. Furthermore, given comparable performance of an abbreviated 6-test version of the CBS battery in our study, its use in future studies would address the issue of test fatigue related to battery length. While we did not administer the abbreviated 6-test battery in this study, we expect that its duration will be approximately half of the 12-test battery (i.e., approximately 20 minutes). This will need to be confirmed in dedicated future studies. In addition, the CBS has recently become available for use on a portable tablet computer, which may further simplify its administration to ICU patients by obviating the need for the hand-computer mouse coordination.
This study has several limitations. Our small sample size allowed us to determine the feasibility of web-based testing using the CBS battery, but prevents us from drawing any definitive conclusions regarding cognitive outcomes in patients. In the future, we will use our feasibility data to optimize CBS battery for ICU use and repeat cognitive assessment in a larger ICU cohort, with repeated measurements in the same patients to assess the ability to track the cognitive recovery process over time.
Web-based cognitive testing in general is not without limitations. The inability to monitor patients during testing to determine their level of engagement, patients’ access to and inexperience with use of a computer, and attrition in follow-up studies are limitations inherent to the use of web-based (remote) cognitive testing that must be addressed prior to large-scale use of such testing platforms in ICU survivors.
In our study, patients’ performance in our study may have been affected by their ability to use a computer mouse to navigate through the tasks. In future studies, we plan to test tablets as an alternative to computer-mouse combination based on patient preference. Finally, in the absence of a “gold standard” test for cognitive impairment in this study, the psychometric characteristics of the CBS could not be assessed, although previous studies in healthy controls [15], and elderly neuropsychiatric patients [40], have confirmed that it is comparable to standard neuropsychological test batteries in terms of its latent structure and relation to age.
We identified that the length of the CBS cognitive battery may be one challenge to its feasibility in larger studies. The proposed abbreviated CBS battery is one potential solution to this challenge. Future studies are needed to assess the feasibility of the abbreviated CBS battery.
Conclusions
We demonstrated that a comprehensive, web-based neurocognitive testing platform is feasible for use in critical illness survivors and detects domain-specific cognitive impairment. We identified battery length as a potential challenge to wider scale use of the CBS battery, which we believe can be addressed by shortening the battery length. Further studies using the CBS cognitive battery are needed to determine its feasibility in assessing cognition and cognitive recovery over time. Further optimization of this tool for ICU patients may possibly establish a novel frontier in ICU cognition research by enabling remote monitoring of cognitive recovery in critical illness survivors, correlation of cognitive scores with neuroimaging data to help identify underlying neural mechanisms, and objective assessment of outcomes associated with preventative, therapeutic, and rehabilitative interventions.
Supporting information
(XLSX)
Acknowledgments
The authors would like to acknowledge Ms. Tanya Tamasi and Mr. Neil Patel for their assistance in patient recruitment and testing. Dr. Slessarev would like to acknowledge Vanier Canada Graduate Scholarship for supporting his PhD studies.
Data Availability
All relevant data are within the manuscript and its Supporting Information files.
Funding Statement
This study was supported by research funds from the Department of Medicine, Schulich School of Medicine & Dentistry, Western University.
References
- 1.Pandharipande PP, Girard TD, Jackson JC, Morandi A, Thompson JL, Pun BT, et al. Long-Term Cognitive Impairment after Critical Illness. NEJM. 2013;369(14): 1306–1316. 10.1056/NEJMoa1301372 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Hopkins RO, Weaver LK, Collingridge D, Parkinson RB, Chan KJ, Orme Jr. JF. Two-year cognitive, emotional, and quality-of-life outcomes in acute respiratory distress syndrome. Am J Respir Crit Care Med. 2005;171(4): 340–347. 200406-763OC [pii] 10.1164/rccm.200406-763OC [DOI] [PubMed] [Google Scholar]
- 3.Mitchell M, Shum D, Mihala G, Murfield J, Aitken L. Long-term cognitive impairment and delirium in intensive care: A prospective cohort study. Aust Crit Care. 2017;S1036-7314(17): 30034–30036. [DOI] [PubMed] [Google Scholar]
- 4.Hopkins RO, Weaver LK, Pope D, Orme Jr. JF, Bigler ED, & Larson-Lohr V. Neuropsychological sequelae and impaired health status in survivors of severe acute respiratory distress syndrome. Am J of Respir Crit Care Med.1999;160(1):50–56. [DOI] [PubMed] [Google Scholar]
- 5.Ely EW, Shintani A, Truman B, Speroff T, Gordon SM, Harrell FE, et al. Delirium as a predictor of mortality in mechanically ventilated patients in the intensive care unit. JAMA. 2004;291(14):1753–1762. 10.1001/jama.291.14.1753 [DOI] [PubMed] [Google Scholar]
- 6.Hopkins RO, Weaver LK, Collingridge D, Parkinson RB, Chan KJ, & Orme Jr. JF. Two-year cognitive, emotional, and quality-of-life outcomes in acute respiratory distress syndrome. Am J of Respir Crit Care Med. 2005;17(4):340–347. [DOI] [PubMed] [Google Scholar]
- 7.Jones C, Griffiths RD, Slater T, Benjamin KS, Wilson S. Significant cognitive dysfunction in non-delirious patients identified during and persisting following critical illness. Intensive Care Med. 2006;32(6):923–926. 10.1007/s00134-006-0112-y [DOI] [PubMed] [Google Scholar]
- 8.Ketterer MW, Alaali Y, Yessayan L. "Alert and oriented x 3?" Correlates of Mini-Cog performance in a post/nondelirious intensive care unit sample. Psychosomatics. 2016;57(2):194–199. 10.1016/j.psym.2015.10.008 [DOI] [PubMed] [Google Scholar]
- 9.Kapfhammer HP, Rothenhausler HB, Krauseneck T, Stoll C, Schelling G. Posttraumatic stress disorder and health-related quality of life in long-term survivors of acute respiratory distress syndrome. Am J Psychiatry. 2004;161(1):45–52. 10.1176/appi.ajp.161.1.45 [DOI] [PubMed] [Google Scholar]
- 10.Wood MD, Muscedere J, Scott SH, Boyd JG, Canadian Critical Care Trials Group. Robotic technology provides objective and quantifiable metrics of neurocognitive functioning in survivors of critical illness: A feasibility study. J Crit Care. 2018;48:228–236. 10.1016/j.jcrc.2018.09.011 [DOI] [PubMed] [Google Scholar]
- 11.Dinglas VD, Hopkins RO, Wozniak AW, Hough CL, Morris PE, Jackson JC, et al. One-year outcomes of rosuvastatin versus placebo in sepsis-associated acute respiratory distress syndrome: prospective follow-up of SAILS randomised trial. Thorax. 2016;71(5):401–410. 10.1136/thoraxjnl-2015-208017 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Needham DM, Wozniak AW, Hough CL, Morris PE, Dinglas VD, Jackson JC, et al. Risk factors for physical impairment after acute lung injury in a national, multicenter study. Am J Respir Crit Care Med. 2014;189(10):1214–1224. 10.1164/rccm.201401-0158OC [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Needham DM, Colantuoni E, Dinglas VD, Hough CL, Wozniak AW, Jackson JC, et al. Rosuvastatin versus placebo for delirium in intensive care and subsequent cognitive impairment in patients with sepsis-associated acute respiratory distress syndrome: an ancillary study to a randomised controlled trial. Lancet Respir Med. 2016;4(3):203–212. 10.1016/S2213-2600(16)00005-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Owen AM, Hampshire A, Grahn JA, Stenton R, Dajani S, Burns AS, et al. Putting brain training to the test. Nature. 2010;465(7299): 775–778. 10.1038/nature09042 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Hampshire A, Highfield RR, Parkin BL, Owen AM. Fractionating Human Intelligence. Neuron. 2012;76(6): 1225–1237. 10.1016/j.neuron.2012.06.022 [DOI] [PubMed] [Google Scholar]
- 16.Wild CJ, Nichols ES, Battista ME, Stojanoski B, Owen AM. Dissociable effects of self-reported daily sleep duration on high-level cognitive abilities. Sleep. 2018;41(12). 10.1093/sleep/zsy182 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Owen AM, Downes JJ, Sahakian BJ, Polkey CE, Robbins TW. Planning and spatial working memory following frontal lobe lesions in man. Neuropsychologia. 1990;28: 1021–1034. [DOI] [PubMed] [Google Scholar]
- 18.Owen AM, Roberts AC, Polkey CE, Sahakian BJ, Robbins TW. Extra-dimensional versus intra-dimensional set shifting performance following frontal lobe excisions, temporal lobe excisions or amygdalo-hippocampectomy in man. Neuropsychologia. 1991;29: 993–1006. [DOI] [PubMed] [Google Scholar]
- 19.Owen AM, James M, Leigh PN, Summers BA, Marsden CD, Quinn NP, et al. Fronto-Striatal Cognitive Deficits At Different Stages of Parkinson’s Disease. Brain. 1992;115(6): 1727–1751. [DOI] [PubMed] [Google Scholar]
- 20.Lange KW, Robbins TW, Marsden CD, James M, Owen AM, Paul GM. L-dopa withdrawal in Parkinson’s disease selectively impairs cognitive performance in tests sensitive to frontal lobe dysfunction. Psychopharmacology. 1992;107: 394–404. [DOI] [PubMed] [Google Scholar]
- 21.Owen AM, Beksinska M, James M, Leigh PN, Summers BA, Marsden CD, et al. Visuospatial memory deficits at different stages of Parkinson’s disease. Neuropsychologia. 1993;31: 627–44. [DOI] [PubMed] [Google Scholar]
- 22.Mehta MA, Owen AM, Sahakian BJ, Mavaddat N, Pickard JD, Robbins TW. Methylphenidate enhances working memory by modulating discrete frontal and parietal lobe regions in the human brain. J Neurosci. 2000;20: RC65 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Bergeron N, Dubois MJ, Dumont M, Dial S, Skrobik Y. Intensive care delirium screening checklist: Evaluation of a new screening tool. Intensive Care Med. 2001; 27(5): 859–864. 10.1007/s001340100909 [DOI] [PubMed] [Google Scholar]
- 24.Jorm AF. The Informant Questionnaire on Cognitive Decline in the Elderly (IQCODE): a review. Int Psychogeriatr Int Psychogeriatr Assoc. 2004;16(3): 275–293. 10.1017/S1041610204000390 [DOI] [PubMed] [Google Scholar]
- 25.Reis MD, Moreno R, Iapichino G. Nine equivalents of nursing manpower use score (NEMS). Intensive Care Med. 1997;23(7): 760–765. [DOI] [PubMed] [Google Scholar]
- 26.Nelson PRC, Taylor PA, MacGregor JF. Missing data methods in PCA and PLS: Score calculations with incomplete observations. Chemom Intell Lab Syst. 1996;35(1): 45–65. 10.1016/S0169-7439(96)00007-X [DOI] [Google Scholar]
- 27.Folstein MF, Folstein SE, McHugh PR "Mini-mental state". A practical method for grading the cognitive state of patients for the clinician. J Psychiatr Res. 1975;12(3): 189–198. 10.1016/0022-3956(75)90026-6 [DOI] [PubMed] [Google Scholar]
- 28.Pfoh ER, Chan KS, Dinglas VD, Girard TD, Jackson JC, Morris PE, et al. Cognitive screening among acute respiratory failure survivors: a cross-sectional evaluation of the Mini-Mental State Examination. Crit Care. 2015;19(1): 220 10.1186/s13054-015-0934-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Randolph C, Tierney MC, Mohr E, Chase TN. The Repeatable Battery for the Assessment of Neuro- psychological Status (RBANS): Preliminary Clinical Validity. J Clin Exp Neuropsychol. 2003;9800(3): 1380–3395. [DOI] [PubMed] [Google Scholar]
- 30.Wechsler D. Wechsler adult intelligence scale—Fourth Edition (WAIS-IV). San Antonio. 2008: 1–3. [Google Scholar]
- 31.Pachet AK. Construct validity of the Repeatable Battery of Neuropsychological Status (RBANS) with acquired brain injury patients. Clin Neuropsychol. 2007;21(2): 286–293. 10.1080/13854040500376823 [DOI] [PubMed] [Google Scholar]
- 32.Cheng Y, Wu W, Wang J, Feng W, Wu X, Li C. Reliability and validity of the repeatable battery for the assessment of neuropsychological status in community-dwelling elderly. Arch Med Sci. 2011;7(5): 850–857. 10.5114/aoms.2011.25561 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Taub GE, Benson N. Matters of Consequence: An Empirical Investigation of the WAIS-III and WAIS-IV and Implications for Addressing the Atkins Intelligence Criterion. J Forensic Psychol Pract. 2013;13(1): 27–48. 10.1080/15228932.2013.746913 [DOI] [Google Scholar]
- 34.van Dijk MJAM, Claassen T, Suwartono C, van der Veld WM, van der Heijden PT, Hendriks MPH. Evaluating WAIS–IV structure through a different psychometric lens: structural causal model discovery as an alternative to confirmatory factor analysis. Clin Neuropsychol. 2017;31(6–7): 1141–1154. 10.1080/13854046.2017.1352029 [DOI] [PubMed] [Google Scholar]
- 35.Beatty WW. RBANS analysis of verbal memory in multiple sclerosis. Arch Clin Neuropsychol. 2004;19(6): 825–834. 10.1016/j.acn.2003.12.001 [DOI] [PubMed] [Google Scholar]
- 36.Dickerson F, Boronow JJ, Stallings C, Origoni AE, Cole SK, Yolken RH. Cognitive functioning in schizophrenia and bipolar disorder: Comparison of performance on the Repeatable Battery for the Assessment of Neuropsychological Status. Psychiatry Res. 2004;129(1): 45–53. 10.1016/j.psychres.2004.07.002 [DOI] [PubMed] [Google Scholar]
- 37.Carlozzi NE, Kirsch NL, Kisala PA, Tulsky DS. An Examination of the Wechsler Adult Intelligence Scales, Fourth Edition (WAIS-IV) in Individuals with Complicated Mild, Moderate and Severe Traumatic Brain Injury (TBI). Clin Neuropsychol. 2015;29(1): 21–37. 10.1080/13854046.2015.1005677 [DOI] [PubMed] [Google Scholar]
- 38.Michel NM, Goldberg JO, Heinrichs RW, Miles AA, Ammari N, McDermid Vaz S. WAIS-IV Profile of Cognition in Schizophrenia. Assessment. 2013;20(4): 462–473. 10.1177/1073191113478153 [DOI] [PubMed] [Google Scholar]
- 39.Owen AM, Evans AC, Petrides M. Evidence for a two-stage model of spatial working memory processing within the lateral frontal cortex: a positron emission tomography study. Cereb Cortex. 1996;6: 31–38. [DOI] [PubMed] [Google Scholar]
- 40.Levine B, Bacopulos A, Anderson ND, Black SE, Davidson PS, Fitneva SA, et al. Validation of a Novel Computerized Test Battery for Automated Testing. In: Canadian Stroke Congress. Canadian Stroke Congress; 2013:E196–E196. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
(XLSX)
Data Availability Statement
All relevant data are within the manuscript and its Supporting Information files.