Instruments Assessing Cognitive Impairment in Survivors of Critical Illness and Reporting of Race Norms: A Systematic Review

OBJECTIVE:

To conduct a systematic review to summarize cognitive instruments being used in long-term outcome studies of survivors of adult critical illness, as well as evaluate whether these measures are reported as using patient demographic norms, specifically race norms.

DATA SOURCES:

A comprehensive search was conducted in PubMed (National Center for Biotechnology Information), Excerpta Medica dataBASE (Ovid), Psychological Information Database (ProQuest), and Web of Science (Clarivate) for English language studies published since 2002.

STUDY SELECTION:

Studies were eligible if the population included adult ICU survivors assessed for postdischarge cognitive outcomes.

DATA EXTRACTION:

Two independent reviewers screened abstracts, examined full text, and extracted data from all eligible articles.

DATA SYNTHESIS:

A total of 98 articles (55 unique cohorts: 22 general ICU, 14 Acute respiratory distress syndrome/Acute respiratory failure/Sepsis, 19 COVID-19 and other subpopulations) were eligible for data extraction and synthesis. Among general ICU survivors, the majority of studies (n = 15, 68%) assessed cognition using multiple instruments, of which the most common was the Mini-Mental State Examination. Only nine of the 22 studies (41%) explicitly reported using patient demographic norms for scoring neuropsychological cognitive tests. Of the nine, all reported using age as a norming characteristic, education was reported in eight (89%), sex/gender was reported in five (55%), and race/ethnicity was reported in three (33%). Among Acute respiratory distress syndrome/Acute respiratory failure/Sepsis survivors, norming characteristics were reported in only four (28%) of the 14 studies, of which all reported using age and none reported using race/ethnicity.

CONCLUSIONS:

Less than half of the studies measuring cognitive outcomes in ICU survivors reported the use of norming characteristics. There is substantial heterogeneity in how studies reported the use of cognitive instruments, and hence, the prevalence of the use of patient norms may be underestimated. These findings are important in the development of appropriate standards for use and reporting of neuropsychological tests among ICU survivors.

KEY POINTS

Question: This systematic review summarizes cognitive instruments used in long-term outcome studies of critically ill patients and evaluates if these measures are reported as using demographic norms, specifically race norms.

Findings: Less than half of the studies measuring cognitive outcomes in ICU survivors reported the use of norming characteristics. There is substantial heterogeneity in how studies reported the use of cognitive instruments, and hence, the prevalence of the use of patient norms may be underestimated.

Meaning: There is a critical need to have the reporting of neuropsychological instruments and subsequent scores be standardized.

More than 5 million patients are admitted annually to ICUs in the United States, of whom 1.1 million suffer from acute respiratory failure requiring mechanical ventilation (1). Among survivors of acute respiratory failure, upward of 50–75% has substantial newly acquired disabilities, including long-term cognitive impairment that is similar in severity to various Alzheimer's Disease and Related Dementias, that persist from months to years postdischarge (2–13). In a clinical setting, neuropsychological testing is used to identify and diagnose cognitive impairment. Test scores are compared with normative data derived from a representative healthy population for accurate interpretation and diagnosis. This comparison facilitates diagnosis as well as helps translate raw scores to population ranks. Since test scores may be associated with demographic factors, normative data are typically stratified using four demographic variables: age, education, gender, and race.

Understanding the use of norms in cognitive testing is particularly important in studies of long-term cognition in ICU survivors because patients are typically assessed and results are reported for a large battery of neurocognitive tests. Interpretation of these scores becomes challenging when some of these tests are normed, whereas others are not. Though this issue cannot be easily addressed without significant methodological work to standardize these scores, understanding what norms are used and reported would increase transparency and facilitate comparisons across studies. To reduce heterogeneity of measurement instruments used in long-term assessments and enable comparisons across studies, various Delphi consensus studies have been conducted over the years to identify a core outcome measurement set that is recommended for use in all clinical research evaluating ICU survivors (14, 15). Despite this, the use of varying demographic norms across different studies limits researchers from comparing these scores to draw conclusions on the consistency of the associations. It is not uncommon for researchers and practitioners to use demographic corrections for cognitive tests without knowing that they are inappropriate or unsupported by scientific evidence (16). Understanding which norms have been validated, and consistently using and reporting these norms facilitate comparison of scores across studies.

The use of race norms in cognitive tests began as practice to reduce false-positive diagnosis of impairment among minorities, but recent events such as the National Football League Players’ Concussion Injury Litigation case have demonstrated that using race norms may also cause harm in the other direction by increasing false negatives, leading to the denial of much needed health services (17–21). Race-norming is used in many tests despite these tests being initially developed and validated in primarily Caucasian populations (22, 23). Some studies have not found any significant racial effect in scores for neuropsychological tests, questioning the need for separate race norms (24). Others have called for demographically corrected norms or regression-based normative approaches that explicitly measure and adjust for social determinants of health (18, 25, 26).

The goal of this systematic review is to summarize cognitive instruments used in long-term outcome studies of critically ill patients and to evaluate if these measures are reported as using norms. We focus particularly on race norms given the growing concerns on how they are being used in algorithms in the healthcare system (27). Future studies can use these results to evaluate if such instruments need to be race-normed and additionally work on contextualizing scores from these instruments. It is crucial that cognitive measures be used and reported on in clinical studies with thoughtful deliberation and understanding of how norms, particularly race norms, may influence patient care. The rapidly growing proportion of minorities in the United States highlights the need for considering these aspects when using such tests in diverse populations (28).

MATERIALS AND METHODS

This protocol was registered in The International Prospective Register of Systematic Reviews (ID: CRD42021293575) and can be obtained at https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=293575. The Institutional review board at Vanderbilt reviewed this study (Institutional Review Board #212127) and determined it to be exempt.

Inclusion Criteria

The study inclusion criteria are provided below.

• 1) Population: Critical illness/ICU survivors greater than or equal to 18 years old

• 2) Exposure: Assessed for postdischarge cognitive outcome

• 3) Outcome: a) Instruments used to assess cognitive impairment via any mode (e.g., face to face, telephone, and virtually), and b) if patient norms were applied. Normative samples in most used cognitive instruments are described.

• 4) Time frame: Any duration of postcritical care follow-up

• 5) Publication type: Original studies

• 6) Study design: Any study design

• 7) Setting: United States. We limited our hospital setting to the United States since race is categorized and defined differently in different parts of the world.

• 8) Article language: Studies published in English

Exclusion Criteria

Publication types that are narrative reviews, abstracts only, editorials, commentaries, dissertations, government reports, books and book chapters, conference proceedings, lectures and addresses, consensus development statements (including guideline statements), case reports, and case series were excluded. Publications that use scales that were created for a specific study without any information about its structure or use as well as survey questions to assess cognition were excluded. Study populations focusing on animals and children were excluded. Participants with stroke, trauma, and major surgery were excluded. The Office of Management and Budget adopted new standards for classifying race and ethnicity with the intent to refine and standardize the way race and ethnicity was collected across all medical research. Hence, we considered only articles published January 2002 onward, after which collection of this information and use of these categories were required for research that meets the National Institutes of Health (NIH) definition of clinical research (29).

Search Strategy and Screening

A comprehensive search was conducted in PubMed (National Center for Biotechnology Information), Excerpta Medica dataBASE (Ovid), Psychological Information Database (ProQuest), and Web of Science (Clarivate) in March of 2022 for English language studies published since 2002. The search strategy was developed by the reference librarian at the Annette and Irwin Eskind Family Biomedical Library and Learning Center (R.L.W.) to find studies with a population of ICU survivors who have been assessed for postdischarge cognitive outcomes using a combination of key words and subject headings. Full search strategies can be found in the Appendix. Abstract and full-text screening was done using the Rayyan web application (Rayyan Systems Inc., MA, USA) (30). These searches are reported using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (31).

Screening was conducted in two phases. The first phase of screening consisted of the first (R.R.) and second reviewers (S.J.D.) independently reviewing the abstracts and titles compiled for potential eligibility. In the full-text screening phase, we downloaded the full text for all abstracts coded as eligible, which was then independently examined by the two reviewers to confirm eligibility. Each step of this process was documented and presented in the PRISMA flow diagram.

Data Extraction and Management

Data from the identified full-text articles were extracted independently by two reviewers. The data extraction form was developed and the following information was extracted: study design, ICU type, sample size at enrollment, sample size at follow-up, participant demographics including race, study inclusion/exclusion criteria; duration(s) of follow-up (months), cognitive outcome measure(s), cognitive subdomains measured, instruments used, how cognitive outcomes were collected (face to face, telephone, and virtually), if the use of norms were explicitly mentioned, and battery used for norming. Multiple studies from the same cohort were grouped under the parent study to avoid overlap and double counting. When multiple studies from the same cohort reported different patient norms, we reported them summatively. Since our primary goal was not to evaluate the validity of findings, but to provide a comprehensive summary of long-term cognitive instruments using race norms, studies were neither excluded nor weighted based on study characteristics or methodological rigor. We had a priori planned that we would not be conducting risk of bias for each included study.

RESULTS

Based on the PRISMA guidelines, a flowchart of the search and selection process is presented in Figure 1. Our search identified 7,530 articles. After removing 2,106 duplicates, 5,660 abstracts were screened, of which 459 full-text articles were evaluated for eligibility. A total of 98 articles were eligible for data extraction and synthesis. These 98 were grouped into unique patient cohorts and stratified by ICU subpopulations to be consistent with other systematic reviews in the field. Across the 55 unique cohorts, 22 were composed of general ICU survivors (>50% ICU patients), 14 focused on specific ICU subpopulations (Acute respiratory distress syndrome [ARDS], Acute respiratory failure [ARF], and Sepsis), and the rest focused on COVID-19 and other subpopulations. The Mayo Clinic Study of Aging cohort included publications pertaining to both the general ICU (32, 33) and ICU subpopulation survivors (34, 35), and one study (36) had ARDS patients from the ALTOS cohort and general ICU survivors from the Awakening and Breathing Controlled trial.

Figure 1.

PRISMA flow diagram. NCBI = National Center for Biotechnology Information; PsycINFO = Psychological Information Database; PRISMA = Preferred Reporting Items for Systematic Reviews and Meta-Analyses

General ICU Survivors

Table A1 (http://links.lww.com/CCX/B115) presents characteristics of the 22 studies from general ICU survivors. Of these 22, 15 (68%) had patients from mixed ICUs, three (14%) enrolled only from MICUs, and four (18%) did not report ICU types included in their study. The majority of studies (n = 15, 68%) assessed cognition using multiple tools. Single instruments used in studies include the Telephone Interview for Cognitive Status (TICS), NIH Toolbox Cognition Battery Dimensional Change Card Sort Test, Mini-Mental State Examination (MMSE), Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) (although this is a single instrument, it is comprised of multiple domains), and the Montreal Cognitive Assessment (MoCA/MoCA-Blind). These assessments were conducted in-person in 16 (73%) studies, videophone/telephone in four (18%), and the mode of assessment was not reported in two studies. Nine of the 22 studies (41%) explicitly reported using patient demographic norms for scoring neuropsychological cognitive tests. Of the nine, all used age as a norming characteristic, education was reported in eight (89%), sex/gender was reported in five (55%), and race/ethnicity was reported only in three (33%).

The most used instrument in assessing long-term cognition in general ICU survivors was the MMSE (37) (n = 11, 50%), where orientation, repetition, verbal recall, attention/calculation, language, and visual construction domains are evaluated. This was followed by Trail Making Test (TMT) A (38, 39), TMT B (38, 39), RBANS (40), Digit Symbol, Rey-Osterrieth Complex Figure, MoCA/MoCA-Blind (41), and Consortium to Establish a Registry for Alzheimer's Disease Neuropsychological Battery (CERAD-NB) (42) (Table A2, http://links.lww.com/CCX/B115). All of these tests adjusted for age, whereas five of these adjusted for education. TMT A, TMT B, and CERAD-NB adjusted for gender/sex, and only TMT A and TMT B adjusted for race/ethnicity.

ICU Subpopulations

There were 34 studies focused on specific ICU subpopulations (ARDS, ARF, Sepsis, COVID-19, and Other), where five (15%) were ARDS, four (12%) were ARF, five (15%) were Sepsis, seven (21%) were COVID-19, and the remaining 13 (38%) primarily used population-based cohort studies of community-dwelling people. Table A3 (http://links.lww.com/CCX/B115) provides the characteristics for ARDS, ARF, and Sepsis subpopulations. Since the COVID-19 and other subpopulation group is extremely heterogeneous, it would be misleading to present pooled results. Therefore, characteristics of these studies are provided in Table A4 (http://links.lww.com/CCX/B115).

Among the 14 studies on ARDS, ARF, and Sepsis patients, nine (64%) assessed cognition with multiple tests, whereas others used a single test such as Mini-Cog, MoCA, MMSE, Minimum Data Set -Cognition Scale, and TICS. The mode of assessment was mixed (telephone, in-person, and mail) in four studies: only telephone in three and only in-person in five. Two studies did not report how the assessments were conducted. Norming characteristics were reported in only four (28%) of the 14 studies, of which all used age. Race/ethnicity was not reported as a norming characteristic in any of these studies. The most used instruments in these studies were Similarities, Digit Span, Vocabulary, Logical Memory I, Controlled Oral Word Association (COWA), Hayling Sentence Completion, Verbal Fluency Test, Logical Memory II, MMSE, MoCA, and TICS.

DISCUSSION

Our systematic review identified 55 unique cohorts that studied ICU survivors with postdischarge cognition outcomes. Among studies of general ICU survivors, the majority assessed cognition using multiple tools and were conducted in-person. All general ICU and ARDS/ARF/Sepsis studies that reported the use of norming used age as a norming characteristic. Only a third reported using race/ethnicity in the general ICU survivor studies, whereas none used it in ARDS/ARF/Sepsis studies.

Our review identified several gaps in knowledge regarding how cognitive outcomes are reported in ICU survivors. We found that less than 50% of the studies in both general ICU and ARDS/ARF/Sepsis survivors explicitly reported using norming characteristics. Tests such as the MMSE use age and education norms based on published normative data (43), but many studies in our review that used the MMSE as a cognitive instrument did not explicitly report the use of patient norms. Hence, it is likely that our review underestimates the use of norms due to underreporting.

For many of these tests, there often exist multiple published norming batteries. For example, there are at least two commonly used batteries available for the TMT, of which one adjusts for age and education (44), whereas the other adjusts for age, education, sex, and race (45). The Digit Span, which measures verbal short-term and working memories, is a subtest in both RBANS as well as the Wechsler scales. Though both use age-norms, the conversion of the raw scores to the normed score would not be comparable due to different participants used in their normative sample. We also noted that the same subtest used to assess cognition could be administered as a part of different scales between studies. The COWA, a measure of verbal fluency, uses age, education, gender, and race norms. The verbal fluency test from the Delis-Kaplan Executive Functioning battery only uses age when adjusting the raw score. This inconsistency, particularly when researchers apply different norming characteristics, may lead to differences in interpretation. A further challenge is that virtually no norms exist for patients from a wide array of different race/ethnicities.When patient norms and norming batteries are not reported, it leads to decreased transparency and limits the ability to compare cognitive outcomes between studies.

This heterogeneity calls for consistent use of terminology and reporting standards for neuropsychological cognitive instruments. At a minimum, studies should clearly report: 1) the raw score, 2) the normed score, 3) norming characteristics used for each test, and 4) norming battery and version. Including the raw score alongside the normed scores allows for direct comparisons between studies that may use multiple norms as well as comparisons of scores over time (46). Among the scientific community, it is now widely recognized that race is a social construct, and that its association with poor outcomes is due to its correlation with poverty, income, and other socioeconomic determinants (47–50). The interplay between race and other determinants of health is complex, and efforts need to be made to understand how neuropsychological tools can be modified to incorporate socioeconomic determinants (25). More work should be done to validate the discriminatory ability of cognitive instruments based on race/ethnicity and to understand if it still stands after adjusting for other factors such as education. The process used to create the Core Outcome Measure set for long-term outcomes for acute respiratory failure survivors serves as a good template for establishing reporting standards for long-term cognitive measures (15). Furthermore, journals requiring the use of relevant reporting guidelines can help promote transparency and facilitate comparison of scores across studies.

There have been several systematic reviews that have identified cognitive instruments used in ICU survivors. A recent systematic review reported the frequency of cognitive impairment in ICU survivors and concluded with the suggestion that future studies focus on developing ICU-specific cognitive batteries to allow comprehensive cognitive assessment across different etiologies of critical illness (51). Two others evaluated cognitive instruments used in survivors of critical illness (8, 52). Types of cognitive domains assessed in sepsis patients and consideration of race and ethnicity relative to the cognitive trajectory were also addressed in another systematic review (53). However, to our knowledge, there have been no systematic reviews on the reporting of patient norms, specifically race norms in neuropsychological cognitive tests in ICU survivors. Understanding the prevalence of the reporting and use of norms in these tests are particularly important given that these may influence post-ICU care and access to support groups for patients. Other strengths of this systematic review include the inclusion of both general and ICU subpopulation survivors.

Despite the strengths, our review also has limitations. This review was focused on ICU survivors. To not allow differences in reviewer knowledge to bias inclusion of studies, we had a priori set a criterion that there be some mention of patients belonging to the ICU. This may have excluded some relevant studies. Many studies did not explicitly mention their norming characteristics but cited the battery used. These studies were categorized as not having patient norms explicitly reported since many of these manuals need to be purchased and may not be accessible to readers. Studies that reported using a whole battery versus the subtests were only categorized as using that battery. This may create some bias in the way we have reported the most common instruments used since some subtests are part of multiple batteries. Many cognitive instruments could have also been used as a standalone screening instrument versus as part of a comprehensive neuropsychological battery. The use of screening instruments versus comprehensive batteries has advantages and disadvantages regarding cost, time to administer, and test sensitivity, which could have possibly influenced its prevalence in the clinical environment.

We limited the hospital setting to the United States since race is categorized and defined differently in different parts of the world. However, this could potentially inflate the relevance of race/ethnicity in determining cognitive outcomes and limit the generalizability of results. In line with other systematic reviews conducted in critical care, we stratified our review by the type of critical illness insult versus the severity, which may limit the applicability of results to U.S. hospital settings.

Race norms have been in use in neuropsychology with the goal being improving the sensitivity and specificity of neuropsychological measures in detecting cognitive impairment (22, 54). Last year, the American Academy of Clinical Neuropsychology released a statement, emphasizing race as a social construct and signaling their support for the elimination of race as a variable in demographically based normative test interpretation (55). It is beyond the scope of this review to provide recommendations on whether to use race norms. However, we hope that this review contributes to understanding the degree to which norms, particularly race norms, are reported in ICU survivor studies and helps begin a conversation on the need for this adjustment as well as how to interpret such normed measures between studies.

CONCLUSIONS

This systematic review demonstrates that less than half of the studies measuring cognitive outcomes in ICU survivors reported the use of norming characteristics. There is substantial heterogeneity in how studies reported the use of cognitive instruments, and hence, the prevalence of the use of patient norms may be underestimated. There is a need to have the reporting of neuropsychological instruments and scores be standardized. Future studies should aim to report both the raw and normed score, norming characteristics, and norming battery. This will aid in the interpretation and direct comparison of cognitive scores between studies.