Challenges in Conducting Long-term Outcomes Studies in Critical Care

Abstract

Purpose of review:

Evaluating longer-term mortality, morbidity, and quality of life in survivors of critical illness is a research priority. This review details the challenges of long-term follow-up studies of critically ill patients and highlights recently proposed methodological solutions.

Recent findings:

Barriers to long-term follow-up studies of critical care survivors include high rates of study attrition due to death or loss to follow-up, data missingness from experienced morbidity, and lack of standardized outcome as well as reporting of key covariates. A number of recent methods have been proposed to reduce study subject attrition, including minimum data set selection and visits to transitional care or home settings, yet these have significant downsides as well. Conducting long-term follow-up even in the absence of such models carries a high expense, as personnel are very costly, and patients/families require reimbursement for their time and inconvenience.

Summary:

There is a reason why many research groups do not conduct long-term outcomes in critical care: it is very difficult. Challenges of long-term follow-up require careful consideration by study investigators to ensure our collective success in data integration and a better understanding of underlying mechanisms of mortality and morbidity seen in critical care survivorship.

Introduction

As more patients survive critical illness, attention has shifted towards evaluating longer-term mortality, morbidity, and quality of life. However, this increased interest has shed light on many important methodological challenges of conducting this research (Figure 1). Important challenges facing investigators are how best to retain subjects in follow-up studies, reduce threats to both internal and external validity, and achieve sufficient statistical power, obtain external funding necessary to staff and conduct such research, and incorporation of family and loved ones by which to assess the full extent of the long-term burden of critical care. Further, greater attention has been given in recent years the best methods by which to perform analyses in long-term follow-up studies of ICU subjects when primary outcome measures are often truncated by death or other events. Lastly, having a core outcome set and uniformly reporting key covariates may facilitate the intersection of various dataset to answer complex questions, however such initiatives may inadvertently dampen innovation and creativity. This article provides a review of methodological challenges encountered in conducting long-term follow-up studies of critically ill patients and evolving solutions.

Figure 1.

A pictorial representation of challenges in conducting long-term outcomes studies. This list is not exhaustive but includes limits in estimating pre-morbid status, challenges in defining a study population of interest, delineating ICU-associated as compared to hospitalization-associated risk, establishing time to outcome assessment, and determination of which outcomes measures to assess.

Mortality

Survivors of critical illness have poor long-term survival. In a recent large cohort of ICU survivors from 21 ICUS in France and Belgium, 1-year mortality was 21% [1]. The need for mechanical ventilation portends a worse long-term prognosis, with an epidemiological study of over 35,000 Medicare patients having survived ICU admission demonstrating that mechanical ventilation was associated with a 58% risk of death at 3 years [2]. The need for greater in-ICU resource intensity has been shown to necessitate greater resource utilisation after discharge leading to greater likelihood of readmission [3]; as high as 84% of ICU survivors return to the emergency department with 65% requiring hospital readmission. The majority of rehospitalizations occur within the first year following discharge, a time period during which long-term follow-up in ICU survivorship trials is concentrated [3]. Beyond one year, 25% of ICU survivors are readmitted to the ICU during the 5 years following index hospitalization. Age and pre-existing chronic illnesses are the greatest influences on predicted risk of death, with overall 1-year survival for patients over the age of 80 years being reported as low as 42% [4]. Depending on the patient population (e.g. admitting diagnosis of sepsis or more aged patients) as well as the duration of follow-up (i.e. longer duration greater number of subjects likely to die before end of study) special consideration should be taken into the calculation of sample size to ensure there will be adequate number of subjects to assess at long-term follow-up. Achieving and maintaining adequate numbers of subjects is of increasing importance as the state of research shifts from primarily descriptive reviews of mortality and functional outcomes towards attempting to ascertain causal inference.

Loss to follow-up

Low follow-up rates are an important limitation in the interpretation of long-term ICU outcomes studies, especially because the patients lost to follow-up likely have important cognitive and/or physical deficits leading to their lack of retention. Researchers continue to debate the minimum participant retention rate acceptable to preserve study validity. The social science literature suggests a minimum rate of 70–80% [5]; no such guidelines exist for studies of ICU survivorship. It seems however that in both longitudinal cohorts and randomized controlled trials with long-term follow-up, investigators are achieving on average rates of 70–80%. For example, Herridge and colleagues in the Towards RECOVER study (n=391) evaluated 90% of subjects in hospital, and then 71%, 74% and 83% of eligible ICU survivors at 3-, 6- and 12-months respectively [6]. These follow-up rates were achieved by a well-experienced team with expertise in retention strategies for long-term follow-up. Comparing follow-up in the BRAIN-ICU [7] and COGWELL [8] studies both are studies of long-term cognition after ICU discharge one may get an idea of the size of team required to achieve such numbers (Figure 2). In the BRAIN-ICU study at 1-year follow-up, a large team of over 10 dedicated research personnel were able to achieve follow-up rates of 94%. In contrast, a single researcher and part-time research assistant in the COGWELL study achieved follow-up rates of only 73% (62% evaluated).

Figure 2.

Consort diagrams of two studies (A. BRAIN-ICU study [7] and B. COGWELL study [8] unpublished) of long-term cognitive follow-up of varying numbers of both participants and study personnel. All efforts should be made to limit study attrition from loss to follow-up to minimize bias, enhance generalizability as well as study validity. Strategies to mitigate loss to follow-up are heavily dependent on numbers of study personnel, dedicated time to allocate to retention strategies, and adequate funding to enact such aggressive strategies.

It is common to underestimate the number of study personnel required to ensure adequate follow-up, further, to misjudge the financial expense needed from external funding for such ventures. Strategies suggested for subject retention include using a systematic method for patient contact, scheduling, and cohort retention monitoring (e.g. obtaining multiple contacts for each participant, including two contacts not residing with the participant); minimizing participant burden through characteristics and procedures of follow-up study clinics (e.g. offering flexible clinic appointments); and, specifically training and managing study personnel (e.g. assigning one primary physician to each participant or hiring culturally sensitive staff with strong interpersonal skills) [9, 10], effective incorporation of these measures requires time, dedication, and training. Most importantly, adequate follow-up requires a large budget for adequate staffing and to allow staff the time to pursue these resource-intense endeavours to limit study attrition [11].

Expanding on the need to further understand and develop strategies to maximize subject retention, future strategies might include off-site or home visits, either in person or via available technologies. Waters and colleagues recently analyzed patient-related factors for requiring off-site or home visits in follow-up of subjects participating in either the Toronto 5-year Acute Respiratory Distress Syndrome (ARDS) outcomes study and RECOVER program [12]. Patients with the most significant functional dependency and medical complexity were more likely to require an off-site or home visit for follow-up [12]. Further study of the need for off-site or home visits may provide not only an opportunity to limit study attrition rates but might identify and therefore allow investigators to address issues that might prevent study follow-up, and possibly even readmission or death.

Although the current evidence supports the effectiveness of transitional care models in reducing hospital readmissions, the component of transitional care delivery from which patients are most likely to benefit has yet to be determined [13]. In a randomized controlled trial of elderly patients discharged from hospital, visits from an advanced practice nurse during the first four weeks after hospital discharge significantly delayed the time to first readmission and reduced the total number of multiple readmissions [14]. In a pair of studies of patients with chronic congestive heart failure, those patients who received home visits within 7 to 14 days after discharge had fewer unplanned readmissions and longer survival [15]. In a retrospective cohort of Medicaid recipients requiring complex care for their chronic conditions, home visits reduced the likelihood of a 30-day readmission by almost half, as compared to less intensive forms of nurse-led transitional care support [16]. Higher risk patients seemed to experience the greatest benefit in terms of number of inpatient admissions and total cost of care 6 months following discharge [16]. Outside of home visits, thinking of other solutions to meet the needs of follow-up subjects may need to be more creative. For example, a research study “car service” may facilitate one making in-person appointments. For an ICU survivor who lived 27 kilometres from the follow-up clinic and lacked the confidence in her own motor dexterity and response time to drive on the highway, our study team drove out to pick her up so that she could undergo a full cognitive evaluation and an electroencephalography study. By her second follow-up, her motor dexterity improved and as had her confidence, as a result she was enthusiastic to report that she was able to transport herself to clinic for her 12-months follow-up visit. Further study of the needs of patients unable to attend in-person follow-up is desperately needed to investigate the benefit of such a laborious endeavor for researchers and to minimize the inherent bias introduced by high rates of loss to follow-up.

Missing data points

Subjects who are not necessarily lost to follow-up but have missing data from incomplete study visits also contribute to decreased precision and statistical power, thus introducing selection bias. Many ICU survivors have poor baseline health and health related quality of life, and often face new or worse physical, psychological and cognitive morbidity after hospitalization [17]. These impairments may present difficulties for subjects to participate in longitudinal studies as follow-up assessments tend to be lengthy and involve multiple neuropsychological and physical surveys as well as performance-based tests [18–23]. As an example, to perform the minimum acceptable Core Outcome Measurement Set, comprising 42 questions, the estimated completion time is 12 minutes. Including the optional Montreal Cognitive Assessment instrument with the Short Form-36 raises the total number of questions to 91 and requires an estimated 26 minutes [21]. These times of course do not take into account initial time spent speaking with the subject, and possibly a family member or friend, about how things have been going since hospital discharge or the reassurance needed to complete the assessments if significant functional, cognitive or mood impairment.

A real-world illustration of how morbidities experienced by this patient population may influence subject performance is shown in Figure 3. Although a minimum outcome dataset is an important initiative to help guide measurement selection, it may also be seen as a challenge in and of itself in this type of research. As investigators we need to maintain a degree of adaptability to meet the needs of our patients, while they are trying to provide us with the answers to ICU survivorship. We need to remain sensitive to the individual trajectories of disability and ensure that we don’t contribute to the frustrations that our patients face during recovery. Sensitivity to the needs of our patients should always supersede the need for a complete dataset. Further, one might argue that in many settings of long-term outcomes research, a minimum acceptable dataset may pose a very real set of problems that could stifle discovery through the restriction of research questions or possibly limitations in a more mechanistically driven sequence of measures. The focus of the Core Outcome Measurement Set is very heavy in functional measures and one might argue, giving an option for a cognitive screening tool where patients have made it clear that the maintenance of cognitive function after critical illness is a leading priority is in and of itself a limitation. We do however recognize the importance of standardization for making between study comparisons, however we caution investigators to use such measurement sets as a guide and to prioritize what is required in the interests of your patients and for answering the research question of interest.

Figure 3.

56-year-old university educated woman status post lung transplant. After hospital discharge, she reported difficulties with concentration and found reading to be overwhelming. She started to do word searches in the hopes that that would mediate some of her experienced symptoms. Shown on the left is a puzzle completed around her 6-months follow-up appointment; as can be seen her fine motor dexterity had not returned to her baseline at that time. Outside of this finding, her cognitive testing was within normal limits. Shown on the right is a puzzle she again completed but this time proximate to her 1-year follow-up. At this time, she not only subjectively felt that her concentration had improved but also her confidence in her daily activities, both of which seemed to lag behind her functional recovery.

A greater understanding of the barriers to complete data collection during follow-up visits may assist investigators in anticipating and tailoring their follow-up efforts. These factors are just starting to be elucidated. Recently, Heins and colleagues evaluated risk factors for missed assessments over a 6 to 24 months follow-up period in survivors of acute respiratory distress syndrome; number of dependencies in activities of daily living at hospital discharge was associated with higher odds of missed assessments at the initial visit of 3 months follow-up [24]. Variables associated with higher odds of missed assessments at subsequent visits were two or more dependencies of instrumental activities of daily living at hospital discharge and having missed assessments at the prior follow-up [24]. Just like adaptive trial design for mortality outcomes can reduce missing assessments, longitudinal clinical research studies may benefit from modifying parameters of follow-up based on subject performance. For example, if a subject has significant functional impairment at 3-months, future outcome measures might be modified such that those most important to the study are evaluated first; this then becomes the dataset carried through for that subject until the end of the trial.

Outcome measure selection across studies

Heterogeneity in the outcomes and measurement instruments used in ICU survivor studies has created a major barrier in synthesizing results. In a recent review by Turnbull et al., more than 300 original research publications on ICU survivors’ outcomes after hospital discharge since 2000 had highly variable time to follow-up and measures [25]. A further scoping review found that 250 unique measurement instruments were used between 1970 and 2013 to assess ICU survivors after hospital discharge [21]. As of May 2019, there are 16 projects registered with the Core Outcome Measures in Effectiveness Trials (COMET) Initiative (http://www.comet-initiative.org/).

Other than establishing a core set of outcomes and measurement instruments, work must also focus on standardizing definitions of clinically important difference. As an example of the difficulty in defining a clinically important difference, we can draw on our own experience in studying cognitive outcomes. Our attempt to compare rates of cognitive impairment across studies proved difficult as different batteries have been employed and widely divergent definitions of impairment used (Table 1). We found no consistency in the definitions used for impairment (e.g. 1 SD, 1.5 SDs or 2 SDs below the normative mean). Further, there was inconsistent reporting of whether cut-off scores for impairment were adjusted for age, education or premorbid ability.

Table 1.

Definitions used and covariates reported in studies of cognitive impairment after ICU discharge

Study Dates	Study Design Population	Cognitive Test(s)	Definition of abnormality	Quality of outcome measured	Covariates reported in Table 1
Ambrosino 2002 [31] January 1996 –December 1998	Prospective; controlled; consecutive 63 COPD patients requiring MV; control group 34 stable COPD patient on LT oxygen therapy admitted to inpatient pulmonary rehabilitation program	Mini Mental Status Exam (MMSE)	MMSE < 24	2 (screening test)	Sex, age, BMI, PFTs, MIP/MEPs, PaCO2/PaO2
Baumbach 2016 [32] September 2014 – March 2015	Prospective cohort; self-selected by response to contact via mail 127 patients aged 18 to 85 years; mixed MSICU with minimum ICU LOS of 24 hrs	Functional Assessment of Cancer Therapy-Cognitive Function Adapted (FACT-Cogadapted) Informant Questionnaire of Cognitive Decline in the Elderly (IQCODE)	Likert scale; no clear definition of CI	1 (self-reported)	Sex, age, education, employed (yes/no), APACHEII, hospital LOS, ICU LOS, days of MV, admission diagnosis
Bruck 2018 [33] January 2012 – February 2013	Prospective cohort; patients with sepsis who were part of the PRE-DELIRIC study who responded to questionnaire 125 patients; mixed MSICU with minimum ICU LOS of 24 hrs	Cognitive Failures Questionnaire (CFQ)	Total CFQ score > 25	2	Sex, age, history of cognitive impairment, alcohol use, drug use, diabetes, vascular disease, cardiac disease, APACHEII, type of ICU admission
Calsavara 2018 [34]--	Prospective cohort; severe sepsis or septic shock during ICU stay 33 patients; 16 patients at 1 year	Consortium to Establish Registry for Alzheimer’s Disease (CERAD) battery	--	3 (Neuropsychological testing)	Sex, age, education, APACHEII, SOFA, ICU LOS, lab parameters (hemoglobin, Cr, lactate, CRP, glucose), site of infection, comorbidity index, IQCODE, cumulative doses of analgesia/sedatives/inotropes/pressors, steroid exposure, dialysis, days of MV
Christie 2006 [35]--	2 cross-sectional studies 1) 79 ARDS patients; internet ARDS support site 2) 34 ARDS patients discharged [in-person cognitive interviews]	Neurobehavioral Cognitive Status Exam (NCSE): Judgment Wechsler Memory Scale )WMS)-III: Digit Span; Letter-Number Sequencing; Logical Memory I/II; Similarities Controlled Oral Word Association Hayling Sentence Completion Test	2 test scores ≥1 SD or more below the population norm or a single test score ≥1.5 SDs below population norm	3	Sex, age, education, race, marital status, alcohol use, smoking status, comorbid conditions, prehospitalization function, prehospitalization cognition, prehospitalization depression
Chung 2017 [36] June 2014 – May 2015	Retrospective; consecutive 30 patients from MICU, CCU and SICU	Mini-Cog test	Recall of 0 items or recall of 1–2 items with an abnormal clock face	2	Sex, age, high school graduate (yes/no), type of ICU, reason for admission, GCS at admission, SOFA, days of MV, days delirious, ICU LOS
Davydow 2013 [37] September 2010 – August 2011	Prospective cohort; consecutive 150 nontrauma patients without cognitive impairment or dementia diagnosis who were admitted to an ICU for > 24 hrs; 120 patients completed FU	Telephone Interview for Cognitive Status modified (TICSm) Mini International Neuropsychiatric Interview	--	2	Sex, age, education,
De Azevedo 2017 [38] March 2014 – February 2015	Prospective cohort; consecutive 413 adult patients mechanically ventilated	Digit span forward and backward Rey Auditory Verbal Learning Test (RAVLT) Clock-drawing test Verbal fluency test MMSE	Mild or moderate impairment if 2 test scores 1.5 SDs below the mean or test scores 2 SDs below the mean; severe CI if ≥3 test scores 1.5 SD below the mean	3	Sex, age, education, APACHE IV, SOFA, days of MV, use of sedative agents, admission diagnosis, delirium (yes/no), ICU LOS
De Oliveira 2014 [39] February 2001 – March 2009	Prospective cohort; consecutive 234 patients with severe TBI (GCS > or = 8); 46 were evaluated at 1 year	WMS-III Logical Memory First Recall; Logical Memory I/II; Visual Reproduction I/II/ III RVALT Total; Retention; Delayed Memory Wechsler Adult Intelligence Scale (WAIS)-III-Digit Span; Vocabulary; Similarities; Block Design; Letters Fluency; Category Fluency	--	3	Sex, age, education, hand dominance, glucose, CT head characteristics, SAH (yes/no), multisystem trauma (yes/no), type of trauma, GCS, pupils
De Rooij 2008 [40] January 1997 – December 2002	Retrospective; consecutive 164 patients >80 yrs; mixed MSICU who underwent elective surgery	IQCODE-SF Katz Activities of Daily Living(ADLs) EQ-5D	Score > 3.9 on IQCODE-SF severe CI; 3.1–3.8 mild-moderate CI	2	Sex, age, education, social status, cardiopulmonary resuscitation, GCS after 24 hrs, SAPS II, APACHE II, planned or unplanned admission, BMI, MV (%), ICU LOS
Duggan 2017 [41]--	Prospective; consecutive 826 patients mixed MSICU population for respiratory failure, cardiogenic shock, or septic shock	Behavior Rating Inventory of Executive Function–Adult (BRIEF-A) Trail Making Test B Beck’s Depression Index (BDI)-II Short Form(SF)-36 ADLs Instrumental ADLs (IADLs)	BRIEF-A impairment defined as t-score ≥ 65; Trails B impairment is defined as t-score ≤ 35	3	Sex, age, race, marital status, employment status, baseline clinical status, pre-existing CI, history of depression, history of nondepressive mental illness, comorbidity index, admit diagnosis, ICU type, SOFA, days of MV, septic (yes/no), stroke in hospital (yes/no), ICU LOS, discharge destination
Duning 2010 [42] January 2004 – December 2007	Case-control 74 patients 18–80 yrs of age; 37 patients had at least one hypoglycemic event during SICU admission	MMSE Boston Naming Test Nuernberg Gerontopsychological Inventory Digit symbol substitution Color word interference tasks WMS (revised) Regensburg Word Fluency Test Trail Making test Rey-Osterrieth Complex FigureRAVLT Recognition	--	3 (compared to 2)	Matching criteria: sex, age, SAPS II, yr of ICU treatment; disease related criteria: type of OR, CP resuscitation, DM I or II, ICU LOS, mean am blood glucose, duration of sedation (<3 days, 3–7 days, 1–2 weeks, > 2 weeks), PF ratio, CV failure (pressors, MV assist device), renal failure, hepatic failure, medications (steroids, immunosuppressants)
Ehlenbach 2010 [43] 1994 – 1996 (2581 participants); 2000 – 2002 (additional 811 individuals)	Prospective; cohort Enrolled patients are evaluated q2yrs; 41 persons were hospitalized for critical illness	Cognitive Abilities Screening Instrument (CASI)	<86 prompted a full standardized clinical exam	2	Sex, age, race, education, CAD (yes/no), CV disease (yes/no), pulm dz (yes/no), D< (yes/no), renal dz, malignancy, follow-up time, study visits
Girard 2010 [44] October 2003 – March 2006	Nested in RCT; prospective 76 MICU patients	MMSE Digit span Trail Making tests A and B Digit Symbol Coding Rey-Osterrieth Complex Figure RVALT	2 cognitive test scores 1.5 SDs below the mean; one cognitive test score 2 SDs below the mean	3	Sex, age, education, APACHE II, admission diagnoses (sepsis, ARDS, MI/CHF, COPD/asthma, altered MS, hepatic or renal failure, malignancy, alcohol withdrawal, other), delirium in ICU (prevalence, duration), sedation exposure (benzos, opiates, propofol)
Girard 2018 [45] March 2007 - May 2010	Multicenter; prospective; cohort 586 patients managed in a medical or surgical ICU with respiratory failure, septic or cardiogenic shock, or both	RBANS MMSE Trail Making Test B IQCODE	--	3	Sex, age, race, education, short IQCODE, pre-existing CI, comorbidity index, admission diagnosis, APACHEII, SOFA, days of MV, dexamethasone/benzodiazepine/opioid/ propofol exposure, ICU LOS, hospital LOS, Framingham stroke risk profile, duration of severe sepsis, no. of 15 mins intervals with hypoxia
Godbolt 2012 [46] January 2010 – June 2011	Multicenter; prospective; cohort 110 patients with severe TBI	Barrow Neurological Institute Screening of Cognitive function	Total score < 2 SD below the mean	2	--
Guillamondegui 2011 [47] July 2006 – June 2007	Prospective; cohort 108 patients with moderate-severe TBI; patients with hypoxemic event (SaO2 < 85%) within first 48 hrs of admission	Employment questions and battery of validated neuropsychological testing instruments	2 test scores 1.5 SDs below the mean or 1 test score 2 SDs below the mean	3	Sex, age, ISS, ED GCS, ED pulse, ED SBP, transfusion, MV days, ICU LOS, SpO2 < 90% or 85% for > 5 mins; delirium (prevalence)
Hope 2013 [48] January 2003 - December 2007	Prospective; consecutive; cohort 385 adults admitted to a respiratory care unit for treatment of chronic critical illness; undergone elective tracheostomy for weaning	Validated telephone version of the Confusion Assessment Method (CAM)	Three possible values: dead, alive with brain dysfunction, and alive without brain dysfunction	2	Sex, age, race, cognitive impairment at baseline, residence prior to hospitalization, FIM at admission, IADLs at admission, admission diagnosis, type of ICU, ICU LOS, APACHE II, APS, comorbidity index
Hopkins 1999 [49] February 1994 – July 1988	Prospective; consecutive; cohort 67 ARDS survivors	WAIS-R WMS-R RAVLT Rey-Osterrieth Complex Figure Trail Making Tests A and B Verbal Fluency (verbal production)	2 test scores > 1.5 SD or 1 test score > 2 SD below the normative population mean	3	Sex, age, education, ICU LOS, duration MV
Hopkins 2004 [50]--	Prospective; consecutive; cohort 66 ARDS survivors	WAIS-R WMS-R RAVLT Rey-Osterrieth Complex Figure Trail Making Tests A and B Verbal Fluency (verbal production)	<80%, <85% or <90% (compared to normative data)	3	Sex, age, education, hospital LOS, ICU LOS, duration MV, APACHE II, mean MOF score, PF ratio, PaO2 at time of enrolment, mean PaO2, FiO2 at time of enrolment, days from ARDS onset to enrolment in 1 yr outcome study
Hopkins 2005 [51] February 1994 – December 1999	Prospective; consecutive; cohort 66 ARDS survivors (low vs. high tidal volume ventilation study)	WAIS-R WMS-R RAVLT Rey-Osterrieth Complex Figure Trail Making Tests A and B Verbal Fluency (verbal production)	2 or more test scores > 1.5 SDs or 1 test score > 2 SDs below normative population mean	3	Sex, age, education, race, number of ARDS RFs, duration MV, ICU LOS, hospital LOS, at study enrolment (APACHE II, MOF score, PF ratio, FiO2, PaO2); total ICU stay (mean MOF score, mean PF ratio, Mean FiO2, mean PaO2)
Hopkins 2005 [52]--	Prospective; cohort 32 patients having received >5 days of MV	Neuropsychological testing--	≥2 test scores > 1.5 SDs or 1 test score > 2 SD below normative population mean	3	Sex, age, education, Charlson comorbidity index, number of ARDS RFs, duration MV, ICU LOS, hospital LOS, at study enrolment (APACHE II, MOF score, PF ratio, FiO2, PaO2); total ICU stay (mean MOF score, mean PF ratio, Mean FiO2, hrs oximetry SaO2 < 90%, hrs MBP < 60 mmHg, hrs MBP < 50 mmHg, days reciving either sedatives, narcotics or paralytics)
Hopkins 2010 [53] February 1994 – December 1999	Prospective; consecutive; cohort 66 ARDS survivors (low vs. high tidal volume ventilation study)	WAIS-R WMS-R RAVLT Rey-Osterrieth Complex Figure Trail Making Tests A and B Verbal Fluency (verbal production)	2 or more test scores > 1.5 SD or 1 test score > 2 SD below normative population mean using age, gender and education corrected t-scores	3	Sex, age, education, hospital LOS, DM I or II, total corticosteroid dose, at study enrolment (APACHE II, MOF score, PF ratio, FiO2, PaO2); total ICU stay (duration MV, ICU LOS, Mean FiO2, mean PaO2, hrs oximetry SaO2 < 90%, total insulin units/ICU hours, total potassium dose, mean blood glucose, lowest glucose, highest glucose)
Hughes 2017 [54] March 2007 - May 2010	Multicentre; prospective; cohort 1040 patients with major noncardiac surgery during hospital admission and with nonsurgical medical illness	RBANS Trail Making Test B	--	3	Sex, age, race, education, SES, IQCODE, clinical frailty score, functional activities questionnaire, comorbidity index, Framingham stroke risk profile, SOFA< APACHE II, sepsis in ICU, days of MV, ever delirious, coma, sedative or analgesia use, ICU LOS, hospital LOS
Hughes 2018 [55]	Multicentre; prospective; cohort 419 adults admitted to a MICU or SICU with respiratory failure and/or shock	RBANS Trail Making Test B Katz ADLs Functional activities questionnaire	--	3	Sex, age, race, education, SES, IQCODE, clinical frailty score, functional activities questionnaire, comorbidity index, Framingham stroke risk profile, SOFA< APACHE II, sepsis in ICU, days of MV, ever delirious, coma, sedative or analgesia use, ICU LOS, hospital LOS
Iwashyna 2010 [56] 1998 – 2006	Prospective; nonconsecutive; cohort Enrolled patients are evaluated q2yrs; survivors of severe sepsis	35-point scale; tests of memory, serial 7 subtractions, naming and orientation ADLs IADLs	--	2	Sex, age, education, race, LOS, required MV, required dialysis, used ICU, underwent major surgery, Charlson score, organ dysfunction score, acute conditions (CV dysfunction, neurologic dysfunction, hematologic dysfunction, hepatic dysfunction, renal/respiratory dysfunction), baseline cognitive impairment, baseline functional disability
Jackson 2003 [57] February 2000 – May 2001	Prospective; cohort; consecutive 34 patients; MICU and CICU; requiring MV	Modified Blessed Dementia Rating Scale (mBDRS) MMSE Digit Symbol Coding Thurstone Word Fluency Letter Number Sequencing Sequencing Verbal Paired Associates Digit Symbol Paired Recall Recall (Faces) Rey-Osterrieth Complex Figure	2 test scores > 2 SDs below the norm-referenced mean or 3 test scores ≥1.5 SD below norm-referenced mean	3	Sex, age, education, race, ADL, APACHE, Charlson, SOFA, Admission diagnosis
Jackson 2007 [58] January 2003 – December 2003	Prospective; nonconsecutive; cohort 58 trauma patients without ICH or focal neurologic deficits or moderate to severe TBI	Digit span Digit symbol FAS IQCODE-SF MMSE RAVLT Rey-Osterrieth Complex Figure Trail Making Tests A and B Katz ADLs	≥2 test scores > 1.5 SDs or 1 test score > 2 SDs below normative population	3 (compared to 2)	Sex, age, education, race, ISS, type of trauma, mental health history, employment status
Jackson 2010 [22] October 2003 – March 2006	Randomised controlled trial of SAT/SBT 80 patients requiring MV for > 72 hrs	Digit span Digit symbol IQCODE-SF MMSE RAVLT Rey-Osterrieth Complex Figure Trail Making Tests A and B Verbal Fluency test Katz ADLs	≥1.5 SDs below the mean on ≥2 of the nine cognitive tests or scored ≥2 SDs below the mean on ≥1 of the nine cognitive tests	3	Sex, age, APACHE II, SOFA, Admission diagnosis, pre-existing cognitive impairment, baseline ADL, baseline IADL, pre-enrolment sedative exposure, sedative exposure, lorazepam equivalents, fentanyl equivalents, propofol
Jackson 2011 [59] July 2006 – June 2007	Prospective; cohort 108 patients with moderate to severe trauma; no ICH	Digit span Digit symbol Verbal Fluency test (FAS) IQCODE-SF MMSE RAVLT Rey-Osterrieth Complex Figure Trail Making tests A and B	≥2 test scores > 1.5 SDs or 1 test score > 2 SDs below normative population	3	Sex, age, education, race, ISS, admission GCS, long-bone fracture, concussion, ICU LOS, hospital LOS, MV status, duration of MV, CAM+ days, type of trauma
Juan 2018 [60] July 2012 - May 2015	Prospective; cohort 50 survivors included from a prospective cohort of 138 patients admitted at the ICU for cardiopulmonary arrest	Naming subtest of the Lexis battery California Verbal Learning Test Doors and People test Digit span forward subtest of WAIS-IV Block tapping WMS-R Five-points test Digit-symbol subtest of the WAIS-IV Alert and Divided attention subtests of the Test battery Trail Making and Stroop tests from the GREFEX battery	z score less than or equal to –1.65 SDs of the mean	3	Sex, age, cardiac etiology arrest, out of hospital cardiac arrest, time to return of spontaneous circulation, first shockable rhythm, therapeutic hypothermia (yes/no)
Jones 2006 [61] March 2003 – November 2004	Prospective; cohort 30 long-stay, MV patients	Cambridge Neuropsychological Test Automated Battery (CANTAB)	≤25 percentile compared with an age-, sex-matched control population	3	Sex, age, APACHE II, ICU LOS, diagnostic groups (peritonitis, pneumonia, asthma/COPD, sepsis, ARDS, trauma)
Larson 2007 [62] February 1994 – December 1999	Interventional trial; higher vs lower tidal volume ventilation strategy 66 ARDS survivors	WAIS-R (FSIQ; Vocabulary; Block Design) WMS-R (Attention Index; Verbal; RAVLT; Visual; Rey-Osterrieth Complex Figure) Trail Making Test B	≥2 test scores > 1.5 SDs or 1 test score > 2 SDs below normative population mean values using age, gender and education	3	Sex, age, hospital LOS, ICU LOS, duration of MV, APACHE II, Charlson Comorbidity index, mean MOF score, mean PaO2, mean FiO2, PF ratio, days receiving sedatives/narcotics/paralytics
Lippert-Gruner 2006 [63]	Prospective; cohort 41 patients with severe TBI	Neurobehavioural Rating Scale	--	2	Sex, age
Maley 2016 [64] January – May 2014	Prospective; mixed-methods investigation 43 survivors from two MICUs	Health Utilities Index - 3 cognitive questions Hospital Anxiety and Depression Scale (HADS) Connor-Davidson Resilience Scale Life-Space Questionnaire	--	1	Sex, age, race, marital status, no. of hospitalization in prior yr, comorbidity score, days of MV, sepsis LOS, shock LOS, ICU LOS, hospital LOS, disposition destination
Marquis 2000 [65]	Prospective; parallel controlled cohort 33 ARDS survivors; 24 critically ill controls	Trail Making Test B Symbol Digit Modalities test Test of Everyday Attention (Elevator Counting with Distraction)	--	3	--
Mikkelsen 2009 [66]	Cross-sectional 79 self-reported ARDS survivors	NCSE (Orientation, Judgment) WMS-III (Digit Span, Letter – Number Sequencing, Logical Memory, Similarities) Hayling Sentence Completion Test Controlled Oral Word Association Test	≥2 test scores > 1 SD or 1 test score > 1.5 SDs below normative population	3	Sex, age, education, employment status, hospital LOS, precipitating factors (pneumonia, surgery, sepsis, trauma, other)
Mikkelsen 2012 [67] June 2000 – October 2005	Prospective; multicenter; cohort 75 ALI survivors	Neuropsychological test battery (45–60 minutes); nil other specifics reported	1 test score > 2 SDs below normative population	3	Sex, age, race, primary lung injury, coexisting conditions (none, DM, HIV/AIDS, cirrhosis, solid tumor, leukemia, lymphoma, immunosuppression), APACHE III, GCS, MAP, vasopressor use, PF ratio, conservative fluid strategy, PAC
Mitchell 2016 [68] November 2011 – December 2014	Prospective; cohort 148 adult surgical, medical and trauma patients enrolled; 88 tested at 3-months and 79 tested at 6-months	RBANS Trail Making Test Part B MMSE	Classified as severely impaired if they scored 1.5 SDs below the mean on ≥3 of the index scores or 2 SDs below the mean on ≥2 of the index scores	3	Sex, age, education, admission diagnosis, APACHE II, APACHE III, ICU LOS, hospital LOS, Propofol/benzodiazepine/opioid dose, days of MV, delirium in ICU days
Needham 2013 [19] July 2008 – May 2012	Multicentre; prospective; cohort 174 patients	Controlled oral word association (COWA) Digit span Hayling sentence completion Logical memory I/II Similarities	<1.5 SDs on any of tests	3	Sex, age, race, high school education (yes/no), BMI, steroids (yes/no), living independently at home prior to admission (yes/no), employed (yes/no), SF-36, functional performance inventory score, comorbidity score, comorbidities at admission, critical illness characteristics (pneumonia, sepsis, baseline shock, baseline PF ratio, APACHE II, days of MV, hypoglycemia, steroids, insulin, benzodiazepines, NMBs, narcotics, ever coma, ever delirious, ICU LOS, hospital LOS)
Pandharipande 2013 [7] March 2007 – May 2010	Multicentre; prospective 467 adults admitted to a medical or surgical ICU with respiratory failure, cardiogenic shock, or septic shock	RBANS Trail Making Test B	1.5 and 2 SDs below the population means	3	Sex, age, race, ICU type, education, short IQCODE, comorbidity score, APACHE II, SOFA, admission diagnosis, days of MV, days delirious, coma (yes/no), hospital LOS, use of sedatives/analgesia in ICU
Pasternak 2008 [69] February 2000 – April 2003	Posthoc analysis of data from Intraoperative Hypothermia for Aneurysm Surgery Trial 878 patients with SAH; underwent aneurysm surgery	Benton Visual Retention test Controlled Oral Word Association Rey-Osterrieth Complex Figure Test Grooved Pegboard test Trail Making test Glasgow Outcome Score (GOS)	1 test score > 2 SDs below normative population	3	Blood glucose at aneurysm clipping, age, sex, race, BMI, preoperative hx (DM, HTN, smoking, time from SAH to induction), WFNS, Fisher grade, NIHSS, preoperative Rankin score, hydrocephalus on initial CT
Pfoh 2015 [70] July 2008 – May 2012	Cross-sectional secondary analysis of data from two prospective studies of acute respiratory failure patients requiring mechanical ventilation in an ICU	MMSE COWA Logical memory I/II Digit span total score, forward and backward	MMSE conservative cutoff score of <24	2	Sex, age, education, employed, comorbidity score, psychiatric condition, severity illness score, days of MV, ICU LOS, hospital LOS
Pierrakos 2017 [71] January 2013 - January 2014	Prospective; consecutive; cohort 28 patients with sepsis	MMSE IADLs	--	2	Sex, age, APACHE II, IADL, pCO2, MAP, septic shock, days of MV, sedation, relapsing infection (yes/no), delirium (yes/no)
Rothenhausler 2001 [72] January 1985 – January 1995	Prospective; consecutive; cohort 46 ARDS survivors	Short Cognitive Performance test (SKT)	SKT total scores (profound cognitive impairment: 24–27; severe: 19–23, moderate: 14–18, mild: 9–13; subthreshold: 5–8)	2	Sex, age, RF for ARDS (trauma, sepsis, pneumonia, other)
Sacanella 2011 [73] --	Prospective; consecutive; Cohort 112 elderly patients electively admitted to MICU	MMSE IQCODE	MMSE < 24	2	Sex, age, APACHE II, SOFA, ICU LOS, cardiac dx, respiratory dx, severe sepsis, CV dx, other medical dx, % MV, % RRT, OMEGA score, Charlson index
Sakuramoto 2015 [74] July – December 2009	Prospective; consecutive; cohort 79 adults admitted to MICU or SICU	MMSE	MMSE < 24	2	Sex, age, comorbidity score, vision deficits, hearing deficits, mBDRS score, APACHE, SOFA, days of MV, ICU LOS, hospital LOS, admission diagnosis
Schielke 2005 [75] January 1999 – June 1999	Prospective; consecutive; cohort 27 patients treated for ischemic stroke requiring MV	MMSE National Institutes of Health Stroke Scale Barthel Index modified Rankin Scale	MMSE < 24	2	--
Semmler 2013 [76] January 2004 – August 2006	Two center; prospective; non-consecutive; cohort 25 survivors of sepsis; 19 non-septic ICU survivors	Neuro Cognitive Effects (NeuroCogFx) Trail Making tests A and B Auditory Verbal Learning Test Rey-Osterrieth Complex Figure Test	> 1.5 SD from z differences scores	3	Age, estimate premorbid verbal ability, APACHE II, SOFA, ICU LOS, duration MV, electrolyte levels (Na and K), PF ratio (max), creatinine (max), HCT (max), MAP < 70 mmHg, ARDS %, surgery (emergent or elective), admission diagnosis, comorbid medical disorders (cardiac, respiratory, live, DM, immunosuppressed, cancer, renal dx, CV dx, GI dx, multiple disorders), drugs (sedatives, analgesics, vasopressor, other drug)
Suchyta 2004 [77]	Prospective; cohort 30 ARDS patients	--	≥2 test scores that were > 1 SD for mild, >1.5 SDs for moderate or ≥2 SDs for severe cognitive impairment	3	Sex, age, education, APACHE II, hospital LOS, PF ratio, RF for ARDS
Suchyta 2010 [78] July 2003 – June 2004	Prospective; non-consecutive; cohort 46 MSICU patients	--	--	--	Duration MV, ICU LOS, hospital LOS, APACHE II at ICU admission, % ARDS, admission diagnosis, co-morbid dx (cardiac, respiratory, liver, DM, immunosuppressed, cancer, renal dx, CV dx, GI dx, multiple disorders); hrs SpO2 < 90%, hrs MAP < 60 mmHg, sedatives (total dose ICU admission to scan) including lorazepam/fentanyl/morphine/midazolam/ propofol/hydromorphone, steroids
Sukantarat 2005 [79] April 2000 – March 2003	Prospective; consecutive; cohort 51 MSICU patients	Hayling Sentence Completion test Modified Six Element test Raven’s Standard Progressive Matrices	Compared to percentiles of population norms	3	Sex, age, LOS, duration MV, APACHE II, TISS points
Teeters 2011 [80] 2004 – 2008	Prospective; cohort 387 elderly patients admitted to ICU	Clinical Dementia Rating Scale FAQ Neurologic evaluation Neuropsychiatric testing	Expert consensus	3	Sex, age, APACHE III
Tembo 2012 [81] --	Qualitative 12 MSICU patients	Face-to-face interviews	Self-reported	2	--
Tobar 2009 [82] September 2008 – April 2009	Prospective; cohort 8 MICU patients	MMSE MOCA	MMSE < 21 (norm for Chile)	2	Sex, age, APACHE II, SOFA, delirium days, duration MV, hospital LOS
Torgersen 2011 [83] January 2008 – February 2009	Prospective; consecutive; cohort (parallel surgical cohort; not requiring ICU admission) 28 SICU patients; 24 surgical patients	CANTAB MMSE DMS (delayed matching to sample) Stocking of Cambridge Paired associate learning test SF-36	MMSE < 24 Z-score below −2 SD on 2 or below 1.5 SDs on 3 out of 10 results reported by CANTAB	3	Sex, age, ICU LOS, duration MV, SAPS II, mac SOFA score, Charlson comorbidity index
Vitaz 2003 [84] October 1995 – March 1998	Prospective; consecutive; cohort 56 patients with moderate TBI	Telephone interview; questions regarding ADLs and mental functioning	Self-report	1	Age, median 24-hr GCS, hospital LOS, ICU LOS, duration MV
Wolters 2017 [85] January 2011 – June 2013	Prospective; consecutive; cohort 363 adult patients in mixed MSICU for > 48 hrs	Cognitive Failures Questionnaire (CFQ)	--	1	Sex, age, comorbidity index, APACHE IV, SOFA, ICU LOS, admission type
Woon 2012 [86] August 2007 – December 2008	Prospective; consecutive; cohort 53 patients MSICU	MMSE Mini-Cog WASI (full-scale, verbal, performance) Trail Making tests A and B Hayling Sentence Completion Test WASI-R (digit symbol) WMS-III (logical memory) California Verbal Learning Test-II Rey-Osterrieth Complex Figure Test Finger Tapping test Controlled Oral Word Association test Wide-Range Assessment test-3 (reading) Golden Stroop test (inference trial)	MMSE < 24 MiniCog considered impaired if recalled no words, or recalled 1 or 2 words with an abnormal clock drawing score NP testing: ≥2 or test scores > 1.5 SDs or 1 test score that was > 2 SDs below population norms	3	Sex, age, education, ICU LOS, hospital LOS, duration MV, max FiO2, Min PaO2, APACHE II, ICU admission diagnosis
Zhao 2017 [87] January 2013 to September 2013	RCT 332 patients; 165 patients were included in the control group and 167 in the cognitive intervention group	Montreal Cognitive Assessment (MoCA)	< 26 was considered CI	2	Sex, age, education, ICU LOS, ICU type, comorbidity index, medications (steroids, analgesia, sedation)

ARDS: acute respiratory distress syndrome; ALI: acute lung injury; COPD: chronic obstructive pulmonary disease; LT: long-term; MV: mechanical ventilation; LOS: length of stay; ICU: intensive care unit; MSICU: medical-surgical ICU; CCU: coronary care unit; SICU: surgical ICU; CICU: cardiac ICU; FU: follow-up; TBI: traumatic brain injury; GCS: Glasgow coma score; RCT: randomized controlled trial; SAT: spontaneous awakening trial; SBT: spontaneous breathing trial; ICH: intracerebral hemorrhage; SAH: subarachnoid hemorrhage

Reporting of key covariates

Interestingly, but not surprisingly, there is variable reporting of important covariates in studies of ICU survivors after hospital discharge. Again, using the example of long-term cognitive impairment, the most commonly reported covariates were age and sex in follow-up of ICU survivors; level of education was reported in only half of the studies (31/61; 51%). Severity of illness, duration of ICU or hospital stay, use of sedative agents, and incident delirium were also inconsistently reported. In order to inform a comprehensive understanding of outcomes after ICU, more uniform reporting of key covariates is necessary to synthesize the results of different cohorts. Such an understanding is essential for researchers and clinicians to advance research and enhance future care of ICU survivors.

Family and friends of ICU survivors

Physical, cognitive, emotional and social problems are common among ICU survivors. These wide-ranging issues are also seen frequently in family members and friends [26]. While many of these problems are new, unmasked, or exacerbated by acute illness, in-hospital events may also strip away compensatory strategies that had helped patients or their caregivers cope in the past. Social relationships have a two-way influence on health and well-being; social isolation in many disease processes is known to exacerbate conditions and predict mortality [27]. Conversely, social relationships are also known to have a protective impact on health, especially in disease processes such as cancer and depression [28, 29]. Therefore, understanding the emotional and social needs of not only our patients, but also their caregivers, will help improve our understanding of ICU survivorship. Given the key role of family members and friends in the lives of our patients, future research should include further understanding the relationship between survivors and their caregivers, including their respective social networks [30], to determine how acquired co-morbidities may be managed and mitigated.

Conclusions

Many methodological challenges face the researcher and clinician in understanding the long-term consequences of critical illness, such as high mortality rates in long-term follow-up. The current focus should be on ensuring adequate sample sizes, limiting attrition, standardizing outcome measures, and reporting key covariates to better understand the context of described outcomes. These challenges require diligent focus to enhance our collective ability to collect and integrate data, with the hope of creating a better understanding of the morbidity associated with ICU survival and life after hospital discharge.

Key Points.

1. Methodological consideration needs to be integrated in all stages of study design for long-term follow-up of ICU survivors after hospital discharge.

2. Heterogeneity in outcome definitions, measurement instruments and reported covariates used in studies of ICU survivors creates a major barrier in synthesizing the existing literature.

3. High financial burden of personnel as well as the reimbursement of subjects and family members for their time and inconvenience is an important limitation to many investigators and research groups.

4. Family members and friends are key contributors to understanding the landscape of ICU survivorship, and their knowledge and data should be included in the methodological design and outcomes assessments.