Participant retention in follow-up studies of intensive care unit survivors – A Scoping Review

Abstract

Objective:

To synthesize participant retention data and related reporting in studies evaluating post-hospital outcomes of survivors of critical illness after an intensive care unit (ICU) stay.

Study Design and Setting:

A synthesis of literature following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Extension for Scoping Reviews (PRISMA-ScR) checklist.

Results:

We included 243 publications, from 225 unique studies of 87,602 participants. Participant retention could not be calculated for any time-points in 13% of studies nor in 22% of all follow-up time-points. Retention ranged from 18-100%. When compared to follow-up before 1-month, retention at each later timepoint was not significantly different. Age and sex were not associated with retention and more recent studies had decreased retention (odds ratio: 0.94 (95% confidence interval: 0.92-0.96; p<0.001)). Reporting of retention-related study methodology was inconsistent.

Conclusion:

Retention rate could not be calculated for 22% of study follow-up timepoints, with retention at the remaining time-points generally being high (≥85%), but with high variability (18% – 100%). ICU survivorship research could be improved via: (1) more detailed guidance on reporting participant retention, and (2) use of existing resources and best practices to facilitate better study design and to improve participant retention to preserve statistical power and reduce selection bias.

INTRODUCTION

Advances in the care of the critically ill have resulted in lower hospital mortality.¹ This has, in turn, created greater interest in survivorship.^2-7 Such studies have demonstrated that significant impairments in physical function, cognition and/or mental health after hospital discharge are present in most critical illness survivors.^8,9 Such impairments may persistent for months or years.¹⁰

Longitudinal follow-up studies provide important information about the relationships between baseline status, critical illness, interventions during hospitalization, and long-term outcomes. In such follow-up studies, participant retention is critical for the integrity of study results. Among interventional studies that did not demonstrate a treatment effect on survivors’ long-term outcomes, low participant retention may be an important issue impacting the results.¹¹ To our knowledge, there is no published systematic analysis of participant retention data across survivorship studies from general adult intensive care units (ICUs) (i.e., non-specialty ICUs, such as general medical and surgical ICUs). Therefore, our objective was to systematically review participant retention data, and its reporting, in studies of outcomes after hospital discharge in survivors of critical illness following admission to a general ICU.

METHODS

As our purpose was not to inform clinical practice, but to identify the reporting of participant retention in a broad population, a scoping review, rather than a systematic review, was selected for our methods.¹² We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Extension for Scoping Reviews (PRISMA-ScR) checklist in reporting this manuscript.¹³ The protocol was registered with PROSPERO (CRD42018087835).

Search strategy

Five electronic databases (PubMed, EMBASE, PsycINFO, Cumulative Index of Nursing and Allied Health Literature, the Cochrane Controlled Trials Registry) were searched for articles published after 1970 until March 2022 using a combination of terms to find articles focused on intensive care populations combined with outcome assessment, health status, functional status, quality of life, and follow-up (Table S1). Additionally, we hand searched reference lists and personal files of relevant narrative and systematic review articles.

Article selection

We included peer-reviewed published studies with a sample size of ≥20 adult ICU survivors assessed after hospital discharge. Researchers performing screening were not blinded to author or journal details. Articles were excluded for any of the following: 1) the primary purpose was to evaluate or describe the psychometric properties of a measure, 2) only qualitative methods (e.g., semi structured interview or open-ended questions) were used to assess outcomes, 3) if survival was the only outcome reported, 4) patients were not from a general ICU (i.e. focused on a specific ICU population such as trauma or transplant), 5) there was only a single follow-up timepoint with consent occurring at the same time (i.e., no prospective follow-up performed to evaluate participant retention), 6) if any of the following were true for a majority (>50%) of the study population: i) <16 years old, ii) had neurologic injury, iii) had undergone cardiac surgery, or iv) had not been admitted to an ICU.

Data abstraction

Data abstraction was performed independently, in duplicate, by pairs of trained researchers (AAA ML, VR, AF, SC, SV, MK, SPSC, ND, PK, RN, VS, and VT). Conflicts between reviewers were resolved, by consensus, and if unresolved, then in consultation with a senior researcher (DMN or VDD). Data abstractors were not blinded to author or journal details. Authors were contacted for additional data when necessary. The following data were collected: participant retention rates at each follow-up time-point; utilization of a patient flow diagram; modes of data collection (e.g., in-person, phone, mail); reporting of mortality during follow-up; blinding of assessors (if an interventional study); accounting for loss to follow-up in sample size/power calculation; study exclusion criteria related to barriers to follow-up (e.g., homelessness), and description of participant retention strategies.

Risk of Bias

The Cochrane Risk of Bias methodology was used to assess risk of bias for randomized controlled trials (RCTs).¹⁴ The Newcastle-Ottawa Scale was used for observational studies.¹⁵ The following criteria from the Newcastle-Ottawa Scale were not applicable to this scoping review as the focus was on participant retention and not a specific clinical outcome and thus were not used: demonstration that the outcome was not present at enrollment; assessment of the outcome; follow-up that was long enough for the outcome to occur.

Statistical analysis

The pooled average participant retention rate was calculated as part of the data synthesis. Follow-up times reported in eligible studies were <1 month, 1, 2, 3, 6, 12, 24, 36 and 60 months. To reduce the number of time points to only these most commonly reported, the few studies with other time points were pooled as follows. Five studies <1 month were pooled together, and the two 1.5-month studies were pooled with the 1-month studies, the one 2.5-month study was pooled with the 2-month studies, the two 4-month studies were pooled with the 3-month studies, and the (one) 6.5, (one) 8 and (two) 9-month studies were pooled with the 6-month studies. A 30-month study could not be used in the meta-analysis due to only one study for that time period. A 9-month study visit was not used in the meta-analysis due to there being both 6- and 12-month visits for the same study which were used instead. For studies with age reported as median and interquartile range, the mean and standard deviation were estimated using existing methods.¹⁶ When retention rates and mean age were provided for treatment groups or patient groups within a study at the same follow-up time-point, they were tested for statistical differences using Fisher’s Exact Test. Retention rates with no statistically significant difference (p ≥0.05) were combined. Mean ages for groups were combined when corresponding retention rates were combined, but statistical differences noted. For one study-visit, the “percent male” statistics needed to be modified slightly for treatment groups with unequal cohort retention rates to prevent non-convergence of the regression model (see below). For studies where retention rates were 100%, the Haldane-Anscombe correction was used to correct the confidence intervals.^17,18

Two approaches were used to calculate retention rates. The primary approach calculated retention rate as the number of patients who had a study assessment (numerator) divided by the total number of patients who were eligible for follow-up at that same time-point (denominator), excluding participants who died by that time-point. The alternative approach excluded from the denominator, participants who died and those who permanently discontinued participation or were withdrawn from the study. Retention rates could not be calculated if all required data were not reported or if mortality was combined with lost-to-follow-up data.

A linear random intercept regression model for the log odds of the retention rate (logit transformation) was used to estimate the pooled retention rate. The estimated average or pooled retention rate was computed by applying the inverse-logit transformation. The model was extended to determine if the study pooled retention rate varied as a function of mean patient age, percent male patients and publication year of the study. For these analyses, three separate regression models were constructed, with the patient or study characteristic of interest statistically centered in each model. Statistical heterogeneity among studies was evaluated using the I² statistic (with >50% deemed to be substantial heterogeneity).¹⁹ The I² statistic was calculated for each follow-up when there were more than 2 studies reporting data.²⁰ SAS^® version 9.4 (2016, Cary, NC) was used to conduct all analyses.

RESULTS

In this scoping review, 243 publications, reporting on 225 unique studies (Figure 1), met eligibility criteria. Among these 225 studies, 48 (21%) were RCTs, 176 (78%) were cohort studies, and 1 (<1%) study was a cohort study embedded within an RCT. There were a total of 87,602 patients in the eligible studies. A total of 38 (17%) of the 225 studies were conducted in USA (largest number from a single country) with representation from 35 different countries (Supplemental Materials). The most frequent time-points for follow-up were 3 months (evaluated in 90 (40%) studies), 6 months (99 (44%) studies), and 12 months (67 (30%) studies), with the earliest and latest timepoints being 1 week (2 studies) and 60 months (6 studies).

Figure 1: Flow chart for identifying eligible studies.

Abbreviations: CINAHL, Cumulative Index of Nursing and Allied Health Literature; CENTRAL, Cochrane Controlled Trials Registry, General ICU = non-specialty (i.e., non-specialty ICUs, such as general medical and surgical ICUs) intensive care unit

Risk of Bias assessment

Among the 49 RCTs, a low risk of bias was found for 46 (94%) studies in randomization and 45 (92%) in selective reporting, 42 (86%) studies for allocation concealment, and 29 (59%) studies for blinded outcome assessments. There were only 10 (20%) RCTs with low risk of bias for blinding participants and personnel. Risk for bias in addressing incomplete outcome data among the RCTs was low for 32 (65%) studies (Table 1).

Table 1.

Risk of bias assessment for randomized controlled trials

Study	Random sequence generation (selection bias)	Allocation concealment (selection bias)	Blinding of participants and personnel (performance bias)	Blinding of outcome assessment (detection bias)	Incomplete outcome data addressed (attrition bias)	Selective reporting (reporting bias)
Abdelhamid, 2020	+	+	−	+	?	+
Allingstrup, 2017	+	+	−	+	+	+
Amundadottir, 2019	+	−	+	+	?	+
Azevedo, 2019	+	+	?	?	+	+
Azevedo, 2021	+	+	−	+	+	+
Battle, 2019	+	+	−	−	+	+
Brummel, 2014	+	+	−	+	+	+
Castillo, 2020	+	−	+	+	−	−
Cox, 2018	+	+	−	+	+	+
Cox, 2019	−	?	−	−	+	+
Cuthbertson, 2009	+	+	−	+	+	+
Denehy, 2012	+	+	−	+	+	+
Doig, 2013	+	+	−	−	−	+
Douglas, 2007	+	−	−	−	−	+
Dubois, 2017	+	+	+	+	+	+
Eggmann, 2020	+	+	−	−	−	+
Elliott, 2011	+	+	−	+	?	+
Fossat, 2018	+	+	−	−	+	+
Griffiths, 1997	+	+	+	+	+	+
Guidet, 2017	+	−	−	−	−	+
Harvey, 2016	+	+	−	−	+	+
Hatch, 2017	+	+	+	?	+	+
Irving, 2019	+	+	+	+	+	+
Jensen, 2016	+	+	−	+	+	+
Jones, 2003	+	+	+	+	?	+
Jones, 2010	+	+	−	+	+	+
Kalfon, 2019	+	+	−	−	−	+
Knowles, 2009	+	+	−	−	+	−
Kredentser, 2018	+	+	−	−	−	+
McDowell, 2016	+	+	+	+	+	+
Messika, 2019	+	+	−	+	+	+
Navarra, 2021	?	+	−	−	+	−
Nedergaard, 2021	+	+	+	+	+	+
Nickels, 2020	+	+	−	+	+	+
Olsen, 2019	+	+	−	+	+	+
Rood, 2019	+	+	+	+	−	+
Sayde, 2020	+	+	−	−	?	+
Shelly, 2017	+	+	−	?	−	+
Sosnowski, 2018	+	+	−	+	+	+
Strøm, 2011	+	+	−	+	+	+
Treggiari, 2009	+	+	−	+	+	+
Wade, 2019	+	+	−	+	+	+
Waldauf, 2021	+	−	−	+	+	+
Walsh TS, 2015	+	+	−	+	+	+
Wang, 2021	?	+	?	?	+	?
Wischmeyer, 2017	+	+	−	−	?	+
Wollersheim, 2019	+	+	−	−	−	+
Wu, 2019	+	+	−	+	+	+
Valso, 2019	+	?	−	−	−	+

Based on Cochrane criteria for assessing risk of bias in randomized controlled trials; “+” = low risk of bias, "−" = high risk of bias, and "?” = unclear risk of bias.

In observational cohort studies (n=177), 117 (66%) had low risk for bias in representativeness of the exposed cohort, and 111 (63%) had adequate follow-up (Supplemental Materials).

Retention-related reporting

Only 33% of studies reported using strategies to improve participant retention. Sample size calculation was not mentioned in 146 (65%) of 225 studies, of which 137 (94%) were cohort studies. Among those with a sample size calculation, 51 (23%) of 225 used a sample size calculation that accounted for potential loss to follow-up (Table 2). Flow diagrams were provided in 73% of studies, of which 79% reported retention at each time point and 43% described reason for loss to follow-up. Mortality during follow-up was reported in 77% of studies.

Table 2:

Participant Retention-Related Data in Longitudinal Studies of General ICU Survivors

Participant retention-related issue	Number (%) of studies reporting (total N=225)
Mortality reported during follow-up	173 (77%)
Sample size or power calculation accounting for loss to follow-up a	51 (23%)
Study exclusion criteria included barrier(s) to follow-up b	141 (63%)
Reported use of strategies to improve retention	75 (33%)
Included flow diagram with retention rates for each follow-up time-point	131 (58%)
Reported reasons participants were lost to follow-up	72 (32%)

General ICU = non-specialty (i.e., non-specialty ICUs, such as general medical and surgical ICUs) intensive care unit

^aDuring data abstraction this was a 2-step question: 1) did the study report a sample size calculation? and then if reported 2) did they account for loss to follow-up?

^bE.g., homelessness; also includes 3 studies that had additional exclusion criteria in follow-up stage, e.g., enrolled in hospital, then excluded homeless participants prior to follow-up.

Participant retention

Retention rates could not be calculated for at least one time point in 47 (21%) of the 225 studies, representing 117 (22%) of the 534 time-points. Retention rates could not be calculated for any time-points in 30 (13%) of the 225 studies, representing 76 (14%) of the 534 time-points. Among RCTs, retention rate could be calculated for at least 1 timepoint in 92% of studies; while among cohort studies, this was 85%. Across all time-points in the 225 eligible studies, retention rates ranged from 18-100%. For all 49 RCTs retention rates were reported and ranged from 38% to 100% across both control and interventions groups for available time-points.

Pooled results

The pooled retention rate (95% confidence interval (CI); I² statistic for time points with >2 studies) ranged from 67% (2 – 100%; 61%) at 36 months to 92% (85 – 96%, 95%) at 1 month (Figure 2). For all timepoints, the I² statistic ranged from 61% to 99%. There was no difference when comparing later timepoints to follow-up conducted earlier than 1 month (all p-values >0.1)

Figure 2: Average participant retention rates* in general ICU follow-up studies.

Diamonds in the graph are the pooled average retention rates, while the bars represent 95% confidence interval

* The pooled log odds and 95% confidence intervals were calculated using the logit (log odds) of the retention rates and transformed back to the retention rate scale. Since the retention rate is bounded between 0 and 100%, we did the meta-analysis on the logit (log odds) of the retention rate (the log odds is not bounded). The pooled log odds and 95% CIs were calculated and then transformed back to the retention rate scale; hence, the CIs will not be symmetric around the retention rate (but are symmetric around the logit retention rate). If a standard linear random effects model was used, the bounds of confidence intervals may have been beyond 0% or 100%.

General ICU = non-specialty (i.e., non-specialty ICUs, such as general medical and surgical ICUs) intensive care unit; CI = confidence interval; I²= proportion of the total variation attributable to differences across studies for time points with >2 studies

Age was not associated with retention rates (Odds Ratio [OR] for every 1 year (95% CI) 1.01 (0.99-1.03; p=0.372)). Similarly, the proportion of male patients in the study was not associated with retention rate (0.99 (0.97-1.00; p=0.067)). For publication year, more recent publication year was associated with reduced odds of participant retention ((OR for 1 year (95% CI) 0.94 (0.92-0.96; p <0.001)).

These results did not qualitatively differ when using the alternate definition for calculating retention rates, as described previously (see Supplemental Material for the pooled average retention rates using the alternate definition).

DISCUSSION

In this paper, we synthesized participant retention data for 225 studies evaluating 87,602 general ICU survivors after hospital discharge. Retention rates could not be calculated for at least one time-point in over one-fifth of studies and could not be calculated for any time-points in 13% of these studies. When reported, retention rate varied widely. Notably, less than one-third of studies reported using any strategies to improve participant retention and only one in 5 accounted for potential loss to follow-up in the study’s sample size calculation.

We could not calculate retention rates (incomplete data or mortality combined with loss to follow-up) for a large proportion of cohort studies. Importantly, this reduces ability to assess generalizability of the study results or to know if statistical power has been reduced. The STROBE checklist for cohort studies includes a recommendation for reporting retention rates and our findings reinforce this recommendation.²¹ The CONSORT checklist for RCTs recommends inclusion of loss to follow up information.²² In randomized controlled trials, if lost-to-follow-up differs between groups, then results are at increased risk of bias. In this scoping review, we were able to calculate retention rates for at least 1 timepoint in 92% of RCTs.

The association of study and participant characteristics with participant retention were notable. Both patient age and proportion of males in the study were not independently associated with retention. In contrast, other studies related to healthcare have reported increased adherence among older adults^23,24, a finding that may be explained by increased conscientiousness.²⁵ The finding of more recent publications having lower retention may be surprising. While recent STROBE and CONSORT guidelines may have increased reporting of participant retention,^21,22,26,27 factors not explored in our study must explain the negative trend in retention rate.

In this review, the latest timepoint reported was 5 years. Except for 36-month follow-up with 67% retention (only 2 studies), all other timepoints had pooled retention rates ranging from 85% to 92%. When compared to follow-up within the first month, retention in later timepoints were not significantly different. The stability of retention over time is reassuring. However, earlier, and later follow-up timepoints were less common, with 3, 6, and 12 months being the most common where 90 (40%), 99 (44%) and 67 (30%) studies reported on these timepoints, respectively.

There was heterogeneity in reporting participant retention-related data. For example, only 23% of studies considered lost-to-follow-up in sample size calculation and 65% of studies did not explicitly report sample size calculation. A flow diagram was used in 73% of studies of which 43% reported reasons participants were lost to follow-up. Only 33% of studies reported use of strategies to improve retention. Disclosing the extent of retention efforts is important as they may explain the variability of actual retention rates across studies. A recent systematic review reported that use of more retention strategies was associated with improved participant retention.²⁸ This same review review²⁸ found a total of 618 participant retention strategies, across 12 different themes, which are freely available as an online searchable database (https://www.improvelto.com/sysrevstrategies/). Moreover, as part of a U.S. NIH-funded research infrastructure project (grant #R24HL111895), additional, practical participant retention tools are freely available (https://www.improvelto.com/cohort-retention-tools/).

Strengths and Limitations

To our knowledge, there are no prior publications synthesizing the literature on participant retention rates and related methodology in general ICU survivorship studies. Studies included those published as of March 2022. Despite the important contribution of this analysis, there are potential limitations. To reduce heterogeneity between studies and optimize feasibility of this synthesis, we exclusively focused on studies of general ICU survivors. However, this population is still quite diverse and may result in limitations to precision and generalizability of our findings. Readers interested in retention data for specific ICU populations are referred to existing publications for acute respiratory failure survivors and for trauma survivors.^29,30 Despite this restriction to general ICU studies, there remained substantial heterogeneity in pooled participant retention rates. Additionally, the variety in time points for follow-up led us to combine data from those which were uncommon. Some of the heterogeneity could have been accounted for with greater demographic information for study participants; however, only age and sex were commonly reported across all studies. These limitations require the reader use caution when interpreting our results.

Conclusion

In this evaluation of 225 studies with 87,602 general ICU survivors, retention rates were highly variable across individual studies (range: 18 – 100%) and could not be calculated for 13% of studies. There were substantial differences in reporting of study methodology related to participant retention We recommend greater adherence to existing reporting standards for RCTs and cohort studies (CONSORT and STROBE), and more detailed reporting of retention related information (e.g., sample size calculation, use of participant flow diagram). Use of existing U.S. NIH-funded participant retention resources (www.improveLTO.com) and best practices for optimizing participant retention may facilitate better research design, improve statistical power, and reduce selection bias in critical illness survivorship studies.