Acute Respiratory Failure Survivors’ Physical, Cognitive, and Mental Health Outcomes: Quantitative Measures versus Semistructured Interviews

Abstract

Rationale: Increasingly, patients are surviving acute respiratory failure (ARF), prompting the need to better understand standardized outcome measures commonly used during ARF follow-up studies.

Objectives: Investigate standardized outcome measures (patient-reported physical and mental health measures, and cognitive testing) compared with findings from semistructured, qualitative interviews.

Methods: As part of two ARF multicenter follow-up studies, standardized outcome measures were obtained, followed by qualitative evaluation via an in-depth, semistructured interview conducted and coded by two independent researchers. Qualitative interviews revealed the following post-ARF survivorship themes: physical impairment; anxiety, depression, and post-traumatic stress disorder symptoms; and cognitive impairment. Scores from standardized measures related to these themes were compared for ARF survivors reporting versus not reporting these themes in their qualitative interviews.

Results: Of 59 invited ARF survivors, 48 (81%) completed both standardized outcome measures and qualitative interviews. Participants’ median (interquartile range) age was 53 (43–64) years; 54% were female, and 88% were living independently before hospitalization. The two independent reviewers classifying the presence or absence of themes from the qualitative interviews had excellent agreement (κ = 0.80). There were significantly worse scores on standardized outcome measures for survivors reporting (vs. not reporting) physical and mental health impairments in their qualitative interviews. However, standardized cognitive test scores did not differ between patients reporting versus not reporting cognitive impairments in their qualitative interviews.

Conclusions: These findings support the use of recommended, commonly used standardized outcome measures for physical and mental health impairments in ARF survivorship research. However, caution is needed in interpreting self-reported cognitive function compared with standardized cognitive testing.

Mortality for critically ill patients with acute respiratory failure (ARF) has been decreasing over recent decades (1). With a growing number of ARF survivors, there is increasing awareness of physical, cognitive, and mental health challenges after hospital discharge (2–6). Research aimed at understanding and improving the postdischarge outcomes faced by ARF survivors is challenging because of the complexity of survivors’ experiences (3, 7). To date, much of the knowledge regarding ARF survivorship has been gathered from research using standardized patient outcome measures and clinical testing (8, 9). It is unclear how such standardized measures reflect the ARF survivorship experience demonstrated by survivors’ qualitative reports.

Commonly used outcome measures in post–intensive care unit (ICU) investigations include standardized patient-reported outcome measures (i.e., validated surveys) evaluating symptoms of physical function impairment, anxiety and depression, post-traumatic stress disorder (PTSD), and quality of life, along with standardized tests of cognitive function (e.g., memory impairment) (8). These measures quantitatively evaluate survivors’ outcomes. A limitation of such outcome measures is that patient responses are restricted to questions and topics that are specifically evaluated using the standardized outcome measure. Patients are unable to offer their own perspective on their survivorship experience. To elucidate such experiences, qualitative research methods can be used to identify meaningful patient outcomes (4, 10, 11).

In ICU survivorship research, there is very limited comparison of standardized patient outcome measures and patient experiences obtained from qualitative studies. Evaluation of how commonly used standardized patient outcome measures reflect the self-reported narrative of patient experiences can help researchers and clinicians gain novel insights regarding whether such measures sufficiently represent survivors’ related post-ICU morbidities (12). Hence, the objective of this study is to conduct a secondary analysis comparing results of standardized patient-reported outcome measures of physical and mental health and standardized cognitive tests, with narrative reports of ARF survivors’ experience obtained from semistructured qualitative interviews to evaluate how well these outcome measures reflect related patient experiences.

Methods

Participants

A convenience sample of 59 English-speaking patients, without prior evidence of cognitive impairment, recruited from 20 hospitals across the United States were invited to participate in semistructured qualitative interviews after completing patient outcome measures as part of either the ALTOS (Acute Respiratory Distress Syndrome [ARDS] Network Long-Term Outcomes Study) or the ROMA-ARF (Recovery of Muscle after ARF) study. Patients in ALTOS were enrolled from the ARDS Network’s SAILS (Statins for Acutely Injured Lung) trial (13). ROMA participants were part of a National Institutes of Health–funded, multisite observational study of ICU-acquired neuromuscular dysfunction conducted in five ARDS Network study sites. Both studies enrolled patients with ARF (with the SAILS trial exclusively enrolling patients with sepsis-related ARDS) requiring mechanical ventilation via an endotracheal or tracheostomy tube, with major eligibility criteria summarized in the Table E1 in the online supplement. Informed consent was obtained from participants for all aspects of this study. This study was approved by the institutional review board at Johns Hopkins University and all participating sites.

Outcome Measures

Patient-reported outcome measures and cognitive testing

Patients from ALTOS completed a battery of validated patient-reported outcome measures and cognitive tests at 6 and 12 months after ARF, occurring between June 2013 and October 2014, with results for the entire patient cohort published previously (14, 15). Patients from the ROMA study completed the evaluations at 6 months after ARF, occurring between October 2014 and April 2015. All patient outcome measures were completed by phone, with mail or in-person administration methods used if phone-based administration was not feasible. The following patient-reported outcome measures from ALTOS and ROMA are evaluated in this study: Short Form (SF)-36 version 2 Physical Component Summary (PCS) (mean, 50; standard deviation [SD], 10; higher score is better, with scores ≥2 SDs below the matched population mean indicating substantial impairment) (16); EuroQol 5 Dimensions three-level version (EQ-5D-3L) mobility question (range, 1–3; lower score is better, with score ≥2 indicating substantial impairment) (17); Hospital Anxiety and Depression Scale (HADS) subscale scores for anxiety and depression symptoms (for each subscale, range, 0–21; lower score is better, with scores ≥8 indicating substantial symptoms) (18); and Impact of Events Scale-Revised (IES-R) for post-traumatic stress disorder symptoms (range, 0–4; lower score is better, with scores ≥1.6 indicating substantial symptoms) (19, 20). The battery of cognitive tests consisted of two evaluations that are part of the Wechsler Memory Scale–Third Edition (21, 22): 1) immediate and delayed memory via the Logical Memory I and II, and 2) attention/working memory via the Digit Span test (age-adjusted scaled score range, 1 to 19; higher is better, with mean age-scaled score = 10 and a score of 8 equal to the 25th percentile, with scores ≥2 SDs below the matched population mean indicating substantial impairment).

Semistructured in-depth qualitative interviews

In-depth qualitative interviews were conducted, via telephone, by trained research staff (Y.P. and Linda Ugbah) who were blinded to the preceding patient-reported outcome measures and cognitive testing results. Qualitative interviews were semistructured and lasted approximately 30 minutes. On the basis of participant responses to semistructured interview questions, interviewers asked probing follow-up questions to fully understand participants’ experiences. Written transcripts, prepared by a professional transcriptionist, were used to develop a codebook containing patients’ survivorship themes via content analysis methods (23), with coding performed independently by two researchers (Y.P. and P.A.M.-T.) using NVivo 10.0 software (QSR International Pty Ltd, 2013). Discrepancies in coding were addressed by involving a third expert reviewer (M.N.E.). Interviews were conducted until no new themes or ideas emerged for three consecutive interviews, according to established methods for qualitative research (24). Details of this process and full results of this qualitative work alone have been previously reported (10).

Comparing patient outcome measures versus qualitative interviews

The qualitative interview codebook was reviewed by A.N., D.M.N., and M.N.E. to identify themes from the previously coded qualitative interviews that corresponded to patient-reported outcome measures for physical and mental health and the cognitive tests. The qualitative interview themes selected for comparison with patient outcome measures were as follows: 1) physical outcomes: presence of mobility-related impairments compared with scores for the SF-36 PCS and EQ-5D; 2) mental health outcomes: presence of anxiety and/or depression symptoms compared with scores for the HADS Anxiety and Depression subscales, respectively, and endorsement of PTSD symptoms compared with scores for the IES-R; and 3) cognitive outcomes: presence of memory impairments compared with age-adjusted scores for Logical Memory 1 and 2 and Digit Span. For the purpose of this analysis, results from the qualitative analyses of patient interviews were recoded by two reviewers independently and in duplicate (Y.P. and M.N.) via assigning a binary score for the presence or absence of each symptom or impairment related to each of the above themes (0 = absent symptoms or impairments, 1 = present symptoms or impairments). A binary score was only designated if individuals discussed symptoms pertaining to a specific theme in any capacity during the qualitative interview, resulting in a different number of responses per theme. Any discrepancies in binary scoring were resolved by a third expert reviewer (M.N.E.).

Statistical Analyses

For ALTOS, if the patient outcome measures were completed at both 6 and 12 months, the time point closest to the date of the qualitative interview was selected for this analysis. Agreement between the two coders scoring the qualitative interviews was assessed using the κ statistic for each domain, and, if not significantly different, an overall κ score was calculated. The Wilcoxon rank sum test was used to compare patient outcome measures with the presence versus absence of symptoms from the qualitative interviews. For the HADS and IES-R surveys, the proportion of patients meeting established thresholds for symptom burden was compared with the presence versus absence of symptoms from the qualitative interviews using Fisher exact test. Disagreement between the presence of symptoms reported in qualitative interviews and dichotomized scores (using the thresholds described above) for patient outcome measures were evaluated using descriptive statistics (16–22). SAS software version 9.4 (SAS Institute, Inc.) was used for all data analyses, with P ≤ 0.05 considered statistically significant.

Results

Between November 2013 and May 2015, 48 (81% of 59 invited) ARF survivors completed semistructured qualitative interviews, with the complete results and analysis of this qualitative work previously reported elsewhere (10). Of the 11 patients who did not participate in the semistructured interviews, eight could not be contacted, two were deceased, and one declined to participate. Of these 48 participants, the median (interquartile range [IQR]) age was 53 (43–64) years, 54% were women, 81% were white, and 88% were living at home independently before hospitalization (Table 1). The median (IQR) duration of mechanical ventilation was 7 (4–12) days. The interviews in this study were conducted at a median (IQR) 8 (6–12) months since ARF onset. The median (IQR) time from administration of the standardized patient outcome measures to the qualitative interview was 25 (9–50) days.

Table 1.

Baseline characteristics and intensive care data

	N = 48
Baseline patient data
Age, yr	53 (43–64)
Women, n (%)	26 (54)
White, n (%)	39 (81)
Living at home independently, n (%)	42 (88)
Baseline intensive care data
APACHE III score	98 (74–124)
Duration of mechanical ventilation, d	7 (4–12)
ICU length of stay, d	10 (7–17)
Hospital length of stay, d	17 (12–27)
Surveys
Days from ARF onset to outcomes assessment	235 (176–351)
Days between survey and semistructured interview, median (IQR)	25 (9–50)

Definition of abbreviations: APACHE II = Acute Physiology and Chronic Health Evaluation II; ARF = acute respiratory failure; ICU = intensive care unit; IQR = interquartile range.

Values are medians (interquartile range) unless stated otherwise. Proportions might not add to 100% because of rounding.

For this secondary analysis of qualitative interviews, the two reviewers had excellent agreement (25) in all five domains, with an overall κ (95% confidence interval) of 0.80 (0.72–0.89). Examples of quotes from the qualitative interviews demonstrating symptoms and impairment are reported in Table 2. Some participants described constant anxiety about being critically ill again, such as, “I worry all the time now, I don’t know why, but it’s just a thing.” Some reported physical limitations, “I just wonder if I’m going to be back to walking normal.” Others experience discouragement and depression, “I was severely depressed…horrible depression.” Participants also expressed symptoms of PTSD as having “flashbacks of being hooked up to the machines, of being in ICU and sometimes I wake up in a cold sweat because I think I’m in the ICU.” Some participants recognized difficulties with memory impairment, with one mentioning, “I can’t remember where I put something down at, sometimes I can’t remember my children’s birthdays.”

Table 2.

Examples of quotes from the semistructured interviews

Physical impairment–mobility

I get up to walk I wobble a lot, I don’t know, let’s see, but my walk isn't the same. I fall quite a bit, I've got to watch where I'm going.

…if you’re not mobile you can’t do anything

Anxiety

I guess I panic a lot. I cough a lot and I think it’s just an irritation in the lungs still, but nothing seems to be helping it. Any time I have pain in my lung I’m afraid it’s back again

Depression

I find that very depressing that the life I used to enjoy is no longer available.

Yeah, but I would say most of the time I’m just in a blue funk. You know, I just don’t feel like going anywhere and doing anything, basically because I’m tired of struggling to walk and I’m tired of, you know, having to lug around the walker or the cane and stuff, and plus, not only is it uncomfortable, but it hurts to walk on the soles of my feet, so you know, I just don’t do much.

Post-traumatic stress disorder

It was a horror. I recognize that it was absolutely necessary but it scares the hell out of me and I will cry when I think back.

…a lot of bad dreams at night and flashbacks and night sweats

Cognition–memory impairments

It's my short term I'm really worried about because you know, for me to make my way, and this is a really big house. And to make my way into any of the rooms I get there and I'm like, what was I getting…

It is not there anymore and takes me a while to try and remember and that makes it very difficult to interact with people at a level that can’t even possibly bring any enjoyment to myself or to them.

A total of 332 paired assessments across the eight themes from the qualitative interviews were eligible to compare interview results versus standardized patient outcome measures for each theme. Participants who reported (vs. did not report) having physical and mental health symptoms in their qualitative interviews had significantly worse physical and mental health scores from the standardized patient outcome measures (Table 3). For example, the median (IQR) SF-36 PCS scores were significantly worse (33 [26–38] vs. 52 [35–56]; P = 0.002) in those reporting versus not reporting physical symptoms during the qualitative interviews. Similarly, the scores were significantly worse in those reporting versus not reporting mental health symptoms for HADS-Anxiety (8 [4–15] vs. 4 [2–7]; P = 0.002), HADS-Depression (10 [5–12] vs. 2 [1–9]; P = 0.010) and IES-R (PTSD) (1.6 [0.2–2.4] vs. 0.4 [0.0–0.7]: P = 0.017). Furthermore, the proportion of patients above the threshold for substantial symptoms on HADS-Anxiety and IES-R (PTSD) were significantly higher in those who reported (vs. did not report) symptoms during the qualitative interviews: 57% versus 13% (P = 0.002) and 47% versus12% (P = 0.014), respectively.

Table 3.

Patient outcome measures in those reporting versus not reporting symptoms in semistructured interviews

	Interviews	Patient Outcome Measure Scores*			P Value‡
	Qualitative Symptoms Present,†n/N (%)	Overall	Qualitative Symptoms Present	Qualitative Symptoms Absent
Physical outcomes
SF-36 Physical Component Summary	29/40 (73)	35 (27–48)	33 (26–38)	52 (35–56)	0.002
EQ-5D Mobility	30/41 (73)	2 (1–2)	2 (1–2)	1 (1–2)	0.01
Mental health outcomes
HADS Anxiety score	23/47 (49)	5 (2–10)	8 (4–15)	4 (2–7)	0.002
Substantial symptoms (score ≥ 8) n (%)		16 (34)	13 (57)	3 (13)	0.002
HADS Depression score	23/34 (68)	9 (2–11)	10 (5–12)	2 (1–9)	0.01
Substantial symptoms (score ≥ 8) n (%)		19 (56)	15 (65)	4 (36)	0.15
IES-R total score	17/43 (40)	0.5 (0.1–1.7)	1.6 (0.2–2.4)	0.4 (0.0–0.7)	0.02
Substantial symptoms (score ≥ 1.6) n (%)		11 (26)	8 (47)	3 (12)	0.01
Cognitive outcomes
Immediate memory: Logical Memory I age-adjusted score	29/43 (67)	8 (5–12)	8 (5–12)	8 (7–11)	0.69
Delayed memory: Logical Memory II, age-adjusted score	27/41 (66)	8 (6–10)	8 (4–10)	8 (6–11)	0.47
Attention/working memory: Digit Span, age-adjusted score	29/43 (67)	9 (7–11)	9 (7–10)	10 (6–12)	0.59

Definition of abbreviations: EQ-5D = EuroQol 5 Dimensions; HADS = Hospital Anxiety and Depression Scale; IES-R = Impact of Event Scale - Revised; SF-36 = Short-Form 36 version 2.

Patient outcome measure scoring: SF-36 Physical component summary: normalized scores (mean, 50; standard deviation, 10; higher score is better), EQ-5D-3L: range, 1–3; 1 = no problems in walking about, to 3 = confined to bed. HADS Anxiety and HADS Depression: range, 0–21; lower score is better. IES-R: range, 0–4; lower score is better. Immediate Memory, delayed memory, and working memory: range, 1–19; higher is better.

^*Scores reported as median (interquartile range) unless otherwise noted.

^†N is total number of individuals referencing stated topic during interview and a completed patient outcome measure.

^‡P value from Wilcoxon rank sum test for the continuous scores, and Fisher exact test for the comparison of proportion.

Cognitive test scores for immediate and long-term memory, as well as attention/working memory, did not differ for individuals who reported (vs. did not report) memory impairment during qualitative interviews (Table 3). For example, the median (IQR) immediate memory scores (measured by Logical Memory I) in individuals who reported (vs. did not report) memory impairments were similar: 8 (5–12) and 8 (7–11) (P = 0.688), representing memory impairment compared with population normal values (26).

Qualitative interview results were also compared with the dichotomous standardized patient outcome measures for each theme. Among the 332 paired assessments, there was disagreement in 136 (41%) assessments, with 123 (90%) of these disagreements reflecting presence of symptoms on qualitative interviews without survivors’ responses exceeding the threshold for significant symptom burden on the standardized patient outcome measures. This finding was present across the mental, physical, and cognitive health themes (Table E2).

Discussion

This mixed-methods study of 48 survivors of ARF analyzes standardized patient outcome measures compared with the ARF survivorship experiences from in-depth, semistructured qualitative interviews. Patient outcome measures for physical and mental health, but not for cognitive function, generally demonstrated agreement with qualitative analysis of patient experiences, indicating that patient experiences are being reflected by these standardized patient outcome measures.

In our study, patient-reported symptoms of depression, anxiety, PTSD, and impaired mobility were associated with corresponding patient-reported outcome measures, showing consistency between the two different methods of assessing the ARF survivorship experience. The presence of physical and mental impairment described by patients as part of both the patient-reported outcome measures and the qualitative interviews were consistent with prior studies (4–6, 27, 28). Patient-reported outcomes (measured by survey instruments) and experiences (measured by qualitative interviews) both rely on the patient’s perspective and experience of their physical and mental health.

In contrast, cognition, measured with performance-based cognitive tests of memory and attention, was not associated with cognitive impairments reported by patients. Notably, on the basis of cognitive tests, many patients had memory impairment compared with matched population norms, but these test scores were not associated with ARF survivors’ reported experiences of cognitive impairment in the qualitative interviews. Although our findings of cognitive impairment are consistent with prior literature showing a high prevalence in ICU survivors (2, 29), patients may not report cognitive impairments in qualitative interviews because of a lack of awareness of these deficits, as reported in prior literature (30–32). It has also been previously demonstrated that cognitively impaired individuals have awareness of deficits in some cognitive domains but not all, particularly the domains of memory and executive function (33). Furthermore, there is no clear threshold to define what degree of impairment an individual experiences before deeming the impairment severe enough to report. In traumatic brain injury, self-reported cognitive impairment was associated with premorbid factors and not with actual cognitive impairments assessed using neuropsychological tests (34). These findings may explain the absence of reports of cognitive impairment in prior qualitative research of ICU survivorship (4). To optimize insight into cognitive deficits, future studies may need to include measures of patient insight of cognitive impairments and compare those findings to family caregivers.

A recommended core outcome set and associated measurement instruments was recently established to improve consistency and comparability in studies evaluating outcomes after hospital discharge for ARF survivors (35). These core outcome measurement instruments include the HADS, IES-R, EQ-5D, and SF-36 PCS (with SF-36 being optional), which were all used in this study (16–20). The current analyses provide further data demonstrating that these core outcome measures capture data that are important on the basis of patient-reported survivorship experience.

The importance of qualitative research in advancing clinical research involving critically ill patients has been identified in international roundtable discussions with wide support, including from the Society of Critical Care Medicine and European Society of Intensive Care Medicine (36). The best methods for understanding the morbidities faced by ICU survivors is evolving, but it should include both quantitative and qualitative outcome assessments (11), as patient outcome measures and qualitative research are complementary methods that can inform the design and testing of interventions to improve the survivorship experience and inform patients, families, and clinicians. This study is one of the first to prospectively evaluate posthospital physical, mental health, and cognitive outcomes in ARF survivors using patient-reported outcome measures, cognitive assessments, and independent in-depth, semistructured qualitative interviews. Although consistency was generally demonstrated between standardized patient outcomes and qualitative interviews for physical and mental health status, there were survivors reporting symptoms on qualitative interviews whose responses on standardized outcome measures did not exceed the threshold for significant symptoms. This finding may be due to patients’ symptom severity, on qualitative interviews, not being great enough to exceed the traditional threshold used for dichotomization of the standardized outcome measures.

A strength of this study is the inclusion of a cohort of ARF survivors recruited from multiple centers across the country. In addition, the open-ended qualitative interviews were performed and coded, independently and in duplicate, by trained researchers who were blinded to the patient outcome assessments. Assessment of the standardized patient outcome measures and the qualitative interviews was separated by a median of 25 days, which helps limit potential bias in interview responses based on patient recall of the standardized patient outcome measures. However, there are potential limitations of this study. First, although the sample size is comparable to other qualitative studies, it may be considered small in the context of quantitative studies and with low power. Comparability of patient outcome measure data in this study and prior quantitative studies assuages such concerns regarding lack of generalizability of the sample. Second, qualitative research methods have intrinsic limitations, such as potential for researcher bias and inability to consistently quantify patient-reported experiences. However, we used independent double coding of patient qualitative interviews and codebook development and used interviewers and coders who were blinded to the standardized patient outcome measures. Third, although we demonstrated that these qualitative interviews and standardized patient outcome measures are generally congruent for physical and mental health symptoms, it is unclear if the results would change if the qualitative interviews were compared with in-person performance-based testing of physical outcomes (e.g., 6-minute-walk test) or in-depth diagnostic psychiatric evaluations of mental health that could not be included in this study because of feasibility issues. Last, the results may not be generalizable to other ICU populations, because only ARF survivors from the United States were included in this study.

This prospective, mixed-methods study of survivors of ARF demonstrates a substantial burden of physical, mental health, and cognitive impairments after recovery from critical illness. For physical and mental health impairments, the study results were consistent when comparing commonly used patient-reported outcome measurement instruments and in-depth, semistructured qualitative interviews. This agreement supports the use of these recommended core outcome measures in ARF survivorship studies. However, standardized cognitive test scores were not associated with patient reports of cognitive impairment in the qualitative interviews, cautioning against sole use of patient self-report for evaluating cognitive impairment.