Potential for Response Bias in Family Surveys About End-of-Life Care in the ICU

Abstract

Background:

After-death surveys are an important source of information about the quality of end-of-life care, but response rates generally are low. Our goal was to understand the potential for nonresponse bias in survey studies of family members after a patient's death in the hospital ICU by identifying differences in patient demographics and delivery of palliative care between patients whose families respond to a survey about end-of-life care and those whose families do not.

Methods:

We performed a cohort study of patients who died in the ICU at 14 hospitals. Surveys were mailed to family members 1 to 2 months after the patient's death. Chart abstraction was completed on all patients, assessing demographic characteristics and previously validated indicators of palliative care.

Results:

Of the 2,016 surveys sent to families, 760 were returned, for a response rate of 38%. Patients whose family members returned the surveys were more likely to be white (88% vs 74%, respectively; p < 0.001); to be older (71 years vs 69 years, respectively; p = 0.015); and to have received more indicators of palliative care, including medical record documentation of family present at death, involvement of spiritual care, and dying after a decision to limit life-sustaining therapies (p < 0.05).

Conclusions:

Patients whose family members responded to a survey about end-of-life care were more likely to be white, older, and have indicators of palliative care documented in the medical record. Because these patients likely received higher quality palliative care, these findings suggest that the response bias in end-of-life care research is toward overestimating the quality of palliative care.

Trial registration:

ClinicalTrials.gov Identifier: NCT00685893

Death is a common occurrence in the hospital ICU. It is estimated that 20% of the deaths in the United States occur during or shortly after a stay in the ICU.¹ The majority of these deaths occur after a decision to withhold or withdraw life support.^2,3 There is an increasing emphasis on improving end-of-life care in the ICU,⁴ and many challenges exist in assessing the dying experience of a patient in the ICU. Because < 5% of patients are able to communicate at the time decisions are made to withhold or withdraw life-sustaining therapies,³ patient-assessed outcomes usually are not feasible. Instead, outcomes that are more feasible for this type of research include surrogate assessments of the patient experience, family perceptions of their own experiences, and chart-based assessments of care.^5–9

Much of the research on the quality of end-of-life care in the hospital ICU is based on surveys completed by family members. These after-death surveys are an important source of outcome measures for studies of end-of-life care.^4,10 However, survey response rates for after-death surveys of family members often are poor, with rates of ≤ 65%.^11–13 These response rates introduce the potential for nonresponse bias. If response rates are low, and nonresponse is systematic and correlated with the variables of interest, the sample obtained may not be representative. In this situation, external validity of the study may be threatened, and generalizability of conclusions may be limited.

This study examines differences between patients whose family members do or do not respond to a survey about end-of-life care in the hospital ICU. To assess whether research that uses family surveys of end-of-life care is generalizable, it is important to identify differences among patients whose families return surveys about end-of-life care and those whose families do not. This research question is challenging because most studies are able to obtain few or no data about nonresponders. Our study offers an opportunity to compare information obtained from the medical and death certificate records of patients whose family members completed a survey and those whose families did not. This assessment of nonresponse bias provides information important to the interpretation of survey-based research on end-of-life care in the hospital ICU.

Materials and Methods

Design

Data were collected as part of an ongoing cluster randomized trial^14,15 in the Seattle-Tacoma, WA, area that was designed to evaluate the efficacy of a multifaceted, interdisciplinary intervention to improve palliative care in the hospital ICU. For the purpose of this study, only data collected during the baseline assessments (prior to implementation of the intervention) at participating hospitals were included.

Study Participants

All patients who died in the hospital ICU after a minimum stay of 6 h and within 30 h of transfer from the ICU during the preintervention period of the randomized trial were eligible for the study. The study population was drawn from 14 hospitals in the Seattle-Tacoma area, of which two were university-affiliated teaching hospitals, three were community-based teaching hospitals, and nine were community-based nonteaching hospitals. Patients who died were identified through hospital discharge and transfer logs. All study procedures were approved by the institutional review board of all 14 hospital sites.

Data Collection

Surveys:

Surveys were mailed to families of patients who died during the study period between September 2003 and April 2007. Families were identified in two ways. At one site, the patient's next of kin was identified from the electronic medical record; at all other sites, surveys were mailed to the patient homes and addressed, “To the family of [patient's name].” Surveys were written in English and mailed 1 to 2 months after the patient died. The survey packet included a cover letter explaining the study, a consent form, a $10 incentive, a postage-paid return envelope, and the questionnaire booklet. The questionnaire booklet included demographic questions, the Quality of Dying and Death questionnaire,¹⁶ and the Family Satisfaction with the ICU survey.^12,17 The questionnaire booklet generally took about 20 min to complete. Reminder or thank-you postcards were sent 2 weeks after the initial mailing. Second survey packets were sent after 4 weeks if there was no response to the initial mailing. Families were instructed to return a blank questionnaire or to contact the study office by phone to withdraw from the study if they did not want to receive further mailings.

Chart Abstraction:

Chart abstraction was completed on all eligible patients, regardless of whether a family member returned a survey. Data abstractors were trained by two research abstraction trainers. Training included a minimum of 80 h of practice abstraction with instruction on the protocol, guided practice charts, and independent chart review followed by reconciliation with the research abstraction trainers. Training continued until the abstractors reached 95% agreement with the trainers. For ongoing quality control, a coreview of a 5% random sample of patients' charts was done, ensuring agreement of ≥ 95% on all of the 440 abstracted data elements.

Death Certificate Data:

Washington State releases confidential electronic death certificate data linked by a patient identifier for the purposes of research. We used these records to provide data that were unavailable or incomplete in the medical record.

Variables of Interest

Patients' family members were categorized as responders (responding to the survey) or nonresponders (not responding the survey or withdrawing from the study). We examined chart abstraction variables, including patient demographics, and indicators of palliative and end-of-life care. Demographic variables for the patients obtained from the medical record included patient gender, age, type of insurance, admission source, and hospital ICU length of stay. From the death certificate data, we obtained patient race, educational level, marital status, and trauma as cause of death. Additionally, we determined an estimated median income based on patients' home zip codes.

Indicators of palliative care were based on prior consensus documents identifying potential domains and quality indicators for palliative care and include chart documentation of the following: presence of a living will; durable power of attorney identified; do-not-resuscitate (DNR) order in chart; involvement of social work; involvement of spiritual care; palliative care consults; occurrence of a family conference; discussions of prognosis at family conferences; family expressing wishes to withdraw life-sustaining treatments; family present at time of death; and patient dying after a decision to limit life-sustaining treatments.^18,19 In addition, some of these indicators of palliative care have been shown to be associated with high family ratings of the quality of dying²⁰ or high ratings of family satisfaction with care,²¹ providing external validation of their usefulness as indicators of quality palliative care. Table 1 shows the derivation and prior validation of the chart-based indicators of palliative care. Each indicator was derived from one of several prior publications^18–21 regarding potential indicators of palliative care in the hospital ICU setting. Table 1 also provides the rationale for inclusion of each indicator in the current study.

Table 1.

Derivation and Prior Validation of the Indicators of Palliative Care

Variables Documented in the Medical Record	Derived from Prior Statements on End-of-Life Care18,19	Previously Associated With Family Ratings of Satisfaction With Care21	Previously Associated With Family Ratings of the Quality of Dying20
DNR order in chart	X		X
Power of attorney identified	X
Living will present	X		X
No CPR in last 24 h of life	X		X
Occurrence of family conference	X		X
Prognosis discussed at family conference	X
Social work involvement	X
Palliative care involvement	X
Spiritual care involvement	X	X
Family expressed wishes to withdraw life-sustaining therapies	X	X
Patient dying after decision to limit life-sustaining therapies	X	X	X
Family present at death	X		X

CPR = cardiopulmonary resuscitation.

Statistical Analysis

The χ² test was used to compare response rates across the 14 hospital sites. Multivariate logistic regression was performed to identify independent predictors of family response or nonresponse to the survey. We controlled for hospital site in all analyses by using dummy variables. To examine demographic variables, the multivariate model included all demographic variables and hospital site in order to identify independent associations between each demographic variable and the odds of responding to the survey. To examine chart documentation of indicators of palliative care, the multivariate logistic regression model for each indicator was adjusted for patient age, gender, and race as well as for hospital site. All analyses used robust SEs to correct for heteroskedasticity within the data. Significance for all analyses was defined as p ≤ 0.05.

Results

There was a total of 2,083 eligible deaths during the study period. Among these patients, the complete medical record could not be found for 67. Of the remaining 2,016 eligible patients with chart abstraction data, 760 (37.7%) had a family member return the survey packet (Fig 1). Response rates to the survey varied by hospital site (Fig 2), ranging from 20.8% to 63.0%, and were significantly different across all 14 sites (p < 0.001).

Figure 1.

Development of the study population and response rate.

Figure 2.

Variation in response rates at the 14 hospital sites. Each bar represents a different hospital site; the number at the top of the bar shows the total number of eligible patients at each hospital site.

In the analyses of demographic characteristics, adjusting for potential confounders (Table 2), family members of patients who were white (p < 0.001), older (p = 0.006), and married (p < 0.001) were more likely to respond to the survey than family members of nonwhite, younger, or unmarried patients. There were no differences by patient gender, type of insurance, hospital ICU length of stay, patient education level, admission source to the hospital, trauma as a cause of death, or patient median income estimated by home zip code (Table 2).

Table 2.

Demographics of Patients With Responding and Nonresponding Family Members

Variables	Responders (n = 760)	Nonresponders (n = 1,256)	Odds Ratio*	95% CI	p Value
Age, yr	71.35 ± 14.66	69.30 ± 15.55	1.01	1.00–1.02	0.006
Male gender	452 (59.5)	726 (57.8)	0.89	0.62–1.10	0.326
White race	666 (87.6)	933 (74.3)	1.56	1.42–1.67	< 0.001
Insurance
Private	136 (17.9)	195 (15.5)	Reference		0.326
Public	590 (77.6)	1,000 (79.6)	0.81	0.59–1.11
None or unknown	34 (4.5)	61 (4.9)	0.98	0.60–1.62
ICU length of stay, h†	60.3 (23.7–157.3)	68 (28.2–169.6)	1.00	1.00–1.00	0.235
Education‡
< 8th grade	47 (6.2)	101 (8.3)	Reference		0.337
Some high school	60 (7.9)	117 (9.6)	0.96	0.58–1.59
High school/GED	295 (39.1)	500 (41.2)	1.10	0.73–1.65
Some college	193 (25.6)	281 (23.1)	1.28	0.83–1.97
4-yr college	105 (13.9)	142 (11.7)	1.43	0.90–2.28
Postcollege	55 (7.3)	75 (6.2)	1.29	0.76–2.19
Marital status§
Never married	54 (7.2)	147 (11.8)	Reference		< 0.001
Married	436 (57.7)	498 (40.0)	1.81	1.23–2.66
Divorced/separated	103 (13.6)	304 (24.4)	0.71	0.47–1.09
Widowed	162 (21.5)	306 (24.6)	1.08	0.69–1.68
Admission source‖
Home	478 (63.7)	775 (62.7)	Reference		0.125
Facility	107 (14.3)	239 (19.3)	0.76	0.58–1.01
Acute/office	164 (21.9)	209 (16.9)	1.07	0.81–1.41
Homeless	1 (0.1)	12 (1.0)	0.27	0.03–2.52
Trauma as cause of death†	62 (8.2)	88 (7.0)	0.86	0.58–1.27	0.454
Median income†	50,385 ± 13,256	49,171 ± 13,272	1.00	1.00–1.00	0.466

Values are given as the mean ± SD, No. (%), or median (interquartile range) unless otherwise indicated. GED = general educational development.

*The odds ratio is a multivariate model that includes patient age, gender, race, insurance type, hospital ICU length of stay, education, marital status, admission source, trauma as cause of death, median household income by home zip code, and hospital site.

†Two missing.

‡Forty-five missing.

§Fifteen missing.

‖Thirty-one missing.

In the analyses of the nine palliative care indicators, adjusting for potential confounders (patient age, gender, race, and hospital site), three were significantly different between patients whose families responded to the survey and those whose families did not and showed that family members were more likely to respond to the survey if there was chart documentation of the indicator of palliative care (Table 3). These three indicators were family presence at the time of death, dying after a decision to limit life-sustaining therapies, and spiritual care involvement while in the hospital ICU. There were no significant differences in the other indicators of palliative care, which were cardiopulmonary resuscitation in the last 24 h of life, occurrence of a family conference, discussion of prognosis at family conferences, family wishing to withdraw life-sustaining therapies, DNR order in the chart, or social work involvement.

Table 3.

Odds Ratio for Responding to Survey When There Is Documentation in the Medical Record of Having Received Palliative Care Indicators

Variables	Responders (n = 760)*	Nonresponders (n = 1,256)*	Odds Ratio†	95% CI	p Value
Family present at death	574 (79.9)	852 (72.3)	1.62	1.29–2.06	< 0.001
Died after withdrawal of life support	578 (76.9)	865 (69.2)	1.25	1.06–1.39	0.010
Spiritual care	360 (47.5)	534 (42.6)	1.23	1.00–1.52	0.050
No CPR in last 24 h of life	640 (85.4)	1,023 (81.9)	1.19	0.95–1.38	0.109
Prognosis discussed	305 (40.6)	465 (37.2)	1.19	0.98–1.45	0.078
Family wishes to withdraw support	361 (48.1)	533 (42.7)	1.17	0.97–1.42	0.108
Family conference	578 (77.0)	935 (74.9)	1.13	0.90–1.41	0.280
DNR order	619 (82.5)	1,007 (80.5)	1.05	0.82–1.34	0.725
Social work	305 (40.3)	479 (38.2)	0.92	0.74–1.15	0.467

Values are presented as No. (%), unless otherwise indicated. See Table 1 for abbreviation not used in the text.

*May not add to total due to missing data.

†Odds ratio for responding to survey when indicator of palliative care is documented in the medical record, adjusted for patient age, gender, race, and hospital site.

Discussion

This study provides a unique opportunity to examine the potential for nonresponse bias in survey-based research on the quality of end-of-life care in the hospital ICU. Strengths of this study include the large number of patients who died in the ICU; the variety and number of hospitals from which the study population was drawn, including academic and community hospitals; and, perhaps most importantly, the ability to assess differences in demographics and indicators of palliative care among patients with and without family member respondents to a survey of end-of-life care. The response rate to the survey in this study is typical of survey research of this kind.^11–13,22 Our study also documented significant variability in response rates across different institutions that was not explained by teaching status or other available hospital characteristics.

This study shows that family members of patients who were white were more likely to respond to the survey and, therefore, minority patients are underrepresented. Patients did not differ by gender, type of insurance, education level, or estimated median income. This finding is consistent with reports^23,24 of racial differences in both research participation and preferences regarding hospital ICU care. Black patients, on average, prefer and receive more aggressive life-sustaining treatments than white patients²⁴; therefore, the differences in response rates between white and nonwhite patients' family members may introduce additional bias into the assessment of quality of end-of-life care in the hospital ICU.

Patients whose family members responded to the survey received more indicators of palliative care during their end-of-life care in the hospital ICU and, therefore, likely received higher quality palliative care in the ICU. This finding has implications for studies using survey-based research to assess the quality of end-of-life care because such studies, if based on responders alone, may be overestimating the overall quality of end-of-life care in the ICU. This bias likely results in a systematic overstatement of the quality of end-of-life care that patients and their families are experiencing.^25,26

There are a variety of reasons why some deceased patients did not have family respondents. First, some patients may not have had family members, or their family members may not have received the surveys. Our finding that the families of patients who were married returned significantly more surveys than the families of unmarried patients provides some support for this hypothesis. Because we did not have family contact information for 13 of the 14 hospitals, we mailed the surveys to the home of the deceased patient, and this could have resulted in some of the surveys not reaching family members. Furthermore, there are important differences in the preferences for end-of-life care and the actual end-of-life care received by patients with critical illness who do not have family available for surrogate decision making, and this may contribute to the differences we identified.^27–29

A second reason for nonresponse is that some family members received surveys but elected not to complete them. For any respondent, there are both costs and rewards associated with completing a survey. Participants are most likely to return a survey when these factors are balanced or when the rewards are seen as greater than the costs; increasing response rates from these surveys likely will require decreasing the costs and increasing the rewards. The main costs of participation are time and perceived respondent burden. Decreasing cost can be accomplished by shortening surveys and by developing surveys that are esthetically pleasing, minimally distressing, and interesting to participants. There are also many types of rewards to participating in survey research. These include financial reward, interest in expressing one's opinions, altruism, and interest in the topic. In this study, an attempt was made to minimize nonresponse by providing a cover letter explaining the purpose of the study, sending a cash incentive with the survey packet ($10), and making multiple contacts.³⁰ Attempts were made to reduce the respondent burden by making the survey easy to understand and user friendly. However, the survey was relatively long and contained multiitem instruments that assessed both family satisfaction with care and family ratings of the quality of dying. Additionally, the survey was only administered in English, which likely resulted in nonresponse from non-English-speaking family members.^12,16,17

There are several limitations to this study. First, although patients and their families were identified from 14 different hospital sites, these were all within the Seattle-Tacoma area. These results may not generalize to other regions with different patient or family characteristics. Second, these results may be specific to surveys of end-of-life care in the hospital ICU and may not generalize to other types of research in the ICU or other settings in which patients die. Third, we were unable to examine potential hospital-level predictors of nonresponse due to the relatively small number of hospitals. However, we did control for hospital in the multivariate analyses, providing some assurance that our findings are not confounded by hospital site. Fourth, we determined patient race using death certificate data, which may introduce misclassification. However, death certificate data are much more complete than the medical record for race, and we neither had access to a patient's own determination of race nor asked family members to identify the race of the patient. Finally, we were unable to distinguish among the various reasons for nonresponse. Because of the study design and that most surveys were sent to the deceased patient's home, it is possible that surveys did not reach family members. Other than the types of nonresponse summarized in Figure 1, we are unable to assess differences between those who received the survey and chose not to respond and those who never received the survey.

This study provides unique insights into differences between those who respond or do not respond to surveys about quality of end-of-life care in the hospital ICU. Our findings suggest that the quality of end-of-life care received by patients whose families respond to surveys is higher than that received by patients whose families do not respond to surveys. As such, the nonresponse bias may be overestimating the overall quality of end-of-life care in the hospital ICU. Additionally, because patients whose family members responded were more likely to be white, minorities are underrepresented. Future studies are needed to identify ways to improve survey response rates and minimize the potential for nonresponse bias.

Abbreviation:

DNR

do not resuscitate