Abstract
Context
The Centers for Medicare & Medicaid Services have elected to include a bereaved family member survey in public reporting of hospice quality data as mandated in the Affordable Care Act. However, it is not known what timepoint after death offers the most reliable responses.
Objectives
To examine the stability of bereaved family members’ survey responses when administered three, six and nine months after hospice patient death.
Methods
Bereaved family members from six geographically diverse hospices were interviewed three, six, and nine months after patient death. All respondents completed a core survey. Those whose family member died at home, in a free-standing inpatient unit, or in a nursing home also completed a site-specific module. Stability was based on top-box scoring of each item with kappa statistics, and multivariable regression models were used to assess directionality and predictors of change. To analyze the effects of grief, we assessed response stability among respondents at least one standard deviation from the mean change in grief between three and six months.
Results
We had 1532 surveys (536 three-month surveys, 529 six-month surveys, and 467 nine-month surveys) returned by 643 respondents (average age 61.7 years, 17.4% Black, 50.5% a child respondent) about hospice decedents (55.3% female, average age 78.6 years, 57.0% non-cancer, 40.0% at home.) The average kappa for core items between three and nine months was 0.54 (range: 0.42-0.74), 0.58 (0.41-0.69) for home-specific items, and 0.54 (0.39-0.63) for nursing home. Even among individuals demonstrating large grief changes, core items demonstrated moderate to high stability over time.
Conclusion
Bereaved family member responses are stable between three and nine months after the death of the patient.
Keywords: Hospice, bereavement, grief, response stability, quality measurement
Introduction
The provision of hospice and palliative care services for patients at the end of life is growing (1). In 2012, the National Hospice and Palliative Care Organization estimates that over 1.5 million patients received services from hospice. Ensuring high quality of hospice care for the dying, therefore, is essential. Bereaved family interviews are a common and useful way to gather information on hospice quality of care, as evidenced by their use in several influential studies to date (2-5).
The Affordable Care Act (6) requires the Centers for Medicare & Medicaid Services (CMS) to publicly report hospice quality data. CMS has elected to use a bereaved family member survey as part of its Hospice Quality Reporting Program because of its several advantages over prospective interviews with hospice patients. First, the retrospective approach ensures that a similar time frame is compared, which is important given changes in utilization and symptoms closer to death. Second, prospectively identifying dying patients is difficult secondary to limitations of prognostication (7). Third, many dying patients are unable to be interviewed in the last weeks of life and there is an important concern with respondent burden when interviewing a person close to death. Among seriously ill, hospitalized persons enrolled in the Study to Understand Prognoses and Preferences for Outcomes and Risks of Treatment, Wenger and colleagues found that 40% of patients were unable to be interviewed (8). Those unable to be interviewed were sicker and closer to death. Key questions to consider are how bereavement impacts a family member's ratings and perceptions of the quality of care, and whether timing of survey administration will affect publicly reported hospice quality data.
Current research regarding stability of bereaved family member response is conflicting. Casarett and colleagues (9) found respondents interviewed between two and six weeks post death were stable with regard to self-ratings of distress and satisfaction with care, whereas a study by Higginson and colleagues (10) raised concerns with the stability of bereaved respondent perceptions of symptoms and emotional state. Cartwright et al. (11) found no significant differences between bereaved family reports and direct patient reports in factual information such as cause of death, place of death and demographic information, but some small differences in responses concerning patients’ symptoms, care received, and need for help. The National Hospice and Palliative Care Organization (NHPCO) has recommended that the survey be administered between one and three months after death. This decision was based on the a priori logic that this time period was far enough from death to not be burdensome, but close enough that details of care could be recalled. The choice of the timing of survey administration must weigh tradeoffs between the impact of bereavement on the respondent's perceptions and the increased difficulty in locating respondents who move after the death of their loved one. An important policy question with the public reporting of hospice based on bereaved family interviews is whether the timing of the surveys impacts responses to quality of care questions.
The goal of the present study was to assess the stability of bereaved family member responses to a hospice quality survey and to identify a time point most appropriate for administering such a survey. Because the stability of bereaved responses over time may be affected by changes in grief, a secondary goal of this study was to assess whether changes in grief impact perceptions of quality of care at the end of life.
Methods
Sample
We recruited six hospice programs from diverse geographic regions of the country, including rural and urban locations and areas serving Black and Hispanic populations. Survey administration was attempted from each participant at three, six, and nine months after the death of the hospice patient. All eligible respondents were sent a three-month survey and a six-month survey; only eligible respondents who had completed a three-month survey and/or a six-month survey were sent a survey at nine months. Potential respondents were mailed an information packet about the study and a toll-free phone number to opt out of the study. A self-administered survey also was included in the packet. Up to five follow-up calls at different times of day were made to individuals who did not return their survey within two weeks. For those who chose to do so, we conducted a telephone-administered version of the survey. Participants enrolled in this study were the next of kin or close relatives or friends of a person who died while receiving hospice services at home, in a nursing home, or in a free-standing hospice inpatient unit (IPU). English and Spanish versions of the instrument were available to participants. All data collection was conducted through the Survey Center in the School of Public Health at Brown University.
Measures
The revised Family Evaluation of Hospice Care (FEHC) comprises core items for all sites of care and three site-specific modules. Core and setting-specific items were based on review of the existing guidelines, input of an expert panel, and focus groups and cognitive-based interviews with bereaved family members. These items included survey questions from the original FEHC (12,13) survey and new items that were created in the following three modules: 1) a module that focused on the care of hospice patients in the nursing home; 2) a module that focused on care of the hospice patient in the free-standing IPU; and 3) a module that examined hospice's important role in educating and training the family member to provide hands-on care to the dying patient in the home setting. All items were scored based on whether the family member's or close friend's response indicated there was an opportunity to improve the care received by the patient.
Three types of response tasks were used to measure respondents’ perceptions of the quality of care. The first type relied on the methodology of patient-centered reports, which asked respondents to report on their perceptions of the quality of care using response categories of “yes/no” or a four-point scale of “always,” “usually,” “sometimes,” and “never.” The second type of response task measured patient unmet needs by rating the hospice's response to a need as “less than was wanted,” “more than was wanted,” or “the right amount.” The final type of response task measured unmet needs by rating the information received as “all of the [information or training] that was needed,” “most of the [information or training] that was needed,” “some of the [information or training] that was needed,” or “none of the [information or training] that was needed.”
Assessment of complicated grief was done using the Inventory of Complicated Grief—Revised—Short Form (ICG–R) by Prigerson et al. (14). The ICG–R short form is an 11-item evaluation of complicated (as opposed to normal) grief responses over the previous month. The psychometric properties of the instrument have been previously reported (14). For the purposes of our study, the ICG-R short form was used as a measure of symptom intensity rather than as a tool for the diagnosis of complicated grief, with a higher score on the ICG-R indicating more grief.
Analytic Approach
Because high-quality medical care strives for excellence, responses to each question were recoded using problem-based scoring to create dichotomous measures that indicated opportunities to improve (i.e., concerns with care). For the unmet needs questions, responses of “less than was wanted” and “more than was wanted” were considered opportunities to improve and assigned a value of one. Similarly, responses indicating the respondent received anything less than “all of the [information or training] that was needed” were considered opportunities to improve and assigned a value of one.
We first conducted descriptive analyses on demographic characteristics and involvement in care for our sample at each time point using the Chi-square statistic to test for significant differences in demographics and care involvement among respondents at each time point. Stability of bereaved family member response was examined with kappa statistics. All items were dichotomized. As recommended by Landis and Koch (15), we considered a kappa of less than 0.40 to indicate poor agreement. Feinstein and Cicchetti noted an important paradox of high agreement but low kappa as a result of imbalances in marginal totals (16,17). This paradox is explained by the fact that kappa statistics are dependent on the prevalence of the finding under observation. P-positive and p-negative statistics correct for this kappa limitation by distinguishing between agreement on positive findings (p-positive) and agreement on negative findings (p-negative). If, for example, negative findings have low agreement as a result of there being only a small number of observations, reporting a p-positive value will illustrate that the majority of responses are in agreement despite the few negative findings creating the low kappa. A large p-positive and a small p-negative, or vice versa, indicates a skewed marginal distribution that is the likely cause of a low kappa in such circumstances. When we encountered low kappa values, we utilized the approach proposed by Cicchetti and Feinstein of examining the separate omnibus index for the proportion of positive and negative agreement in these instances (16).
We also examined the association of grief and stability of responses. Because the passage of time is often accompanied by changes in grief, it was important for us to determine whether respondents with large changes in grief impacted reports of care quality. To accomplish this, we used the Kappa statistic to capture the association of grief changes between three and six months with the stability of responses to items. Grief was divided into three categories; increasing grief, decreasing grief, and constant grief. The ICG-R short form grief scale was operationalized such that having a symptom “not at all” was given a value of zero and having a symptom “several times a day” or “overwhelmingly” was assigned a value of 5. This operationalization left us with a scale of grief from 0, indicating the least grief, to 45 indicating the most grief. To analyze the change in grief among our respondents, we defined increasing grief to mean at least one standard deviation above the mean increase in grief in the sample. Likewise, decreasing grief was defined as at least one standard deviation beyond the mean decrease in grief. One standard deviation in our sample was equivalent to five points on the grief scale.
This research was approved by the Brown University Institutional Review Board and all analyses were completed using STATA version 13 (18).
Results
Table 1 reports the characteristics of decedents who died at home (n=612, 40.0%), at a nursing home (n=399, 26.0%), or in a free-standing IPU (n=521, 34.0%). There were no significant differences in demographics or involvement in care between those who responded to the survey at each time point, nor were there significant differences in characteristics of decedents being represented at each time period. Decedents were slightly more likely to be female (55.3%) and the average age was 78.6 years (SD = 14.1). Seventy-two percent were White, 17.8% Black, 6.8% Hispanic and 3.2% another race or multiracial. A large percentage (42.9%) of decedents had cancer as the primary diagnosis. The median length of hospice stay was two weeks (14 days). We had limited demographic information on eligible non-respondents. Compared to decedents of respondents, decedents of non-respondents were slightly younger (76.4 years), and were more likely to be Black (28.9%) or Hispanics (11.7%)
Table 1.
Characteristics of Decedents and Respondents
| Decedent Characteristics | 3 month (n=537) | 6 month (n=529) | 9 month (n=467) |
|---|---|---|---|
| Site of death (n, %) | |||
| Home | 224 (41.7) | 209 (39.5) | 179 (38.3) |
| Nursing home | 174 (32.4) | 182 (34.4) | 165 (35.3) |
| Inpatient unit | 139 (25.9) | 138 (26.1) | 123 (26.3) |
| Patient age (mean, 95% CI) | 78.9 (77.7-80.1) | 78.5 (77.3-79.8) | 78.5 (77.2-79.8) |
| Patient gender (%, 95% CI) | |||
| Female | 54.9 (50.7-59.2) | 56.5 (52.3-60.8) | 54.4 (49.9-58.9) |
| Patient race (%, 95% CI) | |||
| Non-Hispanic White | 70.9 (67.1-74.8) | 71.1 (67.2-75.0) | 73.7 (69.7-77.8) |
| Non-Hispanic Black | 18.1 (14.8-21.3) | 18.7 (15.4-22.0) | 16.5 (13.1-19.9) |
| Hispanic | 7.3 (5.1-9.5) | 7.0 (4.8-9.2) | 6.0 (3.8-8.2) |
| Multiracial/Other | 3.5 (1.0-6.2) | 3.0 (0.8-5.5) | 3.2 (0.9-5.5) |
| Patient education (%, 95% CI) | |||
| Less than high school | 26.8 (23.1-30.6) | 24.3 (20.6-28.0) | 25.0 (21.0-29.0) |
| High school graduate/GED | 34.3 (30.3-38.4) | 34.2 (30.1-38.8) | 33.7 (29.4-38.0) |
| Some college or higher | 38.8 (34.7-43.0) | 41.5 (37.3-45.7) | 41.3 (36.8-45.8) |
| Primary diagnosis (%, 95% CI) | |||
| Cancer | 42.1 (37.9-46.3) | 43.3 (39.1-47.5) | 43.5 (39.0-48.0) |
| Heart/circulatory disease | 9.9 (7.3-12.4) | 10.0 (7.5-12.6) | 10.5 (7.7-13.3) |
| Lung/breathing disease | 8.2 (5.9-10.5) | 8.7 (6.3-11.1) | 6.9 (4.6-9.2) |
| Kidney disease | 3.7 (2.1-5.3) | 3.2 (1.7-4.7) | 3.2 (1.6-4.8) |
| Liver disease | 2.8 (1.4-4.2) | 2.1 (0.9-3.3) | 2.8 (1.3-4.3) |
| Stroke | 3.2 (1.7-4.7) | 4.0 (2.3-5.6) | 4.3 (2.4-6.1) |
| Dementia/Alzheimer's | 11.2 (8.5-13.8) | 12.7 (9.8-15.5) | 12.6 (9.6-15.7) |
| AIDS/other infectious diseases | 0.4 (−0.1-0.9) | 0.6 (−0.1-1.2) | 0.4 (−0.2-1.0) |
| Frailty and decline due to old age | 11.0 (8.3-13.6) | 9.5 (7.0-12.0) | 10.5 (7.7-13.3) |
| Other | 7.1 (4.9-9.3) | 5.5 (3.5-7.4) | 4.9 (3.0-6.9) |
| Length of hospice stay in days (median, 25th-95th percentile) | 14 (6-60) | 14 (6-60) | 14 (6-60) |
| Length of hospice stay <=7 days (%, 95% CI) | 36.4 (32.1-40.8) | 35.4 (32.2-39.9) | 34.9 (30.2-39.5) |
| Respondent Characteristics | |||
| Respondent age (mean, 95% CI) | 61.6 (60.5-62.6) | 61.7 (60.6-62.7) | 62.1 (61.0-63.2) |
| Respondent gender (%, 95% CI) | |||
| Female | 74.7 (71.0-78.4) | 73.3 (69.6-77.1) | 75.6 (71.7-79.5) |
| Respondent race (%, 95% CI) | |||
| Non-Hispanic White | 70.1 (66.9-74.6) | 71.1 (67.2-75.0) | 75.2 (71.2-79.1) |
| Non-Hispanic Black | 17.7 (14.5-20.9) | 18.1 (14.9-21.4) | 16.1 (12.7-19.4) |
| Hispanic | 7.4 (5.2-9.7) | 7.6 (5.3-9.8) | 6.2 (4.0-8.4) |
| Multiracial/Other | 3.5 (0.8-6.3) | 3.2 (0.6-5.8) | 2.6 (0.2-4.9) |
| Respondent education (%, 95% CI) | |||
| Less than high school | 7.1 (4.9-9.3) | 6.6 (4.5-8.8) | 7.7 (5.3-10.2) |
| High school graduate/GED | 24.5 (20.8-28.1) | 26.7 (22.9-30.5) | 25.1 (21.2-29.1) |
| Some college or higher | 68.4 (64.5-72.4) | 66.7 (62.6-70.7) | 67.2 (62.9-71.4) |
| Relationship to patient (%, 95% CI) | |||
| Spouse/partner | 32.4 (27.5-37.3) | 32.5 (27.6-37.4) | 33.0 (27.8-38.2) |
| Child | 51.2 (47.0-55.5) | 50.5 (46.2-54.7) | 49.9 (45.3-54.4) |
| Parent | 1.3 (0.3-2.3) | 1.7 (0.6-2.8) | 1.9 (0.7-3.2) |
| Sibling | 6.5 (4.4-8.6) | 5.5 (3.5-7.4) | 4.9 (3.0-6.9) |
| Other relative | 1.7 (0.6-2.8) | 2.1 (0.9-3.3) | 3.2 (1.6-4.8) |
| Friend | 1.3 (0.3-2.7) | 1.3 (0.3-2.3) | 1.3 (0.3-2.3) |
| Other | 5.6 (3.6-7.5) | 5.9 (3.9-7.9) | 5.1 (3.1-7.1) |
| Previous experience caring for dying (%, 95% CI) | 41.7 (37.5-45.9) | 38.9 (34.8-43.1) | 42.0 (37.5-46.5) |
| Previous experience with hospice (%, 95% CI) | 34.1 (30.1-38.1) | 31.2 (27.2-35.2) | 35.1 (30.8-39.5) |
| Involved in medical decisions (%, 95% CI) | 98.7 (97.7-99.7) | 97.7 (96.5-99.0) | 98.3 (97.1-99.5) |
| Power of attorney or health care proxy (%, 95% CI) | 78.4 (74.9-81.9) | 78.1 (74.5-81.6) | 76.0 (72.1-79.9) |
| Involved in care (%, 95% CI) | |||
| Always | 79.3 (75.9-82.8) | 76.4 (72.7-80.0) | 78.2 (74.4-81.9) |
| Usually | 12.8 (10.0-15.7) | 15.9 (12.8-19.0) | 12.6 (9.6-15.7) |
| Sometimes | 7.3 (5.1-9.5) | 7.2 (5.0-9.4) | 8.4 (5.8-10.9) |
| Involved in treatment decisions in last week of life (%, 95% CI) | |||
| Not at all | 1.5 (0.5-2.5) | 1.7 (0.6-2.8) | 1.7 (0.5-2.9) |
| A little bit | 3.0 (1.5-4.4) | 5.3 (3.4-7.2) | 2.4 (1.0-3.7) |
| Somewhat | 7.4 (5.2-9.7) | 6.0 (4.0-8.1) | 9.0 (6.4-11.6) |
| Very much | 87.7 (84.9-90.5) | 86.4 (83.5-89.3) | 86.1 (82.9-89.2) |
| Number of days of contact with patient in last week of life (mean, 95% CI) | 6.2 (6.0-6.3) | 6.1 (6.0-6.3) | 6.2 (6.1-6.4) |
| Completed 3-month survey only (%) | 11.4 | n/a | n/a |
| Completed 6-month survey only (%) | n/a | 7.5 | n/a |
The response rate at three months was 49.4%. Among those who did not complete a three-month survey, 100 individuals completed a six-month survey. Fifty-seven percent of eligible participants completed either a three-month survey, a six-month survey, or both.
Respondents were 61.5 years old on average, mostly female (74.5%), and usually the child (46.9%) or spouse (32.1%) of the decedent (Table 1). Seventy-two percent identified as non-Hispanic White, 17.4% non-Hispanic Black, 7.1% Hispanic and 3.2% another race or multiracial. Most had no experience caring for a dying person (58.6%) and no experience with hospice (65.9%), but nearly all identified themselves as the person who was or would have been involved in medical decisions for the patient (98.2%). Nearly 92% always or usually participated or provided oversight in caring for the patient. Most respondents (86.7%) reported being “very much” involved in treatment decisions in the last week of life, and the average number of days in the last week of life that respondents had contact with the decedent was 6.2 (median=7 days, % no days=3.7%).
Table 2 shows the stability of item responses from three to six months, six to nine months, and three to nine months. The majority of kappas were above the 0.40 cut-off determined by Landis and Koch (15) that suggests an acceptable level of agreement. With the exception of two items—“ How much training did you receive about how to deliver medicines when the patient had difficulty swallowing?” and “How often after hospice became involved did you feel you were on your own to advocate for the patient?”—all low kappas (≤0.40) could be explained by the skewed marginal effects, as described by Feinstein and Cicchetti (16,17), and explained above, in which a low kappa results despite high percent agreement in the data. We report the p-positive and p-negative values for items with skewed marginal effects in Table 2.
Table 2.
Stability of Item Responses Over Nine Months After Death
| Table 2A. Core Survey Items | |||
|---|---|---|---|
| Item | 3 to 6 months | 3 to 9 months | 6 to 9 months |
| While under the care of hospice, did the patient have pain or take medicine for pain? | |||
| % agreement | 93.5 | 94.1 | 94.8 |
| Kappa | 0.71 | 0.74 | 0.78 |
| How much help in dealing with his/her pain did the patient receive? | |||
| % agreement | 93.9 | 96.7 | 97.1 |
| Kappa | 0.50 | 0.74 | 0.73 |
| Did the patient have trouble breathing or receive treatment for trouble breathing? | |||
| % agreement | 84.6 | 82.8 | 86.5 |
| Kappa | 0.68 | 0.65 | 0.72 |
| How much help in dealing with breathing did the patient receive? | |||
| % agreement | 94.5 | 93.7 | 95.5 |
| Kappa | 0.56 | 0.54 | 0.58 |
| Did the patient have any feelings of anxiety or sadness? | |||
| % agreement | 84.1 | 86.3 | 84.5 |
| Kappa | 0.64 | 0.71 | 0.67 |
| How much help in dealing with anxiety/sadness did the patient receive? | |||
| % agreement | 94.9 | 95.0 | 95.1 |
| Kappa | 0.69 | 0.71 | 0.71 |
| How much information did you receive about pain medicines? | |||
| % agreement | 90.1 | 91.6 | 93.1 |
| Kappa | 0.43 | 0.47 | 0.55 |
| How much information did you receive about pain medicine side effects? | |||
| % agreement | 82.8 | 82.5 | 90.5 |
| Kappa | 0.56 | 0.54 | 0.72 |
| How much information did you receive about trouble breathing? | |||
| % agreement | 82.3 | 85.6 | 87.3 |
| Kappa | 0.42 | 0.51 | 0.60 |
| How often did the hospice team keep you informed about the patients condition? | |||
| % agreement | 89.2 | 85.0 | 86.7 |
| Kappa | 0.67 | 0.54 | 0.58 |
| How much information did you receive about what to expect when the patient was dying? | |||
| % agreement | 88.6 | 88.2 | 90.8 |
| Kappa | 0.60 | 0.57 | 0.65 |
| Did you receive enough contact from hospice about your religious and spiritual beliefs? | |||
| % agreement | 93.2 | 89.9 | 92.3 |
| Kappa | 0.56 | 0.42 | 0.49 |
| How much emotional support did you get from hospice prior to the patient's death? | |||
| % agreement | 93.1 | 93.7 | 95.4 |
| Kappa | 0.39 | 0.46 | 0.57 |
| p-positive | 0.96 | ||
| p-negative | 0.43 | ||
| How much emotional support did you get from hospice after the patient's death? | |||
| % agreement | 91.3 | 92.7 | 93.7 |
| Kappa | 0.37 | 0.45 | 0.50 |
| p-positive | 0.95 | ||
| p-negative | 0.42 | ||
| How often did someone from hospice give confusing or contradictory information? | |||
| % agreement | 90.6 | 90.2 | 89.2 |
| Kappa | 0.59 | 0.55 | 0.48 |
| Was there always one hospice nurse in charge of the patient's care? | |||
| % agreement | 92.0 | 89.8 | 91.0 |
| Kappa | 0.59 | 0.51 | 0.55 |
| Were there problems with the hospice team not knowing enough about the patient's medical history? | |||
| % agreement | 94.1 | 94.7 | 95.2 |
| Kappa | 0.52 | 0.53 | 0.47 |
| How often did hospice answer your questions about the patient's care? | |||
| % agreement | 89.7 | 88.0 | 92.9 |
| Kappa | 0.60 | 0.57 | 0.71 |
| How often was the hospice team available to speak with you about concerns? | |||
| % agreement | 86.5 | 86.2 | 85.6 |
| Kappa | 0.60 | 0.61 | 0.58 |
| Did you have any concerns with the quality of the patient's care? | |||
| % agreement | 92.5 | 94.0 | 92.9 |
| Kappa | 0.56 | 0.64 | 0.51 |
| Overall how would you rate the quality the patient received while under the care of hospice? | |||
| % agreement | 88.6 | 85.8 | 86.7 |
| Kappa | 0.70 | 0.64 | 0.66 |
| How would you rate the way the hospice team responded in evenings and weekends? | |||
| % agreement | 81.6 | 76.4 | 78.9 |
| Kappa | 0.61 | 0.51 | 0.57 |
| Based on the care the patient received, would you recommend this hospice to others? | |||
| % agreement | 91.6 | 89.8 | 91.2 |
| Kappa | 0.62 | 0.58 | 0.63 |
| Average core Kappa (excluding skewed marginals) | 0.56 | 0.54 | 0.58 |
| Number of core Kappa ≤0.40 (excluding skewed marginals) | 0 | 0 | 0 |
| Table 2B. Home Module Items | |||
|---|---|---|---|
| Item | 3 to 6 months | 3 to 9 months | 6 to 9 months |
| How much training did you receive about pain medicine side effects? | |||
| % agreement | 79.6 | 79.1 | 82.7 |
| Kappa | 0.46 | 0.48 | 0.57 |
| How much training did you receive about giving extra doses of medicine? | |||
| % agreement | 88.1 | 83.9 | 82.4 |
| Kappa | 0.65 | 0.59 | 0.55 |
| How much training did you receive about how to deliver medicines when the patient had difficulty swallowing? | |||
| % agreement | 82.1 | 80.2 | 77.6 |
| Kappa | 0.48 | 0.41 | 0.37 |
| p-positive | 0.85 | ||
| p-negative | 0.52 | ||
| How much training did you receive to help the patient with breathing? | |||
| % agreement | 88.5 | 85.7 | 86.6 |
| Kappa | 0.68 | 0.64 | 0.65 |
| How much information did you receive about agitation? | |||
| % agreement | 83.9 | 87.5 | 90.0 |
| Kappa | 0.51 | 0.62 | 0.69 |
| How much training did you receive about what to do when the patient was agitated? | |||
| % agreement | 86.8 | 84.4 | 82.0 |
| Kappa | 0.67 | 0.61 | 0.54 |
| How much information did you receive about how to safely move the patient? | |||
| % agreement | 89.8 | 90.1 | 90.3 |
| Kappa | 0.65 | 0.69 | 0.67 |
| How much training did you receive about how to safely move the patient? | |||
| % agreement | 83.8 | 84.1 | 85.3 |
| Kappa | 0.58 | 0.61 | 0.64 |
| Average home Kappa (excluding skewed marginals) | 0.59 | 0.58 | 0.59 |
| Number of home Kappa ≤0.40 (excluding skewed marginals) | 0 | 0 | 1 |
| Table 2C. Nursing Home Module Items | |||
|---|---|---|---|
| Item | 3 to 6 months | 3 to 9 months | 6 to 9 months |
| How much information did you receive about hospice before you decided to enroll? | |||
| % agreement | 88.7 | 93.9 | 88.5 |
| Kappa | 0.53 | 0.63 | 0.35 |
| p-positive | 0.94 | ||
| p-negative | 0.40 | ||
| How much contact did you have with the hospice staff? | |||
| % agreement | 94.4 | 92.2 | 95.3 |
| Kappa | 0.64 | 0.51 | 0.64 |
| How often was the hospice team aware of the patient's condition? | |||
| % agreement | 80.9 | 84.7 | 82.4 |
| Kappa | 0.55 | 0.57 | 0.55 |
| How often after hospice became involved did you feel you were on your own to advocate for the patient? | |||
| % agreement | 71.0 | 69.6 | 76.2 |
| Kappa | 0.42 | 0.39 | 0.51 |
| p-positive | 0.72 | ||
| p-negative | 0.67 | ||
| Were you provided with enough information to know which concerns to discuss with the nursing home and which to discuss with hospice? | |||
| % agreement | 83.5 | 87.9 | 91.1 |
| Kappa | 0.42 | 0.58 | 0.67 |
| How often did hospice and the nursing home work as a team? | |||
| % agreement | 85.7 | 80.8 | 86.8 |
| Kappa | 0.70 | 0.61 | 0.73 |
| How often did information about the patient's care differ between the nursing home staff and the hospice team? | |||
| % agreement | 82.5 | 81.7 | 82.0 |
| Kappa | 0.58 | 0.52 | 0.58 |
| Average NH Kappa (excluding skewed marginals) | 0.55 | 0.54 | 0.61 |
| Number of NH Kappa ≤0.40 (excluding skewed marginals) | 0 | 1 | 0 |
| Table 2D. Inpatient Unit Module Items | |||
|---|---|---|---|
| Item | 3 to 6 months | 3 to 9 months | 6 to 9 months |
| Did the hospice staff speak with you about the decision to transfer the patient to a hospice inpatient unit? | |||
| % agreement | 92.7 | 94.9 | 92.5 |
| Kappa | 0.34 | 0.56 | 0.31 |
| p-positive | 0.96 | 0.96 | |
| p-negative | 0.38 | 0.35 | |
| How much help did you receive about the decision to transfer the patient to a hospice inpatient unit? | |||
| % agreement | 92.7 | 94.9 | 94.4 |
| Kappa | 0.54 | 0.69 | 0.57 |
| In the hospice inpatient unit, did any of the physicians, nurse practitioners or physician assistants speak with you? | |||
| % agreement | 94.5 | 92.0 | 94.7 |
| Kappa | 0.34 | 0.34 | 0.44 |
| p-positive | 0.97 | 0.96 | |
| p-negative | 0.36 | 0.38 | |
| How available to answer your questions was the inpatient team? | |||
| % agreement | 86.5 | 90.0 | 90.0 |
| Kappa | 0.36 | 0.52 | 0.51 |
| p-positive | 0.92 | ||
| p-negative | 0.44 | ||
| How much information did the inpatient team give you about the patient's care? | |||
| % agreement | 92.0 | 92.8 | 94.5 |
| Kappa | 0.38 | 0.26 | 0.41 |
| p-positive | 0.96 | 0.96 | |
| p-negative | 0.42 | 0.29 | |
| How often was information presented in a way you could understand? | |||
| % agreement | 84.6 | 88.3 | 88.2 |
| Kappa | 0.44 | 0.55 | 0.47 |
| How often did the inpatient team update you on the patient's condition? | |||
| % agreement | 93.4 | 90.5 | 95.8 |
| Kappa | 0.54 | 0.43 | 0.68 |
| In the inpatient unit, how often did the patient get help as soon as he/she requested it? | |||
| % agreement | 81.1 | 86.3 | 85.6 |
| Kappa | 0.48 | 0.58 | 0.56 |
| How often did the inpatient unit staff make you feel in the way? | |||
| % agreement | 91.2 | 86.2 | 84.3 |
| Kappa | 0.63 | 0.41 | 0.35 |
| p-positive | 0.91 | ||
| p-negative | 0.44 | ||
| Was the patient's room comfortable for you and your family to visit? | |||
| % agreement | 97.8 | 97.1 | 98.6 |
| Kappa | 0.56 | 0.49 | 0.74 |
| Average IPU Kappa (excluding skewed marginals) | 0.53 | 0.53 | 0.55 |
| Number of IPU Kappa ≤0.40 (excluding skewed marginals) | 0 | 0 | 0 |
| Average Kappa – all items | 0.57 | 0.56 | 0.59 |
As expected, our measure of grief symptom intensity showed a decrease in grief over time in our sample; average grief at three months was 11.5 (out of 45), at six months 10.6, and at nine months 9.4. In Table 3, we show the kappa statistics for core questions between three and six months for individuals with increasing grief (defined as those whose six-month grief score was more than one standard deviation above the mean change in grief between three and six months), stable grief (defined as those whose six-month grief score was less than one standard deviation above or below the mean change in grief between three and six months), and decreasing grief (defined as those whose six-month grief score was more than one standard deviation below the mean change in grief between three and six months). P-positive and p-negative statistics are again included when kappas are low because of skewed marginal effects. For example, the item “While under the care of hospice, did the patient have pain or take medicine for pain?” had a kappa statistic of 0.21 with a p-positive value of 0.92, a p-negative value of 0.29, and 34 total responses among individuals with improved grief between three and six months (Table 3). The high p-positive value and low p-negative value demonstrate that there is high agreement among the 31 affirmative responses and low agreement among 3 negative responses; thus, despite the low kappa, most responses to this item agreed across the two time points. Most items demonstrated adequate kappa values or high p-positive values with skewed marginal results, indicating stability. Exceptions included items on whether the patient experienced anxiety or sadness, whether the hospice prepared the family for what to expect at the time of death, and ratings of hospice team response on nights and weekends among those with improvement in grief. Among those with worsening grief, there was low agreement on items about whether the patient experienced anxiety or sadness, and whether the family received enough information about pain medicines. It should be noted that there appear to be more deflated kappas among those with increasing grief and decreasing grief because these populations are smaller than the population with stable grief, leaving more opportunity for skewed marginal effects.
Table 3.
Percent Agreement on Core Survey Items as Grief Changed or Remained Constant Between Three and Six Months
| Item | Grief improveda (n=39) | Grief remained constant (n=202) | Grief worseneda (n=35) |
|---|---|---|---|
| While under the care of hospice, did the patient have pain or take medicine for pain? | |||
| Kappa | 0.21 | 0.71 | 1.00 |
| p-positive | 0.92 | ||
| p-negative | 0.29 | ||
| How much help in dealing with his/her pain did the patient receive? | |||
| Kappa | 0.34 | 0.68 | −0.05 |
| p-positive | 0.94 | 0.94 | |
| p-negative | 0.40 | 0.00 | |
| Did the patient have trouble breathing or receive treatment for trouble breathing? | |||
| Kappa | 0.60 | 0.69 | 0.79 |
| How much help in dealing with breathing did the patient receive? | |||
| Kappa | n/ab | 0.69 | 0.00 |
| p-positive | 1.00 | 0.95 | |
| p-negative | n/a | 0.00 | |
| Did the patient have any feelings of anxiety or sadness? | |||
| Kappa | 0.39 | 0.70 | 0.28 |
| p-positive | 0.77 | 0.80 | |
| p-negative | 0.62 | 0.46 | |
| How much help in dealing with anxiety/sadness did the patient receive? | |||
| Kappa | 0.62 | 0.74 | 0.63 |
| How much information did you receive about pain medicines? | |||
| Kappa | 0.63 | 0.60 | 0.18 |
| p-positive | 0.89 | ||
| p-negative | 0.29 | ||
| How much information did you receive about pain medicine side effects? | |||
| Kappa | 0.72 | 0.52 | 0.43 |
| How much information did you receive about trouble breathing? | |||
| Kappa | −0.06 | 0.53 | 0.11 |
| p-positive | 0.94 | 0.77 | |
| p-negative | 0.00 | 0.67 | |
| How often did the hospice team keep you informed about the patients condition? | |||
| Kappa | 0.46 | 0.71 | 0.71 |
| How much information did you receive about what to expect when the patient was dying? | |||
| Kappa | 0.38 | 0.56 | 0.42 |
| p-positive | 0.88 | ||
| p-negative | 0.50 | ||
| Did you receive enough contact from hospice about your religious and spiritual beliefs? | |||
| Kappa | 0.61 | 0.51 | −0.03 |
| p-positive | 0.97 | ||
| p-negative | 0.00 | ||
| How much emotional support did you get from hospice prior to the patient's death? | |||
| Kappa | 0.35 | 0.53 | 0.35 |
| p-positive | 0.95 | 0.95 | |
| p-negative | 0.40 | 0.40 | |
| How much emotional support did you get from hospice after the patient's death? | |||
| Kappa | 0.62 | 0.24 | 0.51 |
| p-positive | 0.94 | ||
| p-negative | 0.30 | ||
| How often did someone from hospice give confusing or contradictory information? | |||
| Kappa | 0.72 | 0.57 | 0.71 |
| Was there always one hospice nurse in charge of the patient's care? | |||
| Kappa | 0.83 | 0.62 | 0.67 |
| Were there problems with the hospice team not knowing enough about the patient's medical history? | |||
| Kappa | 0.59 | ||
| p-positive | −0.04 | 0.27 | |
| p-negative | 0.95 | 0.92 | |
| 0.00 | 0.33 | ||
| How often did hospice answer your questions about the patient's care? | |||
| Kappa | 0.44 | 0.59 | 0.71 |
| How often was the hospice team available to speak with you about concerns? | |||
| Kappa | 0.79 | 0.44 | 0.76 |
| Did you have any concerns with the quality of the patient's care? | |||
| Kappa | 0.21 | 0.66 | 1.00 |
| p-positive | 0.92 | ||
| p-negative | 0.29 | ||
| Overall how would you rate the quality the patient received while under the care of hospice? | |||
| Kappa | 0.67 | 0.69 | 0.66 |
| How would you rate the way the hospice team responded in evenings and weekends? | |||
| Kappa | 0.36 | 0.51 | 0.60 |
| p-positive | 0.79 | ||
| p-negative | 0.57 | ||
| Based on the care the patient received, would you recommend this hospice to others? | |||
| Kappa | 0.53 | 0.58 | 0.76 |
| Average core Kappa (excluding skewed marginals) | 0.59 | 0.61 | 0.63 |
| Number of core Kappa ≤0.40 (excluding skewed marginals) | 3 | 0 | 2 |
“Grief improved” is defined as being one standard deviation above the mean change in grief. “Grief worsened” is defined as being one standard deviation below the mean change in grief.
All respondents gave the same response; therefore, we were unable to calculate a kappa.
Discussion
The CMS is required to publicly report hospice quality data, and has elected to use a bereaved family member survey as part of its Hospice Quality Reporting Program. However, the timing of administration of such surveys has been guided largely by tradition and not empirical research. A key question is how bereavement impacts a family member's ratings and perceptions of the quality of care, and whether timing of survey administration will affect publicly reported hospice quality data. We found that between three and nine months after the death of the hospice patient, bereaved family member responses were stable. Because the death of a family member may result in relocation of the spouse or other family member, we recommend that the survey be administered around three months after the death of the patient. A three-month administration will ensure that results are not biased against individuals who may be more likely to relocate after the death of a family member because of physical disability or socioeconomic disadvantage.
One of the criticisms sometimes made of after-death surveys is that bereaved people are likely to be experiencing intense emotions and, therefore, are unlikely to be reliable sources of information on events before the death (19). One small study by Cartwright et al. (11) examined the stability of 13 bereaved family members’ reports of the frequency and severity of the patients’ pain, anxiety, and depression at three and nine months after the patients’ death. They found poor agreement for the frequency and severity of pain, but moderate agreement for anxiety. In contrast, all of our pain items demonstrated moderate to high stability, as did our items that reported on the amount of emotional support given to the patient. In addition, Cartwright et al. did not use a response task comparable to the FEHC survey; their items focused on the severity of various symptoms, whereas FEHC items are intended to evaluate whether adequate care was received for symptoms.
A recent study by Casarett (20) used a bereaved family survey to evaluate end-of-life care received at Veterans Affairs facilities. No association was found between the timing of the interview (range: 6-10 weeks) and ratings of care that the veterans received. Our research examines the time period 12-36 weeks after the patient's death and suggests that bereaved surveys of end-of-life care are reliable over this extended time period as well.
In examining the reliability and validity of the Toolkit After-Death Bereaved Family Member Interview, Teno et al. (21) interviewed family members about end-of-life care under outpatient hospice services, nursing homes, and hospitals three months after death. Twenty-nine respondents were re-interviewed after their initial survey administration four to eight weeks prior. They found high test-retest reliability on items concerning decision making, advance care planning, coordination, respect and emotional support. Our study confirms these findings, while also establishing reliability in the inpatient hospice setting and reliability beyond five months after death.
Addington-Hall and McPherson (19) describe the cognitive psychology literature on the effect of bereavement and emotion on memory. Because individuals experience intense emotions during bereavement that usually resolve with time, there is concern that perceptions of care quality may become more positive as time since patient death increases. As a result, if some hospices were to collect bereaved family surveys soon after death, they may be unfairly characterized as providing lower quality care because newly bereaved respondents may report more problems. Accountability in the public reporting of hospice quality could be compromised if responses to quality of care surveys are time-dependent and there is wide variation in the timing of data collection after death. Our study shows that responses are stable and do not change appreciably over time; therefore, hospices can collect data from families of the deceased between three and nine months after the death of the patient. However, because the death of a family member may result in relocation of the spouse or other family member, particularly a disadvantaged one, we recommend that the survey be administered around three months after the death of the patient.
Our analysis has several limitations that should be noted. First, despite our efforts to oversample Hispanic respondents, our sample size for this ethnic group was small. Second, we did not test for stability prior to three months after the loss of a loved one. It is possible that more recent loss would lead to less response stability as individuals struggle to adjust to life without the patient. Third, we only sampled from six hospice programs, all of which were not-for-profit. For-profit hospices may have different stability patterns. Fourth, although this is the largest study on stability of bereaved family responses to date, our response rate was only 49.4%, which raises concern for non-response bias based on decedent age, race and ethnicity. Moreover, our analyses are based on data from research subjects who agreed to participate. There may have been less variation over time in this engaged population compared to the population that did not agree to participate in our study. Finally, our analysis focused on the level of distress from grief. We did not conduct a structured clinical interview to diagnose complicated grief disorder.
Conclusion
The stability of responses is important to the public reporting of hospice quality. This study shows that bereavement between three and nine months after the loss of a loved one did not impact the stability of perceptions of quality of care. This is illustrated through the kappa statistics that demonstrate moderate stability of perceptions of care quality. There was still high percent agreement and kappa statistics between three and six months for our items even among respondents with changes in grief over this time period. However, it should be noted that our analysis was limited by a small sample of individuals with meaningful fluctuations in grief. Further research should aim to characterize a larger sample of individuals with unstable grief and their perceptions of care quality.
Acknowledgments
The project, “Family Evaluation of Hospice Care” was funded by grant no. 1R01HS019675 from the Agency for Healthcare Research and Quality to principal investigator Joan M. Teno, MD, MS.
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Disclosures
The authors declare no other conflicts of interest.
References
- 1.National Hospice and Palliative Care Organization [July 9, 2014];NHCPO's facts and figures: Hospice care in America. 2013 Available from: http://www.nhpco.org/sites/default/files/public/Statistics_Research/2013_Facts_Figures.pdf.
- 2.Cartwright A, Seale C. The natural history of a survey: An account of the methodological issues encountered in a study of life before death. King Edward's Hospital Fund; London: 1990. [Google Scholar]
- 3.Addington-Hall J, McCarthy M. Regional study of care for the dying: methods and sample characteristics. Palliat Med. 1995;9:27–35. doi: 10.1177/026921639500900105. [DOI] [PubMed] [Google Scholar]
- 4.Centers for Disease Control and Prevention [July 9, 2014];National mortality follow back survey. Available from: http://www.cdc.gov/nchs/nvss/nmfs.htm.
- 5.Teno JM, Clarridge BR, Casey V, et al. Family perspectives on end-of-life care at the last place of care. JAMA. 2004;291:88–93. doi: 10.1001/jama.291.1.88. [DOI] [PubMed] [Google Scholar]
- 6.The Patient Protection and Affordable Care Act (PPACA). Statute. :1192010. Public Law 111-148. [Google Scholar]
- 7.Higginson IJ, Costantini M. Accuracy of prognosis estimates by four palliative care teams: a prospective cohort study. BMC Palliat Care. 2002;1:1. doi: 10.1186/1472-684X-1-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Wenger NS, Oye RK, Bellamy PE, et al. Prior capacity of patients lacking decision making ability early in hospitalization: implications for advance directive administration. The SUPPORT Investigators. Study to Understand Prognoses and Preferences for Outcomes and Risks of Treatments. J Gen Intern Med. 1994;9:539–543. doi: 10.1007/BF02599276. [DOI] [PubMed] [Google Scholar]
- 9.Casarett DJ, Crowley R, Hirschman KB. Surveys to assess satisfaction with end-of-life care: does timing matter? J Pain Symptom Manage. 2003;25:128–132. doi: 10.1016/s0885-3924(02)00636-x. [DOI] [PubMed] [Google Scholar]
- 10.Higginson I, Priest P, McCarthy M. Are bereaved family members a valid proxy for a patient's assessment of dying? Soc Sci Med. 1994;38:553–557. doi: 10.1016/0277-9536(94)90251-8. [DOI] [PubMed] [Google Scholar]
- 11.Cartwright A, Hockey L, Anderson JL. Life before death. Routeledge and Kegan Paul; London: 1973. [Google Scholar]
- 12.Teno JM, Byock I, Field MJ. Research agenda for developing measures to examine quality of care and quality of life of patients diagnosed with life-limiting illness. J Pain Symptom Manage. 1999;17:75–82. doi: 10.1016/s0885-3924(98)00134-1. [DOI] [PubMed] [Google Scholar]
- 13.Byock IR, Teno JM, Field MJ. Measuring quality of care at life's end. J Pain Symptom Manage. 1999;17:73–74. doi: 10.1016/s0885-3924(98)00135-3. [DOI] [PubMed] [Google Scholar]
- 14.Prigerson HG, Maciejewski PK, Reynolds CF, et al. Inventory of Complicated Grief: a scale to measure maladaptive symptoms of loss. Psychiatry Res. 1995;59:65–79. doi: 10.1016/0165-1781(95)02757-2. [DOI] [PubMed] [Google Scholar]
- 15.Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–174. [PubMed] [Google Scholar]
- 16.Cicchetti DV, Feinstein AR. High agreement but low kappa: II. Resolving the paradoxes. J Clin Epidemiol. 1990;43:551–558. doi: 10.1016/0895-4356(90)90159-m. [DOI] [PubMed] [Google Scholar]
- 17.Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol. 1990;43:543–549. doi: 10.1016/0895-4356(90)90158-l. [DOI] [PubMed] [Google Scholar]
- 18.Stata Statistical Software . Stata Release 13 [computer program] StataCorp. LP; College Station, TX: 2013. [Google Scholar]
- 19.Addington-Hall J, McPherson C. After-death interviews with surrogates/bereaved family members. some issues of validity. J Pain Symptom Manage. 2001;22:784–790. doi: 10.1016/s0885-3924(01)00330-x. [DOI] [PubMed] [Google Scholar]
- 20.Casarett D. Bereaved Family Survey. PROMISE Center, Center for Health Equity Research and Promotion; Philadelphia, PA: 2012. [July 24, 2014]. Available from: http://www.qualitymeasures.ahrq.gov/content.aspx?id=27293. [Google Scholar]
- 21.Teno JM, Clarridge B, Casey V, Edgman-Levitan S, Fowler J. Validation of Toolkit After-Death Bereaved Family Member Interview. J Pain Symptom Manage. 2001;22:752–758. doi: 10.1016/s0885-3924(01)00331-1. [DOI] [PubMed] [Google Scholar]