Validation of New Quality Measures for Transitions Between Sites of Care

Abstract

BACKGROUND AND OBJECTIVE:

Assessing and improving the quality of transitions to home from the emergency department (ED) or hospital is critical for patient safety. Our objective was to validate 8 newly developed caregiver-reported measures of transition quality.

METHODS:

This prospective observational study included 1086 caregiver survey respondents whose children had an ED visit (n = 523) or hospitalization (n = 563) at Seattle Children’s Hospital in 2014. Caregivers were contacted to complete 2 surveys. The first survey included the newly developed transition quality measures and multiple validation measures including modified versions of Child Hospital Consumer Assessments of Healthcare Providers and Systems (HCAHPS) composites, assessing communication and discharge planning. The second survey (administered 30 days later) included questions about follow-up appointments and ED return visits and readmissions. Using multivariate regression, we examined associations between the newly developed transition quality measures and each validation measure.

RESULTS:

All transition quality measures were significantly associated with ≥1 validation measures. The hospital-to-home transition measure assessing whether discharge instructions were easy to understand, were useful, and contained necessary follow-up information had the largest association with the Child HCAHPS nurse–parent and doctor–parent communication composites (β = 55.6; 95% confidence interval, 43 to 68.3; and β = 48.3; 95% confidence interval, 36.3 to 60.3, respectively, scaled to reflect change associated with a 0 to 100 change in the transition measure score).

CONCLUSIONS:

Newly developed quality measures for pediatric ED- and hospital-to-home transitions were significantly and positively associated with previously validated measures of caregiver experience. These new measures may be useful for assessing and improving on the quality of ED- and hospital-to-home transitions.

Transitions to home from the emergency department (ED) or hospital may pose safety risks for pediatric patients and their families. Among adults, adverse events after ED- or hospital-to-home transitions are often related to insufficient discharge preparation and communication failures between inpatient and outpatient providers.^1–4 Within pediatrics, a study of primary care providers demonstrated a need to improve the quality of hospital discharge summaries,⁵ and a study of caregivers revealed that 20% experienced postdischarge problems, half of which were medication related.⁶ Pediatric research also has demonstrated that adequate caregiver preparation before discharge is associated with reduced readmission rates.⁷ Despite this demonstrated need to improve the quality of transitional care, few valid quality measures focus on transitions.

With funding from the Agency for Healthcare Research and Quality and the Centers for Medicare and Medicaid Services, the Center of Excellence on Quality of Care Measures for Children With Complex Needs (COE4CCN) developed measures to assess the quality of ED- and hospital-to-home transitions.^8,9 The COE4CCN developed transition measures that were best assessed by caregiver report (because some aspects of care are best measured by those who experience it) and other measures that were best assessed through medical records data. In this article, we report on the validation of the caregiver-reported measures.

This study sought to assess the convergent and predictive validity of 8 newly developed caregiver-reported transition care quality measures: 2 focused on ED-to-home transitions and 6 focused on hospital-to-home transitions.

METHODS

Development and Operationalization of Transition Quality Measures

Details about the development of the ED- and hospital-to-home transition quality measures, survey instruments, and scoring are described elsewhere.^8,10 Briefly, we identified topics for scientific literature review based on a conceptual framework, drafted quality measures based on available evidence, and convened a multistakeholder Delphi panel to prioritize draft measures. The Delphi panel endorsed 2 caregiver-reported ED-to-home transition measures and 6 caregiver-reported hospital-to-home transition measures (Table 1). Two separate surveys were then developed in parallel to assess the endorsed measures.⁸ Survey items specified the eligible population for each measure (denominator) and whether the indicated process of care was received among those eligible (numerator). Cognitive interviews were conducted in English and Spanish to improve the understandability and consistency of the interpretation of survey items in these languages. Transition quality measures were scored either on a 0 to 100 scale such that higher scores indicated better care, or as a percentage of respondents who reported that they received the indicated care.

TABLE 1.

Pediatric Transition Quality Measures Developed by the COE4CCN

ED-to-home transitions
Short measure description	Measure full text
Understood how to use medical equipment at home and who to call with questions	Caregivers of children discharged from the ED during the last month and who were discharged with a prescription for new medical equipment, should report that they:  a. Understood how to use the medical equipment at home, and  b. Knew who to call with questions
Received discharge education in preferred language	Caregivers of children seen in the ED during the last month whose preferred language is other than English should report that they received discharge education in their preferred language
Hospital-to-home transitions
Had the opportunity to ask questions before discharge	Caregivers of children discharged from the hospital during the last month should report that, before their child’s most recent discharge, they had an opportunity to ask questions related to:  a. Instructions for care after discharge, and  b. Follow-up instructions
Understood how to use medical equipment at home and who to call with questions	Caregivers of children discharged from the hospital during the last month and who were discharged with a prescription for new medical equipment should report that they:  a. Understood how to use the medical equipment at home, and  b. Knew who to call with questions
Received discharge education in preferred language	Caregivers of children discharged from the hospital during the last month whose preferred language is other than English should report that they received discharge education in their preferred language
Received assistance with scheduling follow-up appointments	Caregivers of children discharged from the hospital during the last month should report being offered assistance in scheduling all needed follow–up appointments within 72 h of discharge from the hospital
Received written discharge instructions	Caregivers of children discharged from the hospital during the last month should report that before discharge, they received a copy of or access to (e-mail, patient portal app) discharge instructions
Written discharge instructions were easy to understand, were useful, and contained necessary follow-up and contingency plan information	Caregivers of children discharged from the hospital during the last month who report receiving discharge instructions should also report that they had all of the following characteristics:  • Easy to understand  • Useful for caring for their child at home  • Contained all the necessary information for follow–up  • Contained information about who to call if problems arose

Survey Design, Sample, and Administration

The 2 surveys implemented for this study included the transition quality measure survey items, items to assess the validation measures, and items to collect child and caregiver characteristics. The ED-to-home transition survey included 29 items, and the hospital-to-home transition survey included 41 items (see Supplemental Materials).

Eligible survey respondents were defined as English- or Spanish-speaking caregivers (≥18 years old) of patients aged 2 months to 17 years who had an ED visit or hospitalization for any reason at Seattle Children’s Hospital between June 30, 2014 and September 1, 2014. All eligible caregivers of patients with a hospitalization during the study period were invited to complete the hospital-to-home transition survey. An equal number of caregivers of patients who had an ED visit during the study time frame were randomly sampled and invited to complete the ED-to-home transition survey. Eligible caregivers were randomly assigned so that 50% received a mailed survey and 50% were called to complete a telephone interview. All caregivers were contacted within 42 days of their child’s discharge from the ED or inpatient settings and had 42 days to complete the survey. All survey respondents were also invited to complete a second survey 30 days after their response to the initial survey, which included additional validation measures (see online Supplemental Materials).

Validation Measures: initial Survey

The initial survey included 5 validation measures to assess the convergent validity of the quality measures. Two were communication composites (Nurse–Parent and Doctor–Parent Communication) modified from the Child Hospital Consumer Assessment of Healthcare Providers and Systems (Child HCAHPS) Survey, which included questions about listening carefully to the caregiver (ED survey only), communicating in a way that was easy for the caregiver to understand, and treating the caregiver with courtesy and respect.¹¹ Responses were scored on a 0 to 100 scale for each question within the composite (Never = 0, Sometimes = 0, Usually = 0, Always = 100, consistent with the top-box scoring method used in Child HCAHPS), where higher scores reflected better communication. All caregivers who answered ≥1 of the questions within the composite received a composite score calculated as the mean of the nonmissing responses^12,13 (missingness by question ranged from 0% to 1.5%). These 2 communication composites were used as validation measures for both ED- and hospital-to-home transition measures.

A third validation measure (inpatient survey only) was a modified version of the Child HCAHPS Preparing You and Your Child to Leave the Hospital discharge planning composite, which included items about assessing caregiver readiness for discharge, explaining postdischarge care and expectations, and symptoms to watch for after discharge. This composite was scored similarly to the communication composites.

The remaining 2 validation measures (inpatient survey only) assessed parent perceptions of discharge and were associated with decreased readmissions in a previous study.⁷ The measures were caregiver perceptions of their child’s readiness for discharge and caregiver understanding of how to manage their child’s care after discharge, and they were scored dichotomously (consistent with previous studies using these measures⁷) as: strongly agree = 1, all other responses = 0.

Validation Measures: Second Survey

The second survey included 3 additional validation measures to assess predictive validity: whether all needed follow-up visits were scheduled or completed, whether the child had a hospital readmission since discharge, and whether the child had a return ED visit since discharge.

Analyses

Using linear and logistic regression models, we assessed the convergent validity of the newly developed transition quality measures by examining the association between the transition quality measures and the modified Child HCAHPS and parent perceptions of discharge measures. We also used logistic regression models to assess predictive validity by examining the association between the quality measures and successful scheduling or completion of needed follow-up appointments, ED return visits, or hospital readmissions. Only a priori hypothesized associations between the transition and validation measures were tested (Table 2). Each quality measure was evaluated separately, and associations were summarized via adjusted regression coefficients and odds ratios.

TABLE 2.

A Priori Hypothesized Associations

	Validation Measures
ED-to-Home Transition Measures	Nurse–Parent Communication Composite	Doctor–Parent Communication Composite	Discharge Planning Composite	Parent Felt Child Was Healthy Enough to Leave Hospital	Parent Understood How to Manage Child’s Health After Going Home	Needed Follow-Up Appointments Scheduled or Completeda	Inpatient Admissiona	ED Visita
Understood how to use medical equipment at home and who to call with questions	↑	↑	NA	NA	NA	—	↓	↓
Received discharge education in preferred language	↑	↑	NA	NA	NA	↑	↓	↓
Hospital-to-home transition measures
Had the opportunity to ask questions before discharge	↑	↑	↑	↑	↑	—	↓	↓
Understood how to use medical equipment at home and who to call with questions	↑	↑	↑	—	↑	—	↓	↓
Received discharge education in preferred language	↑	↑	↑	—	↑	—	↓	↓
Received assistance with scheduling follow-up appointments	—	—	—	—	—	↑	↓	↓
Received written discharge instructions	—	—	—	—	↑	—	↓	↓
Written discharge instructions were easy to understand, were useful, and contained necessary follow-up and contingency plan information	↑	↑	—	—	↑	↑	↓	↓

^↑denotes a hypothesized positive association; ↓ denotes a hypothesized negative association; —denotes no hypothesized association; NA, validation measures not included on ED-to-home transition survey.

^aObtained from second survey.

We adjusted for child age, sex, medical complexity, caregiver education, and preferred language for medical communication. These characteristics were selected because they were associated with both the transition quality and validation measures. Medical complexity was assessed with the Pediatric Medical Complexity Algorithm (PMCA), which uses International Classification of Diseases, Ninth Revision, Clinical Modification codes to classify children into 3 levels: complex chronic disease, noncomplex chronic disease, and no chronic disease.¹⁴ This study was approved by the institutional review boards at RAND Corporation and Seattle Children’s Hospital.

RESULTS

Of the 1419 eligible caregivers invited, 523 completed the initial ED-to-home transition survey (37% response rate). Of the 1413 eligible caregivers invited, 563 completed the initial hospital-to-home transition survey (40% response rate). Characteristics of the survey respondents and their children are shown in Table 3. Among respondents to the initial surveys, 67% and 70% also responded to the second survey for ED-to-home and hospital-to-home transitions, respectively.

TABLE 3.

Transition Survey Respondent Characteristics

	All Respondents to Initial Survey
	ED-to-Home	Hospital-to-Home
	Respondents (N = 523), n (%)	Respondents (N = 563), n (%)
Child characteristics
Child age, y
0–1	117 (22)	123 (22)
2–5	162 (31)	129 (23)
6–12	171 (33)	160 (28)
13–18	73 (14)	151 (27)
Child female	236 (45)	264 (47)
Child race or ethnicity
White	286 (55)	323 (57)
Hispanic	126 (24)	116 (21)
African American	29 (5)	20 (4)
Asian or Pacific Islander	37 (7)	39 (7)
Other	45 (9)	65 (12)
Lives in a metropolitan statistical area	513 (98)	474 (84)
Public insurance	20 (4)	46 (8)
Medical complexity
No chronic conditions	406 (78)	147 (26)
Noncomplex chronic conditions	67 (13)	138 (25)
Complex chronic conditions	50 (10)	278 (49)
Caregiver characteristics
Caregiver age, y
<35	206 (40)	213 (38)
35–44	215 (41)	211 (38)
45+	98 (19)	130 (23)
Caregiver female	433 (83)	493 (88)
Caregiver race or ethnicity
White	300 (57)	357 (63)
Hispanic	123 (23)	102 (18)
African American	25 (5)	16 (3)
Asian or Pacific Islander	33 (6)	37 (7)
Other	42 (8)	51 (9)
Caregiver education (highest level completed)
Less than high school degree	58 (11)	46 (8)
High school graduate or GED	71 (14)	99 (18)
Some college	141 (27)	159 (29)
4-y college graduate	122 (24)	136 (24)
More than 4-y college degree	126 (24)	116 (21)
Caregiver not child’s parent	11 (2)	11 (2)
Language of survey completion
English	444 (85)	507 (90)
Spanish	79 (15)	56 (10)
Language preference
English	449 (87)	511 (91)
Other	70 (13)	50 (9)

ED-to-Home Transition Measures: Convergent Validity

Higher scores on the 2 ED-to-home transition measures were significantly associated with better Nurse–Parent Communication composite scores (Table 4). Adjusted regression β-coefficients in Table 4 were scaled to reflect the change in the validation metric associated with a 0 to 100 change in the transition measure score. For example, for the ED-to-home transition measure assessing whether the caregiver understood how to use medical equipment at home and whom to call with questions, a score of 100 versus a score of 0 on this measure was associated with a 33.5-point higher Nurse–Parent Communication composite score.

TABLE 4.

Validation of Transition Measures Based on Modified Versions of Child HCAHPS Communication and Discharge Planning Composites

	Nurse–Parent Communication Composite (0–100)		Doctor–Parent Communication Composite (0–100)		Discharge Planning Composite (0–100)
	N	βa (95% CI)	N	βa (95% CI)	N	βa (95% CI)
ED-to-home transition measures
Understood how to use medical equipment at home and who to call with questions	47	33.5 (8.1 to 58.9)*	47	−8.0 (−36.1 to 20.2)	NA	NA
Received discharge education in preferred language	65	22.1 (3.5 to 40.7)*	65	9.9 (−8.3 to 28.0)	NA	NA
Hospital-to-home transition measures
Had the opportunity to ask questions before discharge	553	47.0 (28.6 to 65.3)***	552	40.1 (22.9 to 57.2)***	553	67.0 (53.1 to 80.9)***
Understood how to use medical equipment at home and who to call with questions	105	23.5 (−4.2 to 51.2)	105	24.9 (3 to 46.8)*	105	23.7 (6.1 to 41.2)**
Received discharge education in preferred language	50	21.9 (−41 to 84.7)	50	39.0 (−8.7 to 86.6)	50	73.5 (56.8 to 90.3)***
Received assistance with scheduling follow-up appointments	—	—	—	—	—	—
Received written discharge instructions	—	—	—	—	—	—
Discharge instructions were easy to understand, were useful, and contained necessary follow-up and contingency plan information	533	55.6 (43 to 68.3)***	532	48.3 (36.3 to 60.3)***	—	—

NA, validation measures not included on ED-to-home transition survey; —, no hypothesized association.

^aAdjusted regression coefficients (β) were obtained via linear regression models adjusting for child age, child sex, child PMCA category, caregiver education, and caregiver preferred language for medical communication (except for “Received discharge education in preferred language” measures, which were not adjusted for caregiver preferred language); β has been scaled to reflect the change in the validation metric associated with a 0 to 100 change in the transition measure score.

^*P < .05.

^**P < .01.

^***P < .001.

ED-to-Home Transition Measures: Predictive Validity

Associations between ED-to-home transition measures and ED return visits and hospital admissions were not examined because of the very low occurrence of these events in the eligible study sample (2 for 1 measure, 5 for the second measure).

Hospital-to-Home Transition Measures: Convergent Validity

Higher scores on each of the hospital-to-home transition measures were significantly associated with ≥1 of the modified Child HCAHPS validation measures (Table 4). Adjusted odds ratios (aORs) in Table 5 were scaled to reflect the change in the validation metric associated with a 0 to 100 change in the transition measure score. Higher hospital-to-home transition quality measure scores were also significantly and positively associated with the parent perceptions of discharge survey items (Table 5).

TABLE 5.

Validation of Transition Measures Based on Parent Perceptions of Discharge Items and Scheduling or Completion of Needed Follow-Up Appointments

	Parent Felt Child Was Healthy Enough to Leave Hospital		Parent Understood How to Manage Child’s Health After Going Home		Needed Follow-Up Appointments Scheduled or Completeda
	N	aORb (95% CI)	N	aORb (95% CI)	N	aORb (95% CI)
ED-to-home transition measures
Understood how to use medical equipment at home and who to call with questions	NA	NA	NA	NA	—	—
Received discharge education in preferred language	NA	NA	NA	NA	NA	NA
Hospital-to-home transition measures
Had the opportunity to ask questions before discharge	552	5.2 (1.6 to 18.7)**	550	10.1 (2.8 to 42.3)***	—	—
Understood how to use medical equipment at home and who to call with questions	—	—	103	23.7 (3270.6)**	—	—
Received discharge education in preferred language	—	—	NA	NA	—	—
Received assistance with scheduling follow-up appointments	—	—	—	—	323	3.2 (1.1 to 8.8)*
Received written discharge instructions	—	—	550	3.4 (1.2 to 10.2)*	—	—
Discharge instructions were easy to understand, were useful, and contained necessary follow-up and contingency plan information	—	—	530	12.3 (4.8 to 33.1)***	321	5.2 (0.5 to 43.1)

NA indicates that either the validation measure was not included on ED-to-home transition survey or that there was insufficient sample size for the regression analysis; —, no hypothesized association.

^aObtained from second survey.

^baORs were obtained via logistic regression models adjusting for child age, child sex, child PMCA category, caregiver education, and caregiver preferred language for medical communication; aOR has been scaled to reflect the change in the validation metric associated with a 0–100 change in the transition measure score.

^*P < .05;

^**P < .01;

^***P < .001.

Hospital-to-Home Transition Measures: Predictive Validity

The hospital-to-home transition measure that assessed whether the caregiver received assistance with scheduling follow-up appointments was significantly associated with a higher odds of successful scheduling or completion of all needed follow-up appointments (aOR = 3.2; 95% confidence interval [CI], 1.1 to 8.8). Scores on the hospital-to-home transition measures were not associated with caregiver-reported ED return visits or readmissions (results not shown; overall, 14.3% [N = 56] of respondents to the hospital-to-home transition survey had an ED return visit and 12.7% [N = 50] had a hospital readmission).

DISCUSSION

In this study examining the convergent and predictive validity of new caregiver-reported quality measures for pediatric ED- and hospital-to-home transitions, we found that all measures were significantly associated with ≥1 of the validation measures assessed, including modified versions of previously validated communication and discharge planning composite scores, parent perceptions of discharge survey items, and scheduling or completion of needed follow-up visits. These results are consistent with our a priori hypotheses that higher-quality transitions between sites of care would be associated with higher scores on the caregiver-reported validation measures and a higher likelihood of scheduling or completing needed follow-up visits. No associations were found between the new hospital-to-home transition measures and ED return visits or readmissions.

The development and validation of these new transition quality measures provide an important resource for providers to identify processes of care where quality improvement (QI) efforts can be focused. Although previously developed quality measures, such as Child HCAHPS, provide meaningful assessments of caregiver experience and readiness to leave the hospital, they do not identify specific processes of care that could be improved. The convergent validity of our newly developed measures with these previously developed measures supports their scientific soundness as quality measures and indicates that the processes of care assessed by these measures can be targeted for QI efforts.

In terms of predictive validity, we found that caregivers who received assistance with scheduling follow-up appointments were more likely to adhere to the follow-up care plan. This finding also illustrates the utility of these measures regarding specific actions a hospital can take to improve the quality of care transitions. This result is consistent with findings in previous work that have shown significant associations between providing adult patients with appropriate assistance and information at discharge and higher adherence to follow-up appointments.^15–18

Unlike previous adult studies,^15,19–22 we did not find a significant association between the new measures and health care utilization. Such utilization outcomes are uncommon in pediatrics and may not be the best metrics for transition quality assessment, particularly in this population.^23–25 To better understand transition quality we need to consider aspects of transitional care that can be targeted for QI intervention. These 8 newly validated quality measures may help fill this gap. Although establishing the validity of these measures is important, testing is needed to understand their responsiveness to QI interventions. Responsiveness to QI efforts would strengthen the utility of these measures and potentially allow them to be used to measure performance improvement over time.

Our study has some limitations. First, the majority of the validation measures came from the initial surveys, and therefore associations cannot be interpreted as causal. However, 3 validation measures were obtained from a second survey, 1 month after the initial survey response. It is important to note that the initial survey informed the caregiver that he or she would receive a follow-up survey that would ask about completing follow-up visits. Thus, attendance at follow-up appointments may be higher in this study than in the general population. Second, we were unable to examine associations between ED-to-home transition measures and ED return visits or hospital readmissions because of the small number of patients having 1 of these events. Future studies should include adequate sample sizes to examine these key validation measures. In addition, because ED return visits and hospital readmissions were assessed via caregiver report, there was potential for recall bias. Third, although our response rate of 37% to 40% for the initial survey is comparable to that of similar surveys, such as Adult and Child HCAHPS (which range from 17% to 42%^26,27), the results may be subject to nonresponse bias. Lastly, our sample of caregivers may not be a representative sample because this study was conducted in a single academic institution. Future validation work across geographically and structurally diverse hospitals should be performed. Despite this latter limitation, given our results, it is reasonable to use these measures to guide QI efforts related to transitional care for children seen in the ED or hospital settings.

CONCLUSIONS

These results demonstrate the validity of 8 new caregiver-reported quality measures for pediatric ED- and hospital-to-home transitions. These measures can be used to examine whether specific evidence-based transition care processes were completed by ED or hospital providers from the perspective of caregivers.

WHAT’S KNOWN ON THIS SUBJECT:

Transitions to home from the emergency department or hospital may pose safety risks for pediatric patients and their families. There is a dearth of rigorously developed and validated quality measures to assess these care transitions.

WHAT THIS STUDY ADDS:

We demonstrate the validity of 8 newly developed caregiver-reported emergency department-to-home and hospital-to-home transition quality measures. These new measures may be useful for assessing and tracking the quality of pediatric transitions between sites of care.

ABBREVIATIONS

aOR

adjusted odds ratio

CI

confidence interval

COE4CCN

Center of Excellence on Quality of Care Measures for Children With Complex Needs

ED

emergency department

HCAHPS

Hospital Consumer Assessments of Healthcare Providers and Systems

PMCA

Pediatric Medical Complexity Algorithm

QI

quality improvement