Development and Pilot Testing of Caregiver-Reported Pediatric Quality Measures for Transitions Between Sites of Care

Abstract

BACKGROUND:

Few measures exist to assess pediatric transition quality between care settings. The study objective was to develop and pilot test caregiver-reported quality measures for pediatric hospital and emergency department (ED) to home transitions.

METHODS:

On the basis of an evidence review, we developed draft caregiver-reported quality measures for transitions between sites of care. Using the RAND-UCLA Modified Delphi method, a multistakeholder panel endorsed measures for further development. Measures were operationalized into 2 surveys, which were administered to caregivers of patients (n = 2839) discharged from Seattle Children’s Hospital between July 1 and September 1, 2014. Caregivers were randomized to mail or telephone survey mode. Measure scores were computed as a percentage of eligible caregivers who endorsed receiving the indicated care. Differences in scores were examined according to survey mode and caregiver characteristics.

RESULTS:

The Delphi panel endorsed 6 of 8 hospital to home transition measures and 2 of 3 ED to home transitions measures. Scores differed significantly according to mode for 1 measure. Caregivers with lower levels of educational attainment and/or Spanish-speaking caregivers reported significantly higher scores on 3 of the measures. The largest difference was reported for the measure that assessed whether caregivers received assistance with scheduling follow-up appointments; 92% score for caregivers with lower educational attainment versus 79% for caregivers with higher educational attainment (P < .001).

CONCLUSIONS:

We developed 8 new, evidence-based quality measures to assess transition quality from the perspective of caregivers. Pilot testing of these measures in a single institution yielded valuable insights for future testing and implementation of these measures.

IMPROVING THE QUALITY of transitions between sites of care is essential to address gaps in quality and patient safety in the United States health care system.^1,2 One in 5 adult patients experiences an adverse event during hospital to home transitions³ and 1 in 20 experiences an adverse event after emergency department (ED) to home transitions.⁴ Similarly, among a cohort of pediatric patients, 20% of their caregivers identified a problem during the transition home from the hospital that required active intervention.⁵ Many of these events are related to inadequate preparation of families for transitions^6-8 and communication failures between inpatient and outpatient providers.^9-12 Communication deficits during transitions are also a leading cause of sentinel events (eg, severe disability or death) within or between hospital or ambulatory care settings.¹³

These gaps in quality and patient safety led the Centers for Medicare and Medicaid Services and the Agency for Healthcare Research and Quality to target transitions between sites of care as a priority area for development of pediatric quality measures. The Center of Excellence on Quality of Care Measures for Children with Complex Needs (COE4CCN) was charged with creating these quality measures. The COE4CCN is 1 of 7 Centers of Excellence that constitute the national Pediatric Quality Measures Program mandated by the Children’s Health Insurance Program Reauthorization Act of 2009.¹⁴ The COE4CCN aimed to create measures that were evidence-based, and were ideally informed by high-quality pragmatic clinical trials in the pediatric and adult literature.

This article describes the development and pilot testing of 2 surveys that include items that assess 8 caregiver-reported pediatric quality measures focused on transitions between sites of care, including 6 hospital to home and 2 ED to home transition measures. We report on the measure and survey development process, as well as differences in measure scores according to survey mode and respondent demographic characteristics.

Methods

Evidence Review and Measure Development

The COE4CCN transition survey measures were developed using a multistep approach led by the Care Transitions Working Group, a multistakeholder group of researchers, providers, patient advocates, payers, and state agency representatives with expertise in care transitions. A comprehensive description of the process used by the COE4CCN to develop these quality measures is detailed elsewhere.¹⁵ In brief, the working group first created a conceptual framework for pediatric transitions between sites of care. This informed a targeted literature review of studies that evaluated the effect of transition interventions on patient health outcomes and healthcare utilization outcomes.¹⁶ A major goal of the work was to develop measures related to care processes that engage the family because they often assume primary responsibility for the child’s care during transitions between care settings. Therefore, we included studies in the review if the intervention involved a component of family engagement (eg, processes related to discharge education were included whereas information transfer between providers was not). To be comprehensive, the literature review also included studies in adult and elderly populations, as well as expert consensus guidelines pertaining to pediatric transitions between sites of care. Next, the working group drafted a set of quality measures for transitions between sites of care on the basis of the supporting evidence identified from the review process (Table 1). Because of the content of the quality measures, the working group determined the optimal source of data collection for these quality measures was by caregiver report (ie, reported by a parent or legal guardian).

Table 1.

Caregiver-Reported Pediatric Quality Measures for Transitions Between Sites of Care Developed by the COE4CCN

Short Measure Description	Measure Full Text	Median Validity Rating*,†	Median Feasibility Rating*,‡	Endorsed/Not Endorsed
Hospital to home transitions
Had the opportunity to ask questions before discharge	Caregivers of children discharged from the hospital during the past month should report that, before their child’s most recent discharge; they had an opportunity to ask questions related to:			Endorsed
	a) Instructions for care after discharge and	a) 7	a) 8
	b) Follow-up instructions	b) 9	b) 8
Understood how to use medical equipment at home and who to call with questions	Caregivers of children discharged from the hospital during the past month and who were discharged with a prescription for new medical equipment, should report that they:			Endorsed
	a) Understood how to use the medical equipment at home and	a) 8	a) 7
	b) Knew who to call with questions	b) 8	b) 7	Endorsed
Received discharge education in preferred language	Caregivers of children discharged from the hospital during the past month whose preferred language is other than English should report that they received discharge education in their preferred language	9	8	Endorsed
Received assistance with scheduling follow-up appointments	Caregivers of children discharged from the hospital during the past month should report being offered assistance in scheduling all needed follow-up appointments within 72 hours of discharge from the hospital	8	7	Endorsed
Received written discharge instructions	Caregivers of children discharged from the hospital during the past month should report that before discharge, they received a copy of or access to (e-mail, patient portal, app) discharge instructions	8	7	Endorsed
Written discharge instructions were easy to understand, useful, and contained necessary follow-up and contingency plan information	Caregivers of children discharged from the hospital during the past month who report receiving discharge instructions should also report that they had all of the following characteristics:			Endorsed
	a) Easy to understand	a) 8	a) 7
	b) Useful for caring for their child at home	b) 7	b) 6
	c) Contained all the necessary information for follow-up and	c) 7	c) 6
	d) Contained information regarding who to call if problems arose	d) 8	d) 7
Emergency care plan reviewed and updated	Caregivers of children with chronic conditions discharged from the hospital during the past month who report that their child had an Emergency Care Plan before admission should report that the plan was reviewed and updated before discharge	5	7	Not Endorsed
Emergency care plan developed and copy received	Caregivers of children with chronic conditions who report that their child did not have an Emergency Care Plan before a hospital admission should also report that the Emergency Care Plan was developed during the hospitalization and that they received a copy of the plan before hospital discharge	5	7	Not Endorsed
Emergency Department to home transitions
Understood how to use medical equipment at home and who to call with questions	Caregivers of children discharged from the Emergency Department during the past month and who were discharged with a prescription for new medical equipment, should report that they:			Endorsed
	a) Understood how to use the medical equipment at home and	a) 7	a) 7
	b) Knew who to call with questions	b) 7	b) 8
Received discharge education in preferred language	Caregivers of children seen in the Emergency Department during the past month whose preferred language is other than English should report that they received discharge education in their preferred language	9	7	Endorsed
Received assistance with scheduling follow-up appointments	Caregivers of children seen in the Emergency Department during the past month should report receiving assistance in scheduling all needed follow-up visits before discharge from the Emergency Department	5	5	Not Endorsed

COE4CCN indicates Center of Excellence on Quality of Care Measures for Children with Complex Needs.

^*Each sub-component for multi-component measures was scored individually by Delphi panel members.

^†A mean validity rating of 1–3 indicates the proposed quality measure is not valid; 4–6 indicates the validity of the proposed quality measure is equivocal; and 7–9 indicates the proposed quality measure is deemed to be valid.

^‡A mean feasibility rating of 1–3 indicates the measure is not considered feasible to implement; 4–6 indicates that feasibility of implementation is questionable; and 7–9 indicate that the measure as specified is feasible to implement.

We then used the RAND-UCLA Modified Delphi Method¹⁷ to establish the face validity of these quality measures. This method has been used in numerous studies to assess the face validity of quality measures for pediatric and adult care.^18,19 Briefly, a multistakeholder panel of 9 members evaluated and rated each of these measures on validity and feasibility criteria using a 0 to 9 scale. Panel members were instructed to apply the following criteria for a measure to be considered valid: 1) there should be adequate evidence or expert consensus to support the measure, 2) there should be identifiable health benefits associated with receiving the measure-specified care, 3) that providers and provider groups who adhere more consistently to the measure would be regarded as providing higher-quality care, and 4) that adherence to the measure is in the control of providers or the health care system. Panel members were instructed to apply the following criteria for a measure to be considered feasible: 1) the measure included content a typical respondent would be able to correctly identify, recall, and report within the specified time frame, and 2) quality assessments on the basis of the measure would be reliable and unbiased.

Panel members then participated in a 2-day in-person meeting in Seattle, Washington to discuss controversial quality measures. A measure was considered to be controversial if the median validity score was 4 to 6, if the median feasibility score was <4, or if the scores indicated an indeterminate level of agreement or disagreement among panelists. To determine level of agreement, we tested the hypothesis that 80% of the hypothetical scores would be within the same score domain (1–3, 4–6, or 7–9) as the observed median score. On the basis of a group of 9 scores, agreement required that no more than 2 scores were outside of the 3-point domain that contained the observed median. To determine disagreement, we tested the hypothesis that 90% of hypothetical scores were within 1 of 2 larger, over-lapping score domains (1–6 or 4–9). On the basis of a group of 9 scores, the definition of disagreement was met when 3 or more ratings were in the 1 to 3 range and the 7 to 9 range. If the scores could not be classified as either with agreement or with disagreement on the basis of the definitions given, they were considered to be “indeterminate.”¹⁷

After discussion of all controversial measures, panelists independently rescored all measures. After the second round of scores was tabulated, median rating and level of agreement were ascertained for each measure. All measures with a median validity score of ≥7, a median feasibility score of ≥4, and scored without disagreement were considered to have been endorsed by the panel to move forward in the measure development process. The Delphi criterion is more liberal for feasibility than for validity because all measures go through feasibility testing after the Delphi panel, which helped to confirm that measures scored in the 4 to 6 range are truly feasible to implement as specified.

Survey Development

These caregiver-reported quality measures were operationalized through the development of 2 surveys: a hospital to home transition survey (Supplementary Appendix A) and an ED to home transition survey (Supplementary Appendix B). Survey items were developed to specify: 1) the eligible population for each measure (the measure denominator), and 2) whether the indicated process of care was received among those eligible (the measure numerator). We also included additional survey items to assess respondent demographic characteristics such as caregiver age, gender, race/ethnicity, level of education, relationship to the patient, and preferred language for medical communication. Cognitive interviews were conducted with 6 English-speaking caregivers and 5 Spanish-speaking caregivers to improve the understandability and consistency of the interpretation of survey items.

Survey Pilot Testing

Both surveys were then pilot tested to examine differences in measure scores according to survey mode and respondent demographic characteristics to inform future testing and implementation of the measures. The sample consisted of caregivers (≥18 years old) of patients (2 months-17 years old) who had an inpatient stay or ED visit for any reason at Seattle Children’s Hospital between July 1 and September 1, 2014. Only caregivers fluent in either English or Spanish were eligible for the study. Eligibility was determined using hospital administrative data. All eligible caregivers of patients with an inpatient stay during the study period were invited to complete the hospital to home transition survey. An equal number of caregivers of patients who had an ED visit during the study time frame were randomly sampled and invited to complete the ED to home transition survey. This strategy was used to obtain similar sample sizes for both surveys because there was a higher volume of ED visits compared with inpatient admissions.

Caregivers were randomized to 2 modes of survey administration: 50% were invited to complete a telephone survey and 50% were invited to complete a mailed survey. Caregivers randomized to the telephone survey mode received an initial study invitation letter within 42 days of the patient’s discharge date, and then were called up to 10 times over a 6-week period. Trained bilingual research assistants fluent in Spanish and English conducted telephone interviews. Caregivers randomized to the mailed survey mode received an initial mailing within 42 days of the patient’s discharge date, with a second mailing to nonrespondents 1 month later. Surveys were mailed in English or Spanish according to the caregiver’s indicated preference for medical communication in hospital administrative data. Following the Child Hospital Consumer Assessment of Healthcare Providers and Systems survey administration protocol, only responses received within 42 days of initial contact were accepted as completed surveys.²⁰ A 1-time $25 participant incentive for completing the survey was mentioned in the study invitation letter.

All study procedures were approved by the Seattle Children’s Research Institute and RAND Corporation institutional review boards.

Data Analysis

Chi-square tests were used to examine differences in the demographic distribution of respondents according to survey mode. We also examined survey responses to assess the percentage of missing data, ceiling/floor effects for each measure, and to determine if scores differed on the basis of survey mode or the respondent’s demographic characteristics. To calculate scores for each measure, we first mapped individual measures to corresponding survey items. Through this process we identified the eligible respondents (denominator) and scores (numerator) for each quality measure. Three different scoring algorithms were used to compute the numerator for each measure depending on whether a measure had multiple subcomponents or depending on the available response options. These 3 scoring algorithms are shown in Table 2 with examples. Detailed measure specifications are also available online.²¹ For measures to be scored, all survey items used to determine eligibility had to be completed. Scores for each measure were then compared according to survey mode and respondent demographic characteristics using bivariate linear and logistic regression models.

Table 2.

Examples of Scoring Algorithm According to Type of Measure

Measure Description	Type of Measure	Scoring Algorithm	Example
Caregivers of children discharged from the hospital during the past month should report that, before their child’s most recent discharge, they had an opportunity to ask questions related to: Subcomponents Instructions for care after discharge and Follow-up instructions	Multicomponent measure with a binary response option (yes or no) for each subcomponent	Calculate individual score: Score each sub-component: yes = 100; no = 0 Compute individual mean score on a 0–100 scale: Sum of subcomponent scores divided by the number of nonmissing subcomponents Calculate overall measure score: Sum of individual scores divided by the number of respondents Number of respondents with ≥1 nonmissing subcomponent	Subcomponent response: → a)Yes = 100  b)No = 0 → $\frac{100+0}{2}=50$ → $\frac{50+\dots +\dots }{562}=97$
Caregivers of children discharged from the hospital during the past month and who were discharged with a prescription for new medical equipment, should report that they: Subcomponents Understood how to use the medical equipment at home and Knew who to call with questions	Multicomponent measure with response options (yes, completely; yes, somewhat; or no) for each subcomponent	Calculate individual score: Score each subcomponent: yes, completely = 100; yes, somewhat = 0; no = 0 Compute individual mean score on a 0–100 scale: Sum of subcomponent scores divided by the number of nonmissing subcomponents Calculate overall measure score: Sum of individual scores divided by the number of respondents Number of respondents with ≥1 nonmissing subcomponent	Subcomponent response: → a)Yes, completely = 100  b)Yes, somewhat = 0 → $\frac{100+0}{2}=50$ → $\frac{50+\dots +\dots }{106}=89$
Caregivers of children discharged from the hospital during the past month whose preferred language is other than English should report that they received discharge education in their preferred language	Single-component measure with response options (yes or no)	Calculate individual score: yes = 100; no = 0 Calculate overall measure score: Sum of individual scores divided by the number of respondents Number of respondents with ≥1 nonmissing subcomponent	→ Response: yes = 100 → $\frac{100+\dots +\dots }{50}=98\%$

Results

Evidence Review and Measure Development

Sixteen studies were included in the targeted literature review, with 4 studies conducted in pediatric populations and 12 studies conducted in adult and elderly populations.¹⁶ All of the studies included interventions that were related to either hospital to home transitions or ED to home transitions. We identified 5 family-centered transition care processes in study interventions that were associated with improved patient health outcomes and reductions in health care utilization: 1) providing families with an opportunity to ask questions during the discharge education process, 2) providing families with an opportunity to practice using medical equipment, 3) providing families with discharge education in their preferred language, 4) providing assistance in scheduling follow-up appointments, and 5) providing families with comprehensive, individualized written discharge instructions. We also included 3 expert guidelines published by the American Academy of Pediatrics (AAP) related to transitions between sites of care in the review, which provided supporting evidence for the care processes noted previously, as well as support for the provision of an emergency care plan for children with chronic conditions at the time of discharge.^22-24 We drafted 8 hospital to home and 3 ED to home transition measures that assessed these care processes on the basis of the supporting evidence that was available for the specific care setting (ie, inpatient [13 studies or 2 guidelines] or ED [3 studies or 1 guideline]).

Six of the 8 hospital to home transition measures and 2 of the 3 ED to home transition measures were endorsed by the Delphi panel as being valid and feasible to collect. The measures that were not endorsed had validity ratings <7 and thus did not meet the threshold for panel endorsement (Table 1).

Survey Pilot Test

A total of 1419 caregivers were invited to complete the hospital to home transition survey, and response rates were 54% (n = 381 of 709) by telephone and 26% (n = 188 of 710) by mail (difference, 27%; 95% confidence interval, 22%–32%). A total of 1420 caregivers were invited to complete the ED to home transition survey, and response rates were 51% (n = 365 of 710) by telephone and 22% (n = 159 of 710) by mail (difference, 29%; 95% confidence interval, 24%–34%). The final sample disposition is shown in the Figure.

Figure.

Final disposition of sample. ED indicates emergency department.

The demographic distribution of respondents was significantly different according to survey mode on the basis of the respondent’s race/ethnicity for each survey (Table 3). A larger proportion of Hispanic caregivers randomized to telephone mode completed the hospital to home survey, and a larger proportion of caregivers indicating “another race” randomized to telephone mode completed the ED to home survey. There were no other significant differences in respondent demographic distributions according to survey mode.

Table 3.

Demographic Distribution of Respondents According to Survey Mode

	Hospital to Home Survey			Emergency Department to Home Survey
Respondent Demographic Characteristic	Respondents by Mail (n = 188), %	Respondents by Telephone (n = 381), %	P *	Respondents by Mail (n = 159), %	Respondents by Telephone (n = 365), %	P *
Caregiver age, years			.89			.27
<35	37	39		35	42
35–44	39	38		45	40
≥45	24	23		21	18
Caregiver gender			.12			.05
Male	9	14		12	19
Female	91	86		88	81
Caregiver race/ethnicity†			.04‡			.04‡
White	71	61		67	54
Hispanic	16	19		16	27
Black	3	3		5	5
Asian/Pacific Islander	6	7		7	6
Other	4	11		6	8
Caregiver education			.96			.32
Lower educational attainment§	26	26		22	26
Higher educational attainment¶	74	74		78	74
Caregiver language preference for medical communication			.09			.16
English	94	90		90	85
Another language	6	10		10	15

^*Difference in demographic distribution between respondents according to survey mode.

^†Sum does not equal 100% because of rounding.

^‡Values in bold indicate a P value < 0.05.

^§Lower educational attainment includes respondents who at most received a high school degree or general educational development certification.

^¶Higher educational attainment includes respondents who completed at least some college education.

Missing data accounted for ≤2% of survey items for each quality measure. Scores for each quality measure are presented in Table 4. Overall scores were high, with ceiling effects noted for 3 of the measures. No floor effects were observed.

Table 4.

Caregiver-Reported Pediatric Quality Measure Scores for Transitions Between Sites of Care

Short Measure Description	Eligible Caregivers, n	Measure Score
		Mean (SD)*	%†
Hospital to home transitions
Had the opportunity to ask questions before discharge	562	97 (15)	—
Understood how to use medical equipment at home and who to call with questions	106	89 (24)	—
Received discharge education in preferred language	50	—	98
Received assistance with scheduling follow-up appointments	549	—	82
Received written discharge instructions	561	—	97
Written discharge instructions were easy to understand, useful, and contained necessary follow-up and contingency plan information	541	90 (21)	—
Emergency department to home transitions
Understood how to use medical equipment at home and who to call with questions	47	83 (30)	—
Received discharge education in preferred language	65	—	75

^*Multicomponent measure scores computed as the mean and SD of nonmissing subcomponent scores.

^†Single-component measure scores computed as the proportion of caregivers who endorsed receiving the indicated care.

Scores significantly differed according to survey mode for 1 measure. Caregivers randomized to the telephone mode reported significantly higher scores on the measure that assessed whether caregivers had the opportunity to ask questions before discharge (telephone: mean score = 98 [n = 370] vs mail: mean score = 94 [n = 183]; P <.01).

Caregivers with lower educational attainment and Spanish-speaking caregivers reported significantly higher scores on 2 of the hospital to home measures: 1) endorsed receiving assistance with scheduling follow-up appointments, and 2) endorsed receiving written discharge instructions that were easy to understand, useful, and complete (Table 5). Additionally, in the ED setting, a significantly larger proportion of Spanish-speaking caregivers with lower educational attainment reported receiving discharge education in their preferred language compared with Spanish-speaking caregivers with higher educational attainment. Measure scores were not associated with any other caregiver characteristics.

Table 5.

Transition Quality Measure Scores According to Caregiver Education and Language Preference for Medical Communication

Short Measure Description	Lower Educational Attainment†		Higher Educational Attainment‡		Spanish-speaking§		English-speaking§
	n	Measure Score∥,¶	n	Measure Score∥,¶	n	Measure Score∥,¶	n	Measure Score∥,¶
Hospital to home transitions
Had the opportunity to ask questions before discharge	145	98 (9)	410	96 (16)	50	98 (10)	510	96 (15)
Understood how to use medical equipment at home and who to call with questions	33	92 (18)	72	88 (26)	16	91 (20)	90	89 (25)
Received discharge education in preferred language	47	98%	3	100%	50	98%	—	—
Received assistance with scheduling follow-up appointments	142	92%***	401	79%	47	96%*	500	81%
Received discharge instructions	145	97%	410	97%	50	96%	509	97%
Written discharge instructions were easy to understand, useful, and contained necessary follow-up and contingency plan information	141	95 (15)***	394	88 (22)	48	97 (9)**	491	89 (21)
Emergency department to home transitions
Understood how to use medical equipment at home and who to call with questions	14	86 (31)	33	82 (30)	7	79 (39)	40	84 (29)
Received discharge education in preferred language	57	81%*	8	38%	65	75%	—	—

^*P < .05

^**P < .01

^***P < .001: significance level of difference in scores compared with high educational attainment category or English-speaking caregiver category.

^†Lower educational attainment includes respondents who at most received a high school degree or general educational development certification.

^‡Higher educational attainment includes respondents who completed at least some college education.

^§On the basis of the caregiver’s reported preferred language for medical communication.

^∥Multicomponent measure scores computed as the mean and SD of nonmissing subcomponents.

^¶Single-component measure scores computed as the proportion of caregivers who endorsed receiving the indicated care.

Discussion

Through a rigorous measure development process, the COE4CCN developed 6 hospital to home and 2 ED to home transition quality measures that provide an assessment of quality from the caregiver perspective. The 6 hospital to home measures assess multiple evidence-based transition care processes such as the quality of discharge education, follow-up care coordination, and written discharge instructions. These hospital to home measures were primarily on the basis of evidence from studies conducted in adult and elderly populations; however, Delphi panel member ratings reflected a consensus that these measures represented high-quality pediatric inpatient care as well. In contrast, only 3 survey measures were developed for ED to home transitions because of limited evidence in the pediatric and adult ED literature, and only 2 of these were endorsed by the Delphi panel. As new evidence continues to emerge in the pediatric hospital and ED settings, these measures should be routinely reviewed and revised, and additional measures should be developed to enhance our evaluation of transition quality from the caregiver perspective.

There was strong consensus among members of the Care Transitions Working group that the optimal source of data collection for these measures is caregiver report, which was also validated by the Delphi panel. For example, a caregiver reporting they had the opportunity to ask questions or understood how to use medical equipment was determined to be a better reflection of high-quality discharge education compared with documentation in the medical record that a staff member provided verbal discharge instructions or provided an opportunity to practice using medical equipment. Furthermore, data required to assess the quality of these care processes are not available from more easily accessible data sources such as administrative data.

Because these measures rely on caregiver-reported survey data, pilot testing the surveys was a critical component of the measure development process to inform future testing and implementation of these survey measures. Results from the pilot test were 4 key findings: 1) with 1 exception, measure scores were similar regardless of survey mode, 2) for 3 measures, scores differed on the basis of the respondent level of education or preferred language for medical communication, 3) 3 measures showed ceiling effects, and 4) survey administration by telephone mode yielded a more racially and ethnically diverse sample.

First, measure scores were overall similar regardless of survey mode suggesting that scores on these measures might not be influenced by social desirability bias when a respondent is interacting with an interviewer by telephone. The difference in scores according to mode for 1 of the measures (caregivers reporting that they had an opportunity to ask questions) might be reflective of actual performance at the study site; however it is unclear whether this difference in scores is clinically meaningful (98 by telephone interview vs 94 by mail). Because previous studies have shown mode effects with similar types of surveys,²⁵ further testing in larger samples is needed to fully understand if any mode effects exist for these surveys.

Second, in terms of differences in scores according to respondent demographic characteristics, caregivers with lower educational attainment and Spanish-speaking caregivers reported higher scores on 2 hospital to home transition measures and 1 ED to home transition measure. These higher scores might reflect true differences in performance, different care expectations among these groups compared with caregivers with higher educational attainment or English-speaking caregivers, or differences in survey scale use for these groups.^25,26 For example, previous studies using the Consumer Assessment of Healthcare Providers and Systems survey have shown that Hispanic respondents and those with lower levels of socioeconomic status tend to choose the highest or most positive response option when completing these types of surveys.^27,28 We are less inclined to believe that caregivers with lower educational attainment and Spanish-speaking caregivers interpreted survey items differently, because this was assessed during the cognitive interview phase of survey development. However, if these findings persist with further testing in larger, more representative samples, additional cognitive testing should be considered to understand the etiology of these differences. Case-mix adjustment might also be necessary if these measures are used in the future to compare performance across hospitals.

Third, the ceiling effects observed for 3 of the measures might reflect actual performance at the study institution as a result of multiple quality improvement efforts targeted to-ward optimizing the discharge process and experience for families. Ceiling effects might also be because of nonresponse bias, with caregivers having more negative experiences being less likely to respond to the survey. Although this bias might overestimate the quality of care, it should not affect performance comparisons across hospitals.^25,29 Further testing is needed to examine the distribution of scores across hospitals, and to determine whether the observed ceiling effects would limit the utility of these measures for assessing quality improvement intervention effectiveness.

Finally, survey administration using telephone mode resulted in a more racially and ethnically diverse sample. This suggests administration of these survey measures using telephone mode might help to ensure patients of diverse racial and ethnic backgrounds are adequately represented when assessing the quality of hospital and ED to home transitions.³⁰ Furthermore, using a mixed-mode approach (mailed survey followed by telephone) might provide an optimal approach to obtaining a more representative sample, while maintaining lower survey administration costs. If these measures are to be widely disseminated in the future, further testing using different survey modes and potentially new forms of technology is needed to determine how to obtain a more generalizable sample to examine performance variation across hospitals.

The need for rigorously developed, responsive transition quality measures in pediatrics is particularly important because of the recent focus on quality improvement interventions related to the discharge process in the hospital and the ED settings.^31,32 By providing caregivers with a voice in evaluating pediatric care transitions, these measures might provide a standardized method to evaluate whether quality improvement initiatives are effective in preparing caregivers for these transitions between sites of care.³³ A key strength of this study is that family advocates were involved as members of the COE4CCN Care Transitions working group and were represented on the multistakeholder Delphi panel.

This study has several limitations. First, the pediatric evidence guiding this measure development work was sparse resulting in a need to primarily rely on studies conducted in the inpatient setting and in adult and elderly populations. Second, measure testing occurred at a single academic institution. On the basis of this limited testing, we can not make conclusions regarding whether these measures are useful for comparing quality across hospitals or for accountability purposes. Third, despite providing participant incentives, response rates were low making our findings less generalizable. These lower response rates might have also compromised our ability to detect meaningful variation in quality measure scores according to survey mode and respondent demographic characteristics. Further multi-institution measure testing with larger more representative caregiver samples is needed to fully understand the utility of these measures for making quality comparisons across hospitals, the effect of survey mode on measure performance, and whether case-mix adjustment is necessary to make fair quality comparisons.

In conclusion, the COE4CCN developed 8 caregiver-reported survey measures to assess the quality of transitions between sites of care from the caregiver perspective. These measures are evidence-based and show face validity. Preliminary measure testing revealed minimal mode effects on measures scores; however, we did note differences in measure scores on the basis of respondent demographic characteristics for some measures. Future directions include examining variation in performance across hospitals and in more diverse populations, further validation testing to assess the association between measure performance and health outcomes, and evaluating whether these measures are responsive to quality improvement interventions.

What’s New.

In this report we describe the development and pilot testing of 8 new caregiver-reported quality measures for pediatric hospital and emergency department to home transitions. These measures might be used to standardize the evaluation of transition-related quality from the perspective of caregivers.