Development and validation of a patient experience of care survey for emergency departments

Abstract

Objectives

To (1) develop a survey to assess the patient experience of care in hospital‐based emergency departments (ED) and (2) evaluate the reliability and validity of composite measures of patient experience using data collected through the experimental implementation of the newly developed Emergency Department Patient Experience of Care (EDPEC) Discharged to Community (DTC) Survey.

Data Source

4893 adult patients were treated in the ED of 16 hospitals across the United States in 2018.

Study Design

The study utilized a cross‐sectional survey.

Data Collection

Survey development activities included a literature review, focus groups, and cognitive interviews with recently discharged ED patients, technical expert panels, and multiple field experiments. Survey development resulted in a 34‐item instrument; the analysis reported here focuses on 18 items on patient experience of care. Using data from the EDPEC DTC Survey in the 2018 Feasibility Test, we performed confirmatory factor analysis to group 15 evaluative survey items into composite measures. We examined internal consistency reliability, interunit reliability, and associations between each composite measure and patients' overall rating and willingness to recommend the ED.

Principal Findings

Analyses of 15 evaluative items identified four composite measures: Getting Timely Care, How Well Doctors and Nurses Communicate, Communication about Medications, and Communication about Follow‐up. Patient‐level internal consistency reliability exceeded 0.75 for two of four composites; ED‐level internal consistency reliability exceeded 0.83 for all four composites. Interunit reliability estimates indicated that 450 survey completes per ED results in at least 0.70 reliability for all composites. Higher scores on each composite were associated with higher overall ratings and willingness to recommend the ED.

Conclusions

The composite measures derived from the EDPEC DTC Survey are statistically reliable and valid. These results provide guidance for EDPEC DTC Survey adopters on how to construct meaningful and psychometrically‐sound composite measures for monitoring the quality of care they provide.

What is known on this topic

• There is currently no national publicly‐available patient experience survey for the emergency department setting.

• Emergency department visits tend to be brief and little preparation or prescreening is possible due to limited access to patients' medical histories, making patient experience of care surveys developed for other settings inadequate for capturing the patient experience in the emergency department.

What this study adds

• A national publicly available patient experience survey is developed for the emergency department setting.

• Reliability and validity testing of composite measures support the use of these measures for monitoring the quality of care in the emergency department setting.

• This extensively tested and fully validated survey fills the need for a national standardized survey to measure patient experience in the emergency department setting among patients discharged to home.

1. INTRODUCTION

Since 1995, the Centers for Medicare & Medicaid Services (CMS) have implemented patient experience surveys for a wide range of health care providers, including the Consumer Assessment of Healthcare Providers and Systems (CAHPS) surveys developed under CMS and Agency for Healthcare Research and Quality sponsorship. These settings include Medicare Fee‐For‐Service, Medicare Advantage, Part D Prescription Drug Plans, hospitals, home health agencies, in‐center hemodialysis facilities, hospices, and clinician and group practices. ¹ Results from these surveys guide quality improvement efforts and inform consumer choice through public reporting of facility or provider scores. ² , ³ , ⁴ However, while nearly 20% of United States adults visit an emergency department (ED) each year, ⁵ , ⁶ there is currently no national publicly‐available patient experience survey for the ED setting.

The ED is a unique setting within the health care system. EDs often bridge the worlds of outpatient and inpatient care and provide a critical means of accessing health care when other options are unavailable. Importantly, most patients in ED settings are treated by providers whom they have not previously met, and from whom they are unlikely to receive care in the future. Compared with physician office visits or hospital inpatients' interactions with providers, ED visits tend to be brief and little preparation or prescreening is possible due to limited access to patients' medical histories. ⁷ , ⁸ , ⁹ These factors make patient surveys developed for other settings inadequate for capturing patient experience in the ED.

To provide insight into patient experience of care in the ED setting, we developed an ED‐specific survey for patients who are discharged to the community (rather than admitted to a hospital) following their visit to a hospital‐based ED—the Emergency Department Patient Experience of Care (EDPEC) Discharged to Community (DTC) Survey. In this article, we describe the development of the EDPEC DTC Survey and the construction and psychometric evaluation of its measures of ED experiences. Because this information is meant to guide quality improvement, it is critical that scores on the composite measures provide reliable and valid summary information on patients' experiences with ED services.

2. METHODS

2.1. Instrument development: Overview

The EDPEC survey instrument was developed and tested according to CAHPS design principles. ¹⁰ These principles specify that survey questions should focus on aspects of care for which the patient is the best or only source of information, be understood and answered in a consistent way across a range of patients, ask patients only about the care that they have experienced and can observe, and ask about aspects of health care delivery that patients deem important.

The content and design of the EDPEC DTC Survey were informed by (1) a literature review of existing surveys and measures of patient experience in the ED, (2) a Federal Register call for topic areas, (3) five focus groups conducted in two locations with racially, ethnically, and socioeconomically diverse patients, (4) technical expert panels conducted in 2012, 2017, and 2018 with health care providers and survey methodologists, (5) rounds of English and Spanish cognitive interviews in 2012, 2016, and 2017 to refine item wording and response options, and (6) large‐scale experiments in 2014, 2016, and 2018 to examine the mode of administration and feasibility of electronic modes.

2.2. Literature review

Using traditional bibliographic databases (e.g., PubMed, PsycInfo), we systematically searched the peer‐reviewed literature on patients' experiences with care in EDs to identify possible survey topics, data collection methods, or reporting/quality improvement issues specific to the ED setting. We also searched PsycTESTS and “gray” literature (produced outside traditional academic publishing and distribution channels). The primary inclusion criteria were that the article (1) focus on of patient experience with ED care or (2) include survey questions or questionnaires about patient experience with ED care. Our search identified 159 articles that met the inclusion criteria and an additional 59 surveys, measures, and reports.

A review of these articles identified three categories of survey items: (1) demeanor of ED staff, a category that captured factors such as courtesy, attitude, concern, attention, care, and respect; (2) communication, which assessed multiple aspects of communication, such as asking questions and listening carefully, and whether the patient understood what the health care provider was saying; and (3) overall assessment, a category that captured patients' overall evaluations of their experiences with care. The literature review informed the development of an initial draft survey instrument and focus group discussion guides.

2.3. Federal Register call for topic areas

In response to a call for topic areas published in the Federal Register in December 2012, ¹¹ stakeholders suggested a number of content areas for the survey, including (1) overall quality of care provided in the ED and willingness of patients to recommend the ED to friends and family; (2) wait times and the extent to which patients were kept informed about delays; (3) the registration and admitting process, including helpfulness of the registration staff and timely completion of the process; (4) the extent to which doctors, nurses, and staff communicated with patients and caregivers about their condition, treatment options, medications, and test results, understood their medical history, listened to their needs, and incorporated patient concerns and preferences into treatment decisions; (5) doctor, nurse, and staff concern and courteousness toward patients and caregivers; (6) pain control and management; (7) lab and test experience, courteousness of staff, and timeliness of results; (8) discharge information; and (9) conditions of the facility, including cleanliness, privacy, and comfort of waiting and treatment areas. This input was used to refine the initial draft survey instrument.

2.4. Focus groups

We convened five focus groups (four in English, one in Spanish) with eight or nine adults ages 18+ in each group. Focus group participants had experience with ED care within the past 6 months or were caregivers for someone who had recently received ED care. Focus groups were conducted by trained facilitators using group‐specific discussion guides. The Spanish focus group, which employed a bilingual, bicultural moderator, included only monolingual Spanish‐speaking participants. Focus group discussions informed the refinement of the initial draft survey instrument.

2.5. Technical expert panels

We conducted a series of technical expert panels, in 2012, 2017, and 2018. The technical expert panels were comprised of experts in survey development or emergency department processes and management. The inaugural technical expert panel discussed the literature review, focus group results, and initial draft survey instrument. The technical expert panels made several suggestions about item wording and restructuring the survey to ask about processes, people, and the environment. The 2017 and 2018 technical expert panels focused on instrument refinement, particularly on reducing survey length and revising questions on medication and follow‐up care, and survey administration, particularly on the potential use of web‐based modes and the modes to be tested in the large‐scale field experiments.

2.6. Cognitive testing

One‐on‐one cognitive interviews were conducted in 2012, 2016, and 2018 with DTC adults ages 18+ with an ED visit within the past 6 months. The 2012 interviews (six in English; seven in Spanish) focused on reviewing 82 draft survey items. These interviews indicated that the general flow of topics within the instrument worked well and that item placement seemed appropriate within the flow of topics. The 2016 interviews (seven in English, five in Spanish) focused on medication items and various response options. The 2017 interviews (six in English; six in Spanish) focused on items related to the discussion of follow‐up care at the time of ED discharge. Well‐performing items were retained on the first experimental version of the survey.

2.7. Large‐scale field experiments

Four large‐scale experiments conducted in 2014, 2016 (two experiments), and 2018, tested the survey among patients from 12 hospitals, 50 hospitals, eight hospitals, and 16 hospitals, respectively. The primary aims of these experiments were to test mode protocols for survey administration and to test performance of survey items and modes of survey implementation; results are described elsewhere. ¹² , ¹³ , ¹⁴ , ¹⁵ Item performance results informed the refinement of the survey instrument; item performance results from the 2018 experiment are described here. Mode protocol results informed CMS' recommendation that the EDPEC DTC survey is administered in one of three modes: (1) standard mixed mode (mail with telephone follow‐up), (2) web survey (by email invitation) with telephone follow‐up, and (3) web survey (by email invitation) followed by a mailed survey with telephone follow‐up.

2.8. Final survey instrument

The final survey instrument (Version 5.0) includes 34 items grouped into seven sections: “Going to the emergency room,” “During this emergency room visit,” “People who took care of you,” “Leaving the emergency room,” “Overall experience,” “Your health care,” and “About you” (see Appendix S1); earlier versions of the survey can be found elsewhere. ¹⁶ The analysis reported here focuses on the 18 evaluative items on Version 5.0 of the survey: 15 evaluative items that were used to construct composite measures of four aspects of patients' experiences of care in the emergency room (see Table 1), one stand‐alone item about getting information about test results, and two overall assessment items that also served a validity‐testing purpose.

TABLE 1.

Composite measures, stand‐alone item, and validation measures: item wording and response options

Composite measures

Getting Timely Care

When you first arrived at the emergency room, how long was it before someone talked to you about the reason why you were there? (More than 15 min; 5–15 min; Less than 5 min)

During this emergency room visit did you get care within 30 minutes of getting to the emergency room? (Yes/No)

How well doctors and nurses communicate (Never; Sometimes; Usually; Always for all)

During this emergency room visit, how often did nurses treat you with courtesy and respect?

During this emergency room visit, how often did nurses listen carefully to you?

During this emergency room visit, how often did nurses explain things in a way you could understand?

During this emergency room visit, how often did doctors treat you with courtesy and respect?

During this emergency room visit, how often did doctors listen carefully to you?

During this emergency room visit, how often did doctors explain things in a way you could understand

Communication about medications

During this emergency room visit, did the doctors or nurses ask about all of the medicines you were taking? (Yes/No)

Before giving you medicine, did the doctors or nurses tell you what the medicine was for? (Yes, definitely; Yes, somewhat; No)

Before giving you medicine, did the doctors or nurses describe possible side effects to you in a way you could understand? (Yes, definitely; Yes, somewhat; No)

Before you left the emergency room, did a doctor or nurse tell you what the medicine was for? (Yes, definitely; Yes, somewhat; No)

Communication about follow‐up

Before you left the emergency room, did staff talk with you about follow‐up care? (Yes, definitely; Yes, somewhat; No)

Did emergency room staff give you information about how to get the follow‐up care you needed? (Yes/No; I did not need information about how to get follow up care)

Before you left the emergency room, did staff give you information about what symptoms or health problems to look out for at home? (Yes, definitely; Yes, somewhat; No)

Stand‐alone Item (for patients answering Yes to the screener item, “During this emergency room visit, did you have a blood test, x‐ray, or any other test?”)

During this emergency room visit, did doctors and nurses give you as much information as you wanted about the results of these tests? (Yes, definitely; Yes, somewhat; No)

Validation Measures

Overall ED rating

Using any number from 0 to 10, where 0 is the worst care possible and 10 is the best care possible, what number would you use to rate your care during this emergency room visit? (0–10)

Willingness to recommend the ED

Would you recommend this emergency room to your friends and family? (Definitely no; Probably no; Probably yes; Definitely yes)

Note: All evaluative items had an inappropriate missingness rate of less than 5%.

2.9. Data source

This study uses data from the 2018 EDPEC DTC Survey experiment, referred to as the 2018 Feasibility Test. This test, conducted in English only, was designed as a randomized experiment powered to detect small differences in response rates comparing different modes of administration. For the analysis reported here, the mode of survey administration was treated as a control variable. We recruited 16 hospitals from around the United States with 14,000 or more annual ED visits that routinely collected patient email addresses and had a process to obtain and document consent to contact patients via text message. Hospitals voluntarily participated in the study and did not receive monetary compensation for participation. ED patients 18 years of age or older were eligible to participate in the survey except those who (1) were admitted to the hospital as a result of the ED visit; (2) left the ED without being seen; (3) requested not to be contacted (“no publicity” patients); (4) were excluded due to state regulations; (5) had a primary diagnosis of mental health or substance use (ICD‐10 codes F01‐F99); (6) were transferred to another hospital; (7) were discharged anywhere other than home (e.g., to a skilled nursing facility); (8) had a foreign home address; (9) were court/law enforcement patients (prisoners); (10) preferred a language other than English; or (11) died in the ED or prior to sampling. Exclusions 1–4 were implemented by each participating hospital; exclusions 5–11 were implemented by the study team and are listed in order of decreasing frequency.

A total of 26,991 ED DTC adult patients discharged between January and March 2018 from the 16 hospitals were sampled and randomized to nine protocol arms. ¹³ Eight of these arms were sequential mixed mode protocols (a mode protocol describes the mode[s], sequence, and timing of data collection) involving one or more invitations to a web survey followed by a mailed survey and/or telephone outreach; the reference mode was standard mixed mode (mail with telephone follow‐up). The overall response rate was 18.6%, yielding an analytic respondent sample of 4893 adult patients treated in the ED of these 16 hospitals.

In addition to evaluative patient experience items (Table 1), the survey collected the patient's education, self‐rated overall health status, language spoken at home, self‐reported reason for the ED visit, arrival by ambulance, and information on proxy assistance with survey completion. Patient age was available from hospital administrative data.

2.10. Data analysis

All evaluative items on the EDPEC DTC Survey except the two items used for validity assessment were scored using top‐box scoring to align with previous scoring of this survey ¹⁴ and the Hospital CAHPS (HCAHPS) Survey of hospital inpatients. ¹⁷ In top‐box scoring, the most favorable response option was coded as 100 and all other response options were coded as 0. For example, for the question, “Before giving you medicine, did the doctors or nurses tell you what the medicine was for?, the response option “Yes, definitely” was coded 100, while the options “Yes, somewhat” and “No.” were coded as 0. One question (Did emergency room staff give you information about how to get the follow‐up care you needed?) included a tailored nonapplicable response option. When this response was selected, the item was not scored. All analyses were also conducted using linear mean scoring, which gives “partial credit” for intermediate response options; ¹⁶ results were similar and are not reported here.

We used confirmatory factor analysis to evaluate the factor structure of 15 evaluative survey items using robust maximum likelihood estimation. In preliminary factor analyses, we found that one item, which asked whether doctors and nurses provided sufficient information about test results, cross‐loaded similarly on two factors: How Well Doctors and Nurses Communicate, and Communication about Medications. As such, we removed this item from the factor analytic models presented here and instead included it as a stand‐alone item in the regression models used to evaluate construct validity. Specification of the factor analytic models was based on findings from similar analyses conducted on the data from the 2014 field test ¹⁵ and an earlier mode experiment. ¹⁴

We tested three factor analytic models:

• Model 1: a four‐factor model with Getting Timely Care, How Well Doctors and Nurses Communicate, Communication about Medications, and Communication about Follow‐up as the factors (see Table 1).

• Model 2: a three‐factor model similar to Model 1 but which combined the items in Communication about Follow‐up with the items in How Well Doctors and Nurses Communicate into one factor.

• Model 3: a three‐factor model similar to Model 1 but which combined the items in Communication about Follow‐up with the items in Communication about Medications into one factor.

Because of the nonimpedence of patients within EDs, we estimated intra‐class correlations (ICC; an estimate of the degree of similarity of responses from patients who received care from the same ED) of the items analyzed; item ICCs ranged from 0.004 to 0.045. To account for differences between EDs, we centered each patient's scores within the ED at which they received care by subtracting the ED mean from each item used in the analysis. To assess the appropriateness of each factor model, we examined factor loadings with the criterion that they should be ≥0.40. ¹⁸ We evaluated overall model fit using the Comparative Fit Index (CFI), the root mean square error of approximation (RMSEA), and standardized root mean square residual (SRMR). Prior research indicates that a model with good fit has CFI >0.95, RMSEA <0.06, and SRMR <0.08. ¹⁹

Based on the results of the confirmatory factor analysis, we computed composite measures (scales) of patient experience in the ED by taking the mean of nonmissing item scores for a scale. Because the composite measures were computed at the patient level and then aggregated to the ED level, we estimated the internal consistency reliability of the scales at both levels using between‐ and within‐hospital covariance matrices estimated in a multilevel framework. ²⁰ At the ED level, high internal consistency indicates that an ED with a high percentage of patients endorsing one item in a composite is likely to have a high percentage of patients endorsing other items in the composite. In contrast, high inter‐hospital reliability means that endorsement rates for a composite are similar for patients in the same hospital. To assess inter‐hospital reliability (the degree to which the measure assesses the true difference among EDs), we first computed the ICC of each composite measure and then used that information along with the Spearman–Brown prophecy formula to (1) estimate the number of completed surveys necessary to achieve an inter‐hospital reliability of 0.70 and (2) estimate the expected inter‐hospital reliability for each composite assuming at least 450 completed surveys per ED (the minimum recommended by CMS for this survey).

Finally, we used mixed‐effects linear regression, conducted at the patient‐level with random ED effects, to assess the construct validity of the composite measures by examining the associations of these measures with (1) overall rating of care received during the ED visit (response options 0–10), (2) likelihood the patient would recommend the ED (4 response options). To take full advantage of all the response options, both items used for validity testing were linear mean scored, whereby responses were transformed to a 0–100 scale with equal spacing between response options. Separate models were conducted for each of these two measures.

We began by estimating models that included each measure (separately) as a predictor of the validation measure to assess the predictive utility of each measure. We then estimated a second series of models that included all measures simultaneously to determine whether the measures have unique contributions to the validation measures. All models adjusted for ED, mode of survey administration, and the following case mix adjustment variables: age, education, self‐rated overall health status, language spoken at home, self‐reported reason for the ED visit, arrival by ambulance, proxy assistance with survey completion, and response percentile. Previous analyses have demonstrated the need for adjustment for mode of survey administration and these case mix adjustment variables. ¹³ , ¹⁴ Response percentile captures both ED response rate and how quickly a patient responded to the survey compared to other patients in the same ED and mode of administration; ¹⁴ it is defined as the rank‐ordered number of days between a respondent's discharge date and the date that data collection activities ended for the respondent relative to all eligible patients within ED and mode of administration, scaled from 0 to 1. We adjusted for this variable because later responders often provide more negative responses than those who respond quickly. ²¹ , ²² , ²³ Missing case mix adjustment variables were imputed using the ED‐level mean.

3. RESULTS

Table 2 describes the characteristics of the 4893 survey respondents. The majority were female (62.7%), non‐Hispanic White (68.2%), spoke English at home (91.3%), and did not arrive to the ED by ambulance (81.4%). The self‐reported reason for the ED visit varied, with 44.0% visiting the ED due to a new health problem, 31.9% visiting the ED due to an ongoing health condition or concern, and 23.1% visiting the ED due to an accident or injury.

TABLE 2.

Emergency Department Patient experience of care discharged to community survey respondent characteristics (N = 4893)

Characteristics	Category	N (%)
Age	18–24	348 (7.1%)
	25–34	574 (11.7%)
	35–44	607 (12.4%)
	45–54	669 (13.7%)
	55–64	906 (18.5%)
	65–74	880 (18%)
	75–84	611 (12.5%)
	85 +	298 (6.1%)
Gender	Female	3084 (63%)
	Male	1809 (37%)
Self‐reported overall health	Poor	294 (6%)
	Fair	1063 (21.7%)
	Good	1534 (31.4%)
	Very good	1304 (26.7%)
	Excellent	591 (12.1%)
	Unknown	107 (2.2%)
Highest education completed	8th Grade or less	142 (2.9%)
	Some high school	350 (7.2%)
	High school degree or equivalent	1237 (25.3%)
	Some college or 2‐year degree	1652 (33.8%)
	4‐Year college graduate	674 (13.8%)
	More than 4‐year college degree	706 (14.4%)
	Unknown	132 (2.7%)
Race/ethnicity	Hispanic	457 (9.3%)
	Native American	42 (0.9%)
	Black	548 (11.2%)
	Asian or Pacific Islander	119 (2.4%)
	White	3342 (68.3%)
	Multi‐racial	185 (3.8%)
	Unknown	200 (4.1%)
Language spoken at home	Primary English	4455 (91%)
	Primary Spanish	140 (2.9%)
	Primary other	137 (2.8%)
	Unknown	161 (3.3%)
Arrival by ambulance	No	3979 (81.3%)
	Yes	882 (18%)
	Unknown	32 (0.7%)
Reason for Emergency Department visit	Accident or injury	1105 (22.6%)
	A new health problem	2138 (43.7%)
	An ongoing health condition or concern	1575 (32.2%)
	Unknown	75 (1.5%)

Note: Self‐reported mental health and responses to proxy items not shown here.

3.1. Confirmatory factor analyses

Model modification indices suggested adding four residual covariances among some of the items on How Well Doctors and Nurses Communicate to account for covariance related to the similar wording of items in that domain. Four residual covariances were included in all three models, and their correlation values ranged from 0.15 to 0.34.

A comparison of the fit indices suggested that Model 1 (four‐factor model; χ ² _[80] = 761.02; CFI = 0.96; RMSEA = 0.04; SRMR = 0.03) provided the best fit to the data. The three‐factor model in which all communication items were forced to load on a single factor (Model 2; χ ² _[83] = 1516.49; CFI = 0.91; RMSEA = 0.06; SRMR = 0.04) fit significantly worse than Model 1, suggesting that the items on Communication about Follow‐up share common variance distinct from the more general communication items. The three‐factor model that combined items on Communication about Medications and Communication about Follow‐up (Model 3; χ ² _[83] = 955.71; CFI = 0.95; RMSEA = 0.05; SRMR = 0.03) fit slightly better than Model 2 but not as well as Model 1. This suggested that the items on Communication about Follow‐up have common variance that is statistically distinct from the items on Communication about Medications.

Table 3 shows standardized factor loadings for the four‐factor factor analytic model. Factor loadings were uniformly high (average standardized loading was 0.67). Inter‐factor Correlations, also shown in Table 3, were high and positive, with an average correlation of 0.67. The highest correlation was between Communication about Medications and Communications about Follow‐up (r = 0.81).

TABLE 3.

Standardized factor loadings and inter‐factor correlations from confirmatory factor analytic model 1

Factor/items		Standardized loading	Item‐total correlation
Getting Timely Care
How long until someone talked to you about the reason for the ER visit		0.48	0.28
Did you get care within 30 minutes of getting to the ER		0.55	0.28
How Well Doctors and Nurses Communicate
How often nurses treated you with courtesy and respect		0.67	0.65
How often nurses listened to you carefully		0.76	0.73
How often nurses explained things in a way you could understand		0.78	0.71
How often doctors treated you with courtesy and respect		0.72	0.72
How often doctors listened to you carefully		0.76	0.76
How often doctors explained things in a way you could understand		0.77	0.73
Communication about Medications
Doctors or nurses asked you about all the medicines you were taking		0.40	0.24
Doctors or nurses told you what medicine was for		0.63	0.48
Doctors or nurses explained possible side effects of medicines		0.60	0.38
Doctors or nurses told you what medicine was for (at discharge)		0.67	0.45
Communication about Follow‐up
Talk with you about follow‐up care before leaving ER		0.80	0.68
Information on how to get follow up before leaving ER		0.69	0.61
Information on symptoms to look at for at home		0.71	0.58
Inter‐factor Correlations
	Getting Timely Care	How Well Doctors and Nurses Communicate	Communication about Medications
1. Getting Timely Care	–
2. How Well Doctors and Nurses Communicate	0.61	–
3. Communication about Medications	0.56	0.80	–
4. Communication about Follow‐up	0.53	0.73	0.81

Note: Prior to analysis, item responses were centered within each emergency department to account for clustering of patient responses; factors loadings should be ≥0.40.

Abbreviation: ER, emergency room.

3.2. Internal consistency reliability of composite measures

Patient‐level internal consistency reliability of the composite measures were 0.44 (Getting Timely Care), 0.89 (How Well Doctors and Nurses Communicate), 0.61 (Communication about Medications), and 0.78 (Communication about Follow‐up). At the ED‐level, the composite measures had a higher degree of internal consistency reliability: 0.83 (Getting Timely Care), 0.98 (How Well Doctors and Nurses Communicate), 0.91 (Communication about Medications), and 0.96 (Communication about Follow‐up).

3.3. Interunit reliability of composite measures

The estimated ICCs for the composite measures ranged from 0.005 to 0.055. Based on these estimates, the number of completed surveys per ED needed for adequate measure‐level reliability (defined as 0.70) ranged from 40 for the Getting Timely Care measure, to 447 for the Communication about Medications measure. Inter‐hospital reliability assuming 450 completes per ED ranged from 0.70 to 0.96 (see Table 4).

TABLE 4.

Inter‐hospital reliability of composites from the Emergency Department Patient experience of care discharged to community survey

Composite measure	ICC a	N needed per hospital for reliability = 0.70 b	Pooled inter‐hospital reliability assuming 450 completed surveys/ED b
Getting Timely Care	0.055	40	0.963
How Well Doctors and Nurses Communicate	0.012	193	0.845
Communication about Medications	0.005	447	0.702
Communication about Follow‐up	0.007	336	0.758

Abbreviations: ED, Emergency Department; ICC, intra‐class correlation.

^a ICC estimates adjusted for hospital, survey mode, and the following case mix variables: age, education, self‐rated health status, language spoken at home, self‐reported reason for the ED visit (new health problem vs. ongoing health condition or concern vs. accident or injury), whether the patient arrived by ambulance, response percentile, whether the patient had a proxy answer for them, and whether the patient used proxy assistance (in some way other than answering for them).

^b Calculated with the Spearman–Brown prophecy formula, reliability = (k*ICC)/[(k − 1)(ICC) + 1], where k is the number of completed surveys per ED.

3.4. Construct validity

The first three columns of Table 5 display regression results examining the association between each measure and patients' overall rating of care received and willingness to recommend the ED when each measure and stand‐alone item is included in the regression alone. The correlation between patients' overall rating of care received and willingness to recommend the ED was 0.81. All associations between the composite measures and these two validation measures were positive and significant. The second set of columns of Table 5 presents the multivariate results. When all four composites measures and the stand‐alone item were included together as predictors, all remained significantly associated with the validation measures. When the four measures and the stand‐alone item are considered simultaneously, the strongest positive associations with both validation measures were observed for How Well Doctors and Nurses Communicate, followed by Getting Timely Care.

TABLE 5.

Construct validity results

	One‐at‐a‐time			Simultaneous
	Regression coefficient	Standard error	p‐Value	Regression coefficient	Standard error	p‐Value
Overall rating
Composite measures
Getting Timely Care	0.279	0.009	<0.001	0.105	0.007	<0.001
How Well Doctors and Nurses Communicate	0.498	0.007	<0.001	0.325	0.010	<0.001
Communication about Medications	0.383	0.011	<0.001	0.071	0.011	<0.001
Communication about Follow‐up	0.327	0.008	<0.001	0.079	0.008	<0.001
Stand‐alone item
Doctors and nurses provided sufficient information about test results	0.251	0.007	<0.001	0.057	0.007	<0.001
Willingness to Recommend
Composite measures
Getting Timely Care	0.292	0.010	<0.001	0.115	0.010	<0.001
How Well Doctors and Nurses Communicate	0.504	0.009	<0.001	0.335	0.013	<0.001
Communication about Medications	0.388	0.013	<0.001	0.065	0.014	<0.001
Communication about Follow‐up	0.333	0.009	<0.001	0.083	0.011	<0.001
Stand‐alone item
Doctors and nurses provided sufficient information about test results	0.253	0.008	<0.001	0.053	0.009	<0.001

Note: All regression models adjust for hospital, survey arm, and the following case mix variables: age, education, self‐rated health status, language spoken at home, self‐reported reason for the emergency department visit (new health problem vs. ongoing health condition or concern vs. accident or injury), whether the patient arrived by ambulance, response percentile, whether the patient had a proxy answer for them, and whether the patient used proxy assistance (in some way other than answering for them).

4. DISCUSSION

Our results support the reliability and validity of the four EDPEC DTC Survey composites among ED patients discharged to the community: Getting Timely Care, How Well Doctors and Nurses Communicate, Communication about Medications, and Communication about Follow‐up. Patient‐level internal consistency reliability exceeded 0.75 for two of the four composites. Although patient‐level inconsistency for the Getting Timely Care measure was only 0.44, the composite measures are intended to measure the quality of care at the ED level, and ED‐level internal consistency reliability exceeded 0.83 for all composites, we conclude that these composites have acceptable internal consistency reliability.

Interunit reliability is particularly important for comparing EDs. We found that the average interunit reliability for EDs with ≥450 survey completes exceeded 0.70 for all composites. The ICC estimate was largest for the Getting Timely Care composite and was small (≤0.01) for the three communication composites, both of which are consistent with results observed within the Clinician and Group CAHPS Survey. ²⁴ The ICC estimate was smallest for Communication about Medications, indicating that reliable comparison of EDs on this measure requires large sample sizes. Future research should investigate whether pooling data from multiple survey years, as is done for the CAHPS Hospice Survey, ²⁵ might improve the reliability of comparisons on this measure.

Our factor analysis suggested that Communication about Medications and Communication about Follow‐up should not be combined with How Well Doctors and Nurses Communicate, which is consistent with what has been found for the HCAHPS Survey. ²⁶ In addition, all three communication composites uniquely contributed to the prediction of patients' overall rating and willingness to recommend the ED. This suggests that these three composites capture different aspects of the communication processes that occur in the ED, perhaps reflecting a division of roles characteristic of the ED setting. Different staff members may be responsible for different communications—one staff member may be responsible for talking to patients about medications, another for communicating with patients about their symptoms and treatment, and yet another for talking to patients about follow‐up.

The validity of the ED patient experience measures was supported by the significant positive association between them and overall rating and willingness to recommend the ED. How Well Doctors and Nurses Communicate had the strongest relationship with the two validation measures of the ED experience, which is consistent with earlier studies and implies that communication with doctors and nurses is the strongest driver of a patient's overall experience. ²⁴ , ²⁶ , ²⁷ The other two communication composites on medications ²⁶ , ²⁸ and follow‐up had a moderately positive bivariate association with the validation measures, although their predictive strength was notably reduced when their associations with other measures, especially How Well Doctors and Nurses Communicate, was accounted for. Given the nature of the ED setting, one may have hypothesized that Getting Timely Care would be the most important driver of overall experience, not communication. However, we found Getting Timely Care to be the weakest driver (though still significant) of overall experience. In the hospital inpatient setting, doctor and nurse communication was also found to be the most important driver of overall experience of care, ²⁶ which implies that for both transitory ED patients and admitted inpatients, such communication is perceived as crucial.

4.1. Limitations

While our study is based on a large number of respondents, these were drawn from just 16 hospital‐based EDs and may not be representative of all EDs in the United States. Hospitals with <14,000 annual ED visits were not eligible for this study due to concerns that they would not have a sufficient eligible sample during the field period of the experiment. However, 91% of annual ED visits take place in hospitals with 14,000 or more annual ED visits. ²⁹ Noting that our overall response rate was 18.6%, survey respondents may not be representative of all ED DTC patients. Nonresponse bias is possible if characteristics associated with nonresponse are also associated with care experiences. ¹³ However, previous work suggests that adjusting for differences in case mix, as done in this study, adequately addresses any nonresponse bias. ²¹ Our observed response rate reflects the challenging‐to‐survey ED patient population that is relatively young and mobile compared to those served in other health care settings. ¹⁵ , ²¹ , ³⁰ As well, studies have found that low response rates do not necessarily indicate the presence of nonresponse bias. ²¹ , ³¹ , ³² The relatively small number of hospitals prevents us from examining the factor structure using multilevel confirmatory factor analysis to account for the clustered nature of the data. Future studies that are based on a larger and more representative sample of EDs and that use multilevel models are needed to examine the factor structure at both the within‐ED and between‐ED levels.

The EDPEC survey is designed to provide a broad assessment of patients' experience of care in the ED; thus, patients who left the ED without being seen were not part of the survey sample. This omission likely weakens the measured association between timeliness of care and our validation measures. Assessing this possibility would be an interesting area of future study. Lastly, it is possible that not all visits in this study were clinically necessary ED visits. More work is needed to understand the relationship between ED experience, lack of access to non‐ED care, and clinically appropriate utilization.

5. CONCLUSION

The extensively tested and fully validated EDPEC survey fills the need for a national standardized survey to measure patient experience in the ED setting among patients discharged to home. Results presented here demonstrate that the EDPEC DTC Survey is a valid and reliable instrument for assessing important dimensions of ED patient experience that are amenable to quality improvement. The research presented here may serve as a guide on how to construct meaningful and psychometrically‐sound composite measures for monitoring the quality of care that emergency departments provide.

Since the completion of this study, the EDPEC survey received the CAHPS trademark and is now the ED CAHPS Survey. The ED CAHPS Survey instrument and recommended guidelines describing survey administration and analysis are publicly available. ¹⁶

DISCLAIMER

The content of this publication neither necessarily reflects the views or policies of the Department of Health and Human Services nor does the mention of trade names, commercial products, or organizations imply endorsement by the U.S. Government. The authors assume full responsibility for the accuracy and completeness of the ideas presented.

Supporting information

Appendix S1 Supporting information

Ye F, Parast L, Hays RD, et al. Development and validation of a patient experience of care survey for emergency departments. Health Serv Res. 2022;57(1):102‐112. 10.1111/1475-6773.13853