Person-centred emergency care: translation and cross-cultural validation of the Australian Emergency Department Patient-Reported Experience Measure (ED PREM) in the Netherlands

Abstract

Background

Measuring and evaluating patient experience in the emergency department (ED) is essential for improving the quality of emergency care. However, specific and validated tools to measure patient experience in the ED are lacking in many countries. The aim of this study was therefore to translate and cross-culturally validate the Australian ED Patient-Reported Experience Measure (ED PREM) for the Dutch setting.

Methods

A single-centre validation study was conducted in a Dutch academic hospital. Step 1 involved forward and backward translation to produce a Dutch translation of the ED PREM (prototype). Step 2 involved establishing face and content validity of the Dutch prototype through cognitive patient interviews and researcher team consensus meetings. Step 3 involved administering the resultant Dutch pilot ED PREM to an ED population (n=527), to be able to perform psychometric analyses on this sample. Descriptive statistics and item reduction analyses were conducted prior to confirmatory factor analysis (CFA). CFA was used to confirm the structural validity of the Dutch ED PREM, and internal consistency was assessed.

Results

The face and content validity of the prototype Dutch ED PREM was ‘good’ after 15 cognitive interviews and expert consultation. Over two-thirds of participants (357/527) completed the Dutch pilot ED PREM. Scores were high across all four domains. Several items were removed due to high ceiling effects and a large number of ‘not applicable’ responses. The final 18-item Dutch ED PREM showed acceptable model fit with CFA (χ²(df) 7519.55 (153), p<0.001, Root Mean Square Error of Approximation=0.090, Tucker-Lewis Index=0.942, Comparative Fit Index=0.951, Standardised Root Mean Square Residual=0.035). Internal consistency was high (Cronbach’s alpha ranged 0.85–0.97 per domain).

Conclusions

The final 18-item Dutch ED PREM showed good validity and reliability for measuring ED patient experience. It is suitable and feasible for use in clinical practice to assess and improve the person-centredness of emergency care.

WHAT IS ALREADY KNOWN ON THIS TOPIC

• Patient-reported experience measures (PREMs) are important tools for assessing the perceived quality of care from the patient’s perspective, for evaluating the person-centredness of care, and for improving the quality of health care.

• In many countries, including the Netherlands, there is no specific PREM for the emergency department (ED).

WHAT THIS STUDY ADDS

• By translating and cross-culturally validating the recently developed Australian ED PREM for the Dutch context, this study provides: (1) evidence for the cross-cultural validity of the ED PREM, (2) a usable ED PREM for the Dutch context and (3) a model for future cross-cultural validation of PREMs.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• The validation of existing PREMs contributes to international consistency in research and the building of cross-cultural knowledge.

• The ED PREM is ready for use in practice to inform quality improvement initiatives in healthcare.

Introduction

The emergency department (ED) is a unique and often high-pressure setting that can significantly shape a patient’s overall hospital experience.^1–3 As the starting point for many patient hospital journeys, the ED influences downstream experiences and outcomes. However, rising demand, long wait times, overcrowding and increasingly complex cases (partly driven by ageing populations) pose global challenges to care quality.^4–6 With strong evidence linking patient experience to clinical effectiveness and safety,^7–10 enhancing patient experience in the ED is a critical priority.

The measurement and evaluation of patient experience plays a key role in stimulating healthcare improvement initiatives.^10–12 Patient-Reported Experience Measures (PREM) capture what happened to patients while receiving care, and how it happened.^{13 14} This allows the patient’s perspective to be included in quality assessments, supporting the delivery of person-centred care.^{10 15} PREMs differ from Patient-Reported Outcome Measures (PROMs), which capture a range of health and well-being dimensions such as symptom severity and pain.¹⁶ PREMs have become essential to monitor, evaluate and improve healthcare services and are gaining recognition as indicators of healthcare quality.^{11 12 17} Although many PREMs have been developed for different purposes, there is still a need for PREMs that demonstrate adequate validity and reliability, particularly in acute care settings.^{11 14 18}

A specific PREM for the ED is still lacking in many countries, including the Netherlands. However, the valid and reliable Australian ED PREM has recently been developed through qualitative research, literature reviews and consensus-based research with consumers of ED care.^{19 20} Thus, the aim of the current study was to translate and cross-culturally validate the Australian ED PREM for the Dutch context. Furthermore, as methodological studies regarding PREMs are scarce, our approach can serve as an example for future studies on cross-cultural validation of PREMs.

Methods

Study design

We translated and cross-culturally validated the Australian ED PREM²⁰ in a Dutch academic hospital between October 2022 and September 2023. The Australian ED PREM contains 26 items categorised into four domains for person-centred care.²⁰ The first domain, ‘person-centred relationships between patients and ED care providers’, includes 14 items relating to feeling respected and heard, receiving emotional support and having trust in care providers. The second domain, ‘receiving timely ED care’, includes five items concerning waiting times and progress. The third domain, ‘patient engagement in ED care’, includes four items about partnering with patients and shared decision-making. The fourth domain, ‘privacy and comfort in the ED environment’, includes three items relating to the structure of the ED.

This study was conducted in accordance with international guidelines (including COSMIN) and recommendations from literature, including several steps that are shown in figure 1^21–24 (online supplemental references 1–4).

Figure 1. Summary of methods for translation and cross-cultural validation of the ED PREM. ED PREM, Emergency Department Patient-Reported Experience Measure.

Translation and cross-cultural adaptation

In October 2022, the Australian ED PREM was translated into Dutch using a forward and backward translation process.^{21 24} First, the Australian ED PREM was independently translated into Dutch by two native Dutch speakers fluent in English, one of whom was naive to the study topic (forward translation). Discrepancies were discussed in a research team consensus meeting (RAMT, JtO, JJH, MLM and MEJLH) and a consensus version was produced (Dutch ED PREM prototype). The consensus version was back-translated by two independent bilingual native English speakers with excellent understanding of the Dutch language, with no prior knowledge of the study topic. A final back-translated version was then produced in the consensus meeting and approved by the original author of the ED PREM (CB).

Establishing face and content validity

The Dutch ED PREM prototype was discussed with ED patients during cognitive interviews between December 2022 and January 2023 to confirm its comprehensibility (ie, easy to understand), comprehensiveness (ie, contains complete details) and relevance (ie, contains appropriate and useful information).²³ All adult patients (≥18 years) visiting the ED during the study period were screened for eligibility. Patients could participate if they had sufficient Dutch language proficiency and were mentally competent during the ED visit. Patients were excluded when visiting the acute coronary care unit and if the Dutch Sexual Assault Centre was involved. The recruitment process is described in more detail in online supplemental file 1.

Interviews were scheduled to take place within 2 weeks of ED discharge to minimise recall bias. Participants were asked to read and respond aloud to questions to clarify their thought process and comprehension. The interviewer used ‘probes’ to elicit detailed information (online supplemental references 5,6). Participants also evaluated the questionnaire as a whole. Online supplemental file 2 shows the semistructured interview guide. Interviews ceased when data saturation was reached. Interviews were audio recorded and transcribed verbatim. The transcripts were systematically analysed using ATLAS.ti. Consensus meetings among the research team occurred concurrently to patient interviews to discuss interim results and make adjustments where necessary. This resulted in the Dutch ED PREM pilot version.

Pilot study

Population and design

The pilot version of the ED PREM was administered to ED patients in paper and online formats. It then underwent psychometric assessment involving item reduction, structural validity, internal consistency and discriminant validity. All adult patients (≥18 years) visiting the ED during the study period (September 2023) were screened for eligibility; the same inclusion and exclusion criteria from the cognitive interview study were applied (described under ‘establishing face and content validity’). To create a representative and heterogeneous study population, a researcher was present in the ED between 8:00 and 22:00 hours for patient recruitment. Patients who visited the ED outside these hours were contacted later, either in the hospital ward or by telephone, to minimise selection bias. The recruitment process is described in more detail in online supplemental file 1.

In addition to the ED PREM pilot version, participants were asked to complete demographic questions about their education, employment situation, living situation and mode of arrival to the ED. Participant information regarding diagnosis, attending specialty, time spent in the ED and vital signs was collected through electronic health records. Sample size calculation followed subject-to-item ratio guidelines, recommending 7–10 subjects per item,^{23 24} resulting in an ideal sample size of 260.

Psychometric analysis

Descriptive statistics were used to calculate response rates and describe participant characteristics.

Means and SDs or medians and IQRs were calculated for continuous characteristics, and frequencies and percentages for categorical characteristics. Floor and ceiling effects, that is, responses at the lower/upper end of the scale, were assessed.²⁵ One sample Kolmogorov-Smirnov scores (D) were calculated to assess deviation from normal distribution. ‘Don’t know’ and ‘not applicable’ responses— which were included as item response options after participant interviews—were regarded as missing data. Missing data were assessed and Little’s Test of Missingness was performed to determine whether data were missing completely at random. Analyses were performed using IBM SPSS Statistics (V.29). After excluding participants with >15% missing data, missing data were imputed using multiple imputation by chained equations (MICE). MICE and further psychometric analyses were performed with R Statistical Software (V.4.1.3, 2022).

An iterative process of item reduction analysis, as well as validity and reliability testing, subsequently occurred (table 1). First, we examined correlation matrices, including inter-item correlations (IIC), item-total correlations (ITCs), missing data and ceiling effects, to remove items that did not contribute to the overall ED PREM (item reduction). Second, confirmatory factor analysis (CFA) was performed to test how well the structure of the Dutch ED PREM fit the predetermined factorial structure established by the Australian ED PREM. Third, internal consistency reliability was assessed using Cronbach’s alphas, to evaluate the degree to which the items within the ED PREM as a whole, and individual domains, measure the same construct. Last, discriminant validity was assessed according to known-groups. We hypothesised that younger patients, female patients and patients attending the ED alone would report worse experiences. We also hypothesised that patients with waiting time and longer lengths of stay would report worse experiences¹⁷ (online supplemental references 7–13). Table 1 provides further details on these stages of analysis.^{23 26–28}

Table 1. Dutch ED PREM item reduction, validity and reliability analyses.

Analysis	Explanation of analytical value
Item reduction analysis	Item reduction was performed to remove items that do not contribute to the overall reliability and validity of the questionnaire. Three types of item reduction analysis were performed: Correlations: Inter-item correlations (IICs) <0.3 suggest low strength of correlation between individual items.26 Item-total correlations (ITCs) <0.3 suggest poor correlation between the individual item and the rest of the Dutch ED PREM.26 Items with these low IIC/ITC scores are removed. Discriminating properties: Items with low discriminating properties (eg, with very high ceiling effects >85%) were removed. Missing data: High levels of missing data (>30%) per item suggest that the questions are only applicable to a subgroup of the participant population.26 Items with >30% missing data are removed.
Confirmatory factor analysis (CFA)23 26 27	CFA was performed to confirm the original four-factor structure of the Australian ED PREM and thereby establish structural validity. Several well-known fit indices were reported to describe the fit of the model: Root mean square error of approximation: good ≤0.06, acceptable 0.06–0.10, unacceptable >0.10 Standardised root mean square Residual: good ≤0.08, unacceptable >0.08 χ2 p value: good >0.05, unacceptable <0.05 Tucker-Lewis Index: good ≥0.95, acceptable 0.90–0.94, unacceptable <0.90 Comparative Fit Index: ≥0.95, acceptable 0.80–0.94, unacceptable <0.80
Internal consistency reliability	Internal consistency reliability was calculated using overall Cronbach’s alpha and individual factor Cronbach’s alpha scores.28 Alpha values <0.7 suggest poor inter-relatedness.
Discriminant validity	Mean factor scores of different known groups were compared using Mann-Whitney U tests. Significant differences (p<0.05) between these known groups indicate adequate known-groups validity.

ED PREM, Emergency Department Patient-Reported Experience Measure.

Results

Translation and cross-cultural adaptation

After back translation, minor wording adjustments were made to all but one item. Only item 6 of the Australian ED PREM (phrasing: ED care providers supported my decision to present to the ED) was translated differently. In the Netherlands, most patients do not decide to present to the ED but are referred. Therefore, the item was translated as ‘ED care providers believed that I indeed needed to be in the ED for my health complaints’ to reflect Dutch culture.

Establishing face validity and content validity

The Dutch ED PREM prototype was evaluated for face validity through 15 face-to-face cognitive interviews with ED patients. Interviews lasted between 20 and 90 min, and in three, the participant’s caregiver was present and contributed to the interview. The average age of participants was 65 years, and seven were female. Individuals were referred by varying medical specialties. Seven participants were admitted following their ED visit (47%).

Participants reported that the comprehensibility, comprehensiveness and relevance of the Dutch ED PREM prototype were high. Suggestions made by participants were subsequently discussed in the consensus meetings after content validity was established. Several adjustments were made, resulting in the Dutch ED PREM pilot version:

1. Addition of two additional response options: ‘not applicable’ and ‘don’t know’, because not all items were applicable to all participants’ experiences (ie, not needing to wait for care when presenting).

2. Addition of an open-text box at the end for general comments.

3. Changed from frequency-based response options (never to always) to agreement-based response options (strongly disagree to strongly agree).

4. Changed the order of some items to improve flow.

5. Combined two items about privacy and comfort from the Australian ED PREM into a single item to improve relevance: ‘ED care providers were considerate of my privacy’.

6. Created a new domain with three items about companions in the ED.

Online supplemental file 3, table S1 and figure S1 provide an overview of the translation, cognitive interviews and comprehension per item. The resultant pilot version of the Dutch ED PREM consisted of 28 items across 5 domains, which is shown in online supplemental file 4.

Pilot study

Participant characteristics

Of the 527 patients who gave informed consent, 357 (68%) completed the questionnaire. Figure 2 shows an overview of participant inclusion. No significant differences were found when comparing age and gender between patients approached, responders and non-responders.

Figure 2. Participant inclusion flow chart. Flow diagram of participant recruitment during ED PREM pilot study. ED PREM, Emergency Department Patient-Reported Experience Measure; IC, informed consent.

Participant and ED visit characteristics are shown in table 2. The median age was 62 years, and more than half of the participants were male (58%). The Dutch ED PREM was completed online by 280/357 (78%) participants. Participants who chose the postal option were older and more often male (69% vs 55%) and retired (75% vs 31%). Almost half were hospitalised after their ED visit (47%). More than 70% of participants completed the questionnaire within 5 days, with a median time to completion of 1 day.

Table 2. Study population characteristics.

Characteristics	Total population (n=357)
Participant characteristics
Age, median (IQR)	62.0 (24.0)
Male, n (%)	206 (57.7)
Employment situation, n (%)
Working	132 (37.0)
Studying	10 (2.8)
Retired	146 (40.9)
Sickness benefit act	44 (12.3)
Volunteering	13 (3.6)
Other	11 (3.1)
Unknown	1 (0.3)
Highest level of education, n (%)
No education or primary school not completed	4 (1.1)
Lower education (primary school)	12 (3.4)
Pre-vocational education	60 (16.8)
Secondary/upper vocational education	99 (27.7)
Higher general and preuniversity education	36 (10.1)
Higher professional education	83 (23.2)
Scientific education	50 (14.0)
Other	7 (2.0)
Unknown	6 (1.7)
Mother tongue, n (%)
Dutch	344 (96.4)
Other	12 (3.4)
Unknown	1 (0.3)
ED visit characteristics
Method of transport to ED, n (%)
Own transport	214 (59.9)
Ambulance	119 (33.3)
Other	23 (6.4)
Unknown	1 (0.3)
Urgency of ED visit*, n (%)
U1	2 (0.6)
U2	75 (21.0)
U3	200 (56.0)
U4	59 (16.5)
Unknown	21 (5.9)
Specialty of ED visit, n (%)
Internal medicine	123 (34.5)
Emergency medicine	120 (33.6)
General surgery	39 (10.9)
Urology	11 (3.1)
Neurology	27 (7.6)
Other	37 (10.4)
Hospitalisation following ED visit, n (%)	168 (47.1)
Time to complete questionnaire, n (%)
0–5 days	275 (77.0)
6–10 days	52 (14.6)
10+ days	25 (7.0)
Unknown	5 (1.4)
Median time to complete questionnaire (IQR)	1 (5)

^*According to the Manchester Triage System.

ED, emergency department.

Pilot version Dutch ED PREM item characteristics

Prior to imputation, mean and median item scores were calculated for ED PREM items, showing overall high scores. Item responses were non-normally distributed, with high percentages of ceiling responses. Missing data ranged between 0.3% and 39.5%. Online supplemental table S2 shows an overview of all descriptive statistics per item before imputation.

Item reduction analysis

Prior to imputation, 14 participants were excluded because of >15% missing items. After imputation, ITCs for the Dutch ED PREM ranged between 0.43 and 0.91 (online supplemental table S3). Item 15 and item 16 had IIC <0.3 and were subsequently deleted. Five items (1, 5, 6, 13 and 14) had ceiling effects >85% and were deleted prior to factor analysis.

CFA and internal consistency reliability

Items 26, 27 and 28 (Domain 5: Presence of companion(s) in the ED) were not included in CFA as they were not part of the original four-factor structure. Thus, CFA was performed on an 18-item, four-factor version of the Dutch ED PREM matching the original ED PREM construct (online supplemental figure S2 summarises all steps on item level). Table 3 shows the factor structure, which had acceptable model fit (χ²(df) 7519.55 (153), p<0.001, Root Mean Square Error of Approximation=0.090, Tucker-Lewis Index=0.942, Comparative Fit Index=0.951, Standardised Root Mean Square Residual=0.035). Cronbach’s alpha for the overall Dutch ED PREM was 0.97; values for individual factors ranged between 0.85 and 0.97.

Table 3. Factor structure 18-item Dutch ED PREM with Cronbach’s alpha.

ED PREM items	Factor 1	Factor 2	Factor 3	Factor 4
Cronbach’s alpha	α	α	α	α
Factor 1: Person centred relationships between patients and ED care providers (2) ED care providers treated me with respect. (3) ED care providers did not make me feel like I was trouble to them. (4) ED care providers treated me like a person, not a number. (7) ED care providers believed that I needed to be in the ED for my health complaints. (8) ED care providers were compassionate. (9) I had trust in ED care providers. (10) ED care providers took good care of me. (11) ED care providers were reassuring. (12) ED care providers listened to me.	0.97
Factor 2: Receiving timely ED care (17) ED care providers updated me throughout my ED journey about the progress of my visit. (18) I received timely care, appropriate to the severity of my health complaints. (19) The process in the ED was fast enough, appropriate to the severity of my health complaints.		0.91
Factor 3: Patient engagement in ED care (20) ED care providers informed me of my treatment options. (21) ED care providers encouraged me to ask questions/ (22) ED care providers involved me as much as I wanted in decisions about my treatment. (23) ED care providers consulted with me about my care.			0.93
Factor 4: Privacy and comfort in the ED environment (24) ED care providers were considerate of my privacy. (25) I felt comfortable in the ED environment.				0.85

ED PREM, Emergency Department Patient-Reported Experience Measure.

Discriminant validity

Online supplemental table S4 demonstrates the discriminant validity of the ED PREM. In line with the hypotheses, participants with longer lengths of stay and/or waiting times reported statistically significant worse experiences for factor 2: Receiving timely ED care (p=0.04 and p<0.001, respectively). Similarly, participants with longer waiting time also reported slightly worse experiences for factor 3: patient engagement in ED care (p=0.04).

Discussion

This study describes the cross-cultural adaptation and psychometric evaluation of the 18-item Dutch ED PREM. We established face, content and structural validity, along with internal consistency reliability. There was also evidence of discriminant validity, despite low variability in participants’ responses. Ultimately, the high response rate demonstrates that the Dutch ED PREM is a user-friendly and feasible tool for measuring patient experiences in the ED. The cross-cultural validation performed in this study reinforces the validity of the original ED PREM and encourages the performance of this comprehensive methodology to validate the ED PREM in more settings.

Cross-cultural adaptation and validation is essential to ensure reliable and valid context-specific measures for effective research and quality patient care.^{21 24} Our cross-cultural adaptation resulted in several adjustments. Notably, the Likert scale was changed from a frequency-based to an agreement-based scale. While this was the best choice from the participants’ perspective, this type of Likert scale is more prone to ceiling effects (online supplemental reference 14). Furthermore, the response options ‘not applicable’ and ‘don’t know’ were introduced, leading to data missingness. However, omitting these options would force irrelevant answers, leading to biased responses. Finally, three questions about companions in the ED were added. Although they may eventually be a good addition to the Dutch ED PREM, they were not psychometrically evaluated in this study as they were not part of the original ED PREM factor structure. An important consideration in cross-cultural adaptation is the balance between maintaining adherence to the original instrument and ensuring contextual relevance in the target setting. In our adaptation, some items were modified or removed based on cognitive interview findings and cultural considerations specific to the Dutch ED context. While this may limit strict item-level comparability for some items, the core constructs were retained and validated, allowing for a meaningful comparison at the domain level. There are other patient-reported measures in the Netherlands, for example, the combined PREM/PROM developed by Kremers et al.²⁹ However, due to its blended nature, it is not specifically a PREM. The US Emergency Department Patient Experience of Care (EDPEC) questionnaire measures comparable domains of person-centredness of care,³⁰ though is limited to patients discharged home after ED care. We believe that the Dutch ED PREM is the most comprehensive, reliable and valid measure of patient experience in the ED to date.

The Dutch ED PREM is a valid and reliable instrument ready for use in clinical practice. For effective implementation, it is recommended to educate both staff and patients about the importance of monitoring patient experience and to integrate the PREM into routine clinical workflows. Regular evaluation is essential to identify opportunities for improvement and evaluate these efforts. Such data can also be used for benchmarking and learning from other hospitals. Future research should focus on validating the instrument in different contexts. To use the Dutch ED PREM in other countries, languages or cultures, we recommend following our methodology. Adapting the Likert scale might increase the sensitivity of the instrument and reduce the occurrence of ceiling effects (online supplemental references 14, 16).

Strengths and limitations

This study has several strengths. First, a thorough methodology was used, based on international literature, best practice guidelines and previous work. In addition, end-users were involved at several stages of the research, which is important for person-centred care. Second, this study cross-culturally adapted and validated an existing and robust ED PREM, rather than developing a completely new one, thereby contributing to international consistency and building cross-cultural knowledge. Third, for our pilot sample, we recruited patients around the clock and achieved a high response rate, ensuring a representative sample of the ED population. Fourth, we used cognitive interviewing to ensure the usefulness, comprehensibility, comprehensiveness, feasibility and relevance of the new measure.

The first limitation of this study is the low variability of responses and the overall high item scores, resulting in a high percentage of ceiling effects, which posed challenges to our psychometric analyses. This indicates limited ability to detect variation at the positive end of the spectrum.²⁰ Thus, we removed items from the Dutch ED PREM because: (1) items with >85% identical responses had limited discriminative value; (2) redundancy in the original instrument, indicated by high Cronbach’s alpha, suggested some items could be dropped and (3) a shorter instrument improves feasibility. Second, this was a single-centre study, which may limit generalisability. However, the large sample size and high response rate (68%) suggest that our findings are generalisable, which we confirmed by comparing the age and gender between patients approached, non-responders and responders. In addition, we compared our sample with a random sample of ED patients in another month. For privacy reasons, we could only compare age and gender. Third, the introduction of ‘not applicable’/‘don't know’ response options resulted in missing data. However, these options were included in response to the cognitive interview results. Moreover, treating these responses as missing has previously shown minimal impact on the final outcomes (online supplemental reference 17). Fourth, the CFA of the Dutch ED-PREM model showed a good or acceptable fit with the Australian four-factor model on all indices except the χ² test. However, the overall fit was still acceptable as the value of the chi-square has been the subject of debate (online supplemental references 18).

Conclusions

This study shows that the Dutch ED PREM is a valid and reliable instrument for measuring patient experience in the ED. The results align with the Australian version, indicating cross-cultural consistency. Future research should aim to replicate our findings across different hospitals and countries, contributing to increased international adoption of ED-specific PREMs with robust methodological quality.

Data availability statement

Data are available on reasonable request.