The Consumer Quality Index in an accident and emergency department: internal consistency, validity and discriminative capacity

Abstract

Background

Patients’ experiences are an indicator of health‐care performance in the accident and emergency department (A&E). The Consumer Quality Index for the Accident and Emergency department (CQI A&E), a questionnaire to assess the quality of care as experienced by patients, was investigated. The internal consistency, construct validity and discriminative capacity of the questionnaire were examined.

Methods

In the Netherlands, twenty‐one A&Es participated in a cross‐sectional survey, covering 4883 patients. The questionnaire consisted of 78 questions. Principal components analysis determined underlying domains. Internal consistency was determined by Cronbach's alpha coefficients, construct validity by Pearson's correlation coefficients and the discriminative capacity by intraclass correlation coefficients and reliability of A&E‐level mean scores (G‐coefficient).

Results

Seven quality domains emerged from the principal components analysis: information before treatment, timeliness, attitude of health‐care professionals, professionalism of received care, information during treatment, environment and facilities, and discharge management. Domains were internally consistent (range: 0.67–0.84). Five domains and the ‘global quality rating’ had the capacity to discriminate among A&Es (significant intraclass correlation coefficient). Four domains and the ‘global quality rating’ were close to or above the threshold for reliably demonstrating differences among A&Es. The patients’ experiences score on the domain timeliness showed the largest range between the worst‐ and best‐performing A&E.

Conclusions

The CQI A&E is a validated survey to measure health‐care performance in the A&E from patients’ perspective. Five domains regarding quality of care aspects and the ‘global quality rating’ had the capacity to discriminate among A&Es.

Introduction

Evidence suggests that quality of care varies across accident and emergency departments (A&Es) in the Netherlands.1 A recent report indicated that not all A&Es meet national standards as defined by the Healthcare Inspectorate. These standards measure health‐care performance from the professional point of view. Measurement of the patient's perspective of the quality of care has not yet been included in these standards despite the fact that patients’ experiences are an important indicator of health‐care performance.2, 3 The merit of evaluating health‐care performance from the patients’ perspective in the A&E has been acknowledged.4 Identifying and responding to patients’ needs improve the quality of emergency care. For instance, patients with a positive experience have a reduced tendency to seek further options for treatment, show more compliance with guidelines, and report less problems.5, 6 Besides, acting upon evaluations from the patient's perspective balances improvements of clinical care from the professional's perspective. Therefore, a standardized and validated tool to measure patients’ experiences is needed.

In the USA, Canada and European countries such as England and most Scandinavian countries, a national survey programme for measuring patients’ experiences is performed.7, 8 Inspired by the American Consumer Assessment of Health care Providers and Systems (CAHPS) and the Dutch QUality Of care Through the patients’ Eyes (QUOTE), a family of surveys to measure patients’ experiences, known as the Consumer Quality Indices (CQIs), were introduced in the Netherlands.9, 10 CQIs are available for many community services, care settings and condition‐specific patients’ groups.11 The questionnaires aim to measure health‐care performance as experienced by patients. Strengths of the methodology are standardized analysis and reports of outcomes to provide performance information for several parties such as individual consumers, patient/consumer organisations, health insurers, health‐care providers, the Healthcare Inspectorate and the Ministry of Health.12

Recently, the Consumer Quality Index for the Accident and Emergency department (CQI A&E) was developed and psychometrically tested in a pilot study. Exploratory factor analysis determined five domains with sufficient reliability to be provisionally accepted.13 These preliminary results need to be confirmed and validated. The purpose of this study was to test the internal consistency, the validity and the discriminative capacity of the CQI A&E in a multicentre study design. The following three aspects of the discriminative capacity were explored: (i) detection of significant differences among health‐care providers; (ii) feasibility of sample sizes to obtain reliability; and (iii) meaningfulness of differences for users.14 Moreover, the need for case‐mix adjustment was investigated.15, 16

Methods

Data collection

To recruit A&Es for this study, an announcement of the study was made in an online national medical newsletter at the end of 2009. Following the CQI guidelines, we aimed at including 20 A&Es. Twenty‐one of the hundred A&E departments in the Netherlands decided to participate. Annual A&E patients’ numbers ranged from 8000 to 50 000, representing small, medium and large A&Es in the Netherlands. The A&Es varied in terms of patient throughput, geographical area (urban or rural, and regions), trauma centre or non‐trauma centre, and teaching or non‐teaching status. In the samples, 600–800 patients per A&E were randomly selected out of all patients attending the A&Es during three subsequent weeks. Patients with a known postal address and no reported death were eligible.

The 78‐item CQI A&E was sent by postal mail within 1 month of A&E attendance. Up to three reminders were sent to non‐respondents after 1, 4 and 6 weeks. The recipients could return the questionnaire in a postage‐paid envelope. Descriptive data of the patients were provided by the hospital registration systems. The study protocol was approved by the Medical Ethical Committee of the University Medical Center Utrecht.

CQI A&E questionnaire

The conceptual basis and the design of the CQI A&E, developed according to the CQI guidelines, were described in a technical report and accorded by the CQI scientific advisory board.13, 17 The development and first use have been described in detail in a previous paper.18 In summary, the content validity was ascertained by a literature review, in‐depth interviews with A&E experts and patient focus groups, resulting in a draft questionnaire. The draft questionnaire was cognitively tested on A&E patients and adapted where necessary. Subsequently, a pilot test was performed to assess the psychometric validation of the CQI A&E. The principal components analysis (PCA) to locate underlying domains revealed a distinct five‐factor solution. Preliminary internally consistent domains were as follows: attitude of the health‐care professionals, information and explanation, environment of the A&E, leaving the A&E and rapidity of care. The content of the CQI A&E and examples of questions are presented in Box 1.

Box 1. Content of the CQI A&E.

Categories	45 experience questions; for instance
1. General (3 items)	Was the signposting to the A&E of the hospital a problem?
2. Before arrival in the A&E (11 items)	‐A big problem
3. Reception desk A&E (4 items)	‐A small problem
4. Health professionals in the A&E (8 items)	‐No problem
5. Pain (3 items)	Was the reception staff member polite to you?
6. Examination and treatment (16 items)	‐No, not at all
7. Leaving the A&E (11 items)	‐A bit
8. General A&E (11 items)	‐A great deal
9. About you (11 items)	‐Yes, completely

Data analysis

Descriptive statistics were used to summarize the characteristics of the respondents, such as gender, age and triage code. A&Es prioritized patients to treatment by one of two triage systems: the Manchester Triage System (MTS) or the Emergency Severity Index (ESI). Both systems use five categories.19 Patients of corresponding urgency categories were combined for analysis. Excluded from analyses were questionnaires filled in by respondents aged 11 and younger, answered by someone else other than the respondent or with more than 50% missing answers.

Internal consistency

A PCA was undertaken to optimize the 5‐factor solution out of the pilot study. Firstly, data were analysed to identify item response rates and frequency distributions. Questionnaire items were excluded from further analysis where they had an item non‐response of >10%, extreme skew of >90% of responses in the same category (i.e. a ceiling or floor effect) or item correlation, determined by Spearman's correlation coefficient, above 0.70. Secondly, several criteria needed to be fulfilled in the PCA: (i) the Kaiser–Meyer–Olkin Measure of Adequacy (KMO), a measure of sampling adequacy (threshold: KMO>0.60), (ii) Bartlett's test of sphericity to test the null hypothesis that variables in the population correlation matrix were uncorrelated (threshold: P < 0.05) and (iii) the Eigenvalue represented the amount of the total variance explained by the factor (threshold: Eigenvalue >1, also known as the Kaiser criterion). The PCA was performed with oblique rotation.20 In subsequent steps, factor loading (threshold: factor load > 0.40) and Cronbach's alpha coefficient (α), a measure of internal consistency to estimate the reliability of the reported factors, were calculated. Cronbach's alpha coefficients above 0.70 were considered reliable. The alpha of the total factor should not increase by deleting one of the items. Item‐total correlation (ITC) had to be higher than 0.40. Interfactor correlations were calculated to estimate the overlap among domains (threshold: r > 0.70). After the PCA, the interpretability of the domains for daily practice was evaluated. To enhance interpretability, domains with multiple and dissimilar quality aspects could be broken up into smaller domains, while safeguarding the reliability of the domains.

Domain scores

Patients responded to items on ordinal 2‐, 3‐ or 4‐point Likert scales. For the computation of a domain score, response categories of items that constituted to the domain were recoded from 1 to 4, summed up and divided by the number of items that constituted the domain (i.e. no/big problem/never = 1, sometimes = 2, bit of a problem = 2.5, usually = 3, yes/not a problem/always = 4).21 Items with negative wording were reversed to ensure comparability in the analysis.

Case‐mix adjustment

The need for case‐mix adjustment was investigated using linear mixed effect models. Mixed effect models account for the hierarchical structure of the data: patients within A&Es. The models decomposed variance into that attributable to A&Es and that attributable to other sources, such as individual differences. The methodology has been described and examined in various studies.14, 22 For each domain score, a separate (empty) model was analysed, with the domain score as dependent variable and a random intercept for each A&E department. Patients’ characteristics such as gender, age and triage code as reported in the hospital registry systems, and characteristics such as health status, education and ethnicity as determined out of questions in the questionnaire were added as fixed effects. Only those variables that significantly (P < 0.05) contributed to the (full) model were retained in the final model. In addition, a separate model with the global quality rating of the health‐care performance as dependant variable was analysed. Significance was determined by likelihood ratio tests (P < 0.05).

The impact of case‐mix adjustment on the total variance was assessed by calculating the proportional change in variance (PCV). The proportional change in variance is an estimate to assess the amount of variance in the empty model (V₀) attributable to differences in patients’ characteristics (case‐mix). The PCV was calculated according to Merlo et al. (V₀– variance final model/V₀).23 Total variance is comprised of three components: variance among A&Es, variance due to patients’ characteristics and remaining variance within A&Es (or residual variance).

Discriminative capacity

The discriminative capacity follows three criteria. Firstly, the intraclass correlation coefficient (ICC) was calculated. The ICC expresses the discriminative power of the domains. The discriminative power is a general assessment of differences among health‐care providers; the variance attributable to providers can be tested for significance. The magnitude of the variance among providers may then be expressed as a proportion of the total variance on a scale from 0 to 1.14 Next, the calculations were repeated after adjusting the data for age (eight categories), gender and health status of the respondents (five categories). Low‐, average‐ and high‐performing A&Es were determined by the mean score of all A&Es and the 95% comparative confidence intervals (CI) of individual A&Es.24 An overlap of the CI with the mean domain score implied an average‐performing A&E. CIs without an overlap with the mean score were low or high performers. Mean scores and CIs were plotted in caterpillar plots (see Figure 1 and Appendix S1).

Figure 1.

Patients’ experience scores of the domain ‘timeliness’. Horizontal lines represent ranges of experience scores of A&Es. The vertical line in the middle of the figure represents the aggregated average patients’ experience score (3.39). A&Es performing significantly worse than average were plotted completely left of the vertical line (N, J and H). A&Es performing significantly better than average were plotted completely right of the vertical line (G, R and X).

Secondly, the A&E‐level reliability, given the current sample sizes, was calculated. The reliability expresses the proportion of variation in A&E‐level mean scores attributable to true variation among A&Es, and was estimated using generalizability theory.25, 26 The essence of generalizability theory is the recognition that in any measurement situation, there are multiple sources of error variance, due, for instance, to random sampling. The theory contains two stages. In the first stage, called G‐study, the variances are used to create G‐coefficients, extensions of classical reliability coefficients. G‐coefficients look at the proportion of total variance due to the object of measurement. In the final step, the variances derived from the G‐study are used to set the sample sizes to obtain a reliability of 0.7, 0.8 and 0.9. This is called a D‐study. The D‐study is the third criteria of the discriminative capacity, to determine whether differences among A&Es were detectable with a feasible sample size.

All analyses were performed using the statistical software spss 19.0 (IBM Corp., Poughkeepsie, NY, USA) and R 2.10.1 (R Foundation for Statistical Computing, Vienna, Austria).

Results

In total, 4883 (40%) patients responded. The number of respondents per hospital varied from 173 to 302. The mean age of the respondents was 52.8 (SD 20.5) years, and 49% was male. Non‐respondents were younger (mean age 45.6 years) and more likely to be male (54%) (Table 1).

Table 1.

Patients’ characteristics

	Respondents		Non‐respondents
	Mean	SD	Mean	SD
Age (years)a	52.8	20.5	44.6	23.1

	N	%	N	%
Gendera
Male	2392	49.0	4023	54.1
Female	2491	51.0	3418	45.9
Triage codea
Red	18	0.5	48	1.0
Orange	497	14.4	605	12.0
Yellow	1389	40.2	1749	34.7
Green	1453	42.0	2423	48.0
Blue	99	2.9	219	4.3
Missing	1427			2397
Health status
Excellent	547	11.4	N/A	N/A
Very good	828	17.2
Good	1887	39.3
Moderate	1242	25.8
Poor	302	6.3
Missing	77
Educational level
Low	1196	27.3	N/A	N/A
Medium	1640	37.5
High	1540	31.5
Missing	507
Country of birth
Dutch	4238	89.3	N/A	N/A
Non‐Dutch	506	10.7
Missing	139

N/A, not applicable. Health status, educational level and country of birth were assessed out of completed questionnaires.

^aSignificant difference between respondents and non‐respondents (P < 0.05).

Two‐third of the respondents rated their health status as good, very good or excellent. Respondents were equally divided over three educational levels; the vast majority was born in the Netherlands (89%). The triage code prioritized patients to treatment by the severity of the patients’ symptoms. Most patients were triaged in the yellow (40%) or green (42%) category and according to the triage systems had to be treated within 1 or 2 h, respectively.

Domains

The PCA resulted in a five‐factor solution comprising 26 items (explained variance = 51%, KMO = 0.934, Bartlett's test P < 0.001, N = 4883). Domains were largely comparable with prior analysis in the pilot study. The first domain ‘attitude of health‐care professionals’ comprised nine items, related to multiple quality aspects. Therefore, this domain was broken up into three domains to enhance interpretability and feasibility. The three domains were labelled: (i) attitude of health‐care professionals, (ii) timeliness and (iii) professionalism of received care. Each new domain measures one care aspect. For the face validity and the reliability, we decided to move Q31 ‘received care as quickly as desired’ out of the domain ‘information before treatment’ into ‘timeliness’, and Q61 ‘feeling safe in the A&E’ was moved from the domain ‘attitude of health‐care professionals’ into ‘environment and facilities’. Additionally, four extra items (Q18, Q24, Q64 and Q57) enhanced the reliability of the newly constituted seven domains. The domains were labelled: (i) information before treatment (α = 0.667), (ii) timeliness (α = 0.834), (iii) attitude of health‐care professionals (α = 0.839), (iv) professionalism of received care (α = 0.714), (v) information during treatment (α = 0.764), (vi) environment and facilities (α = 0.723) and (vii) discharge management (α = 0.788). Item‐total correlations were above 0.40 for all items, and the reliability of the domains did not increase if an item was deleted out of the domain (Table 2).

Table 2.

Domains with accompanying items

Quality aspect (α)		ITC	α if item deleted
Information before treatment (α = 0.667, n = 1301)
Q18	Patient's health‐care expectations	0.45	0.61
Q21	Information on the rapidity of the treatment based on acuteness of the health problem	0.51	0.53
Q22	Information on the order of the treatment	0.48	0.57
Timeliness (α = 0.834, n = 1960)
Q24	Total waiting time before treatment	0.68	0.78
Q31	Received care as quickly as desired	0.69	0.78
Q32	Patient's health‐care needs	0.62	0.81
Q64	Total time spent in the A&E	0.67	0.78
Attitude of health‐care professionals (α = 0.839, n = 4728)
Q38	Health‐care professionals listened attentively to patients	0.75	0.74
Q39	Health‐care professionals took time for patients	0.68	0.82
Q40	Taken seriously by health‐care professionals	0.72	0.78
Professionalism of received care (α = 0.714, n = 3883)
Q43	Co‐operation among health‐care professionals	0.57	N/A
Q44	Trust in the competence of health‐care professionals	0.57	N/A
Information during treatment (α = 0.764; n = 2720)
Q33	Information on treatment	0.57	0.71
Q35	Clarity of explanation of results of examinations	0.59	0.70
Q41	Clarity of explanation of health problem	0.66	0.66
Q62	Information towards attendants	0.46	0.76
Environment and facilities (α = 0.723, n = 2499)
Q57	Pleasant atmosphere in the waiting room	0.57	0.64
Q58	Availability of refreshments	0.42	0.73
Q59	Hygiene in the A&E	0.53	0.66
Q60	Quiet environment	0.51	0.67
Q61	Felt safe in the A&E	0.49	0.69
Discharge management (α = 0.788, n = 261)
Q48	Explanation about new medication	0.49	0.77
Q49	Information on side‐effects of the medication	0.56	0.76
Q50	Information on resumption of daily activities	0.69	0.70
Q51	Information on danger signs to watch out for after leaving the A&E	0.69	0.71
Q56	Explanation about how to make an appointment for outpatient care	0.46	0.78

α, Cronbach's alpha coefficient; ITC, item‐total correlation.

Correlations between domains are presented in Table 3. Of interest were correlations exceeding the threshold of 0.7, which indicate an overlap between domains. The third (‘attitude of health‐care professionals’) and the fourth (‘professionalism of received care’) domains partly measured the same aspect of health‐care performance in the A&E (r = 0.722).

Table 3.

Correlation coefficients of domains

No.	Quality aspect	2	3	4	5	6	7
1	Information before treatment	0.379	0.339	0.346	0.382	0.342	0.387
2	Timeliness		0.595	0.589	0.501	0.522	0.374
3	Attitude of health‐care professionals			0.724	0.657	0.535	0.475
4	Professionalism of received care				0.611	0.522	0.467
5	Information during treatment					0.477	0.579
6	Environment and facilities						0.373
7	Information at discharge

Case‐mix adjustment

Age, gender and health status were statistically significant predictors in linear mixed effect models for all domains and the global quality rating. In other words, it is necessary to adjust patients’ experience scores for age, gender and health status to make a fairer comparison among A&Es. Educational level, country of birth and triage code were significant predictors for some (not all) domains and therefore not added to the adjusted models. The effect of case‐mix adjustment as estimated by the PCV was comparable for the seven domains (range: 3.1–5.8% of the total variance) and slightly higher for the global quality rating: 6.9%.

ICCs were slightly higher in empty models (range: 0.0063–0.0354) than ICCs in adjusted models (range: 0.0038–0.0327). The effect of adjustment was the largest for the domain `attitude of health‐care professionals′ and the smallest for the domain ′discharge management′. Experiences regarding interpersonal relations with professionals appeared to be more influenced by patients’ characteristics than was, for instance, discharge management. Overall, patients’ characteristics explained only a very small part of the total variance in their experiences.

Discriminative capacity

Five domains regarding quality of care aspects and the ‘global quality rating’ had the capacity to discriminate among A&Es. The domains ‘information before treatment’ and ′discharge management’ did not demonstrate a discriminative capacity among A&Es (no significant ICCs) (Table 4). The reliability (G‐coefficient) of the mean value of patients’ experience scores given actual sample sizes of A&Es (Table 4) was used to set the sample sizes to obtain a reliability of 0.7, 0.8 or 0.9. We found that two domains, ‘timeliness’ and ‘environment and facilities’, and the ‘global quality rating’ were reliable (G‐coefficient>0.7) for the given numbers of respondents. Sample sizes of the two domains ‘attitude of health‐care professionals’ and ‘professionalism of received care’ were close to the reliability threshold, with G‐coefficients of, respectively, 0.62 and 0.67. Sample sizes of, respectively, 335 and 226 were required to obtain reliability of 0.7. Sample sizes of three domains were insufficient for reliable measurements of differences among A&Es. The domain ‘information during treatment’ had an average response number of 228, whereas a sample size of 488 respondents was required for a more reliable estimate. For the domain ‘information before treatment’ and the domain ‘discharge management’, respectively, 301 and 473 were required to obtain sufficient reliability.

Table 4.

Linear mixed effect models for the domains of the CQI A&E

Quality aspect (α)	Empty model			PCV (%)	Adjusted modela			Mean valid response per A&E	Reliability (G‐coefficient)b	No. of respondents needed for reliability of
	Variance A&E	Variance patients	ICC		Variance A&E	Variance patients	ICC			0.7	0.8	0.9
Information before treatment (α = 0.667)	0.0091	0.8149	0.0111	5.27	0.0053	0.7754	0.0067	79	0.38	301	516	1160
Timeliness (α = 0.753)	0.0158	0.6175	0.0249	5.01	0.0133	0.5883	0.0221	213	0.82	109	186	419
Attitude of health‐care professionals (α = 0.839)	0.0032	0.3505	0.0259	4.07	0.0023	0.3370	0.0155	230	0.62	335	574	1292
Professionalism of received care (α = 0.842)	0.0065	0.3651	0.0116	3.06	0.0036	0.3485	0.0102	185	0.67	226	387	870
Information during treatment (α = 0.764)	0.0029	0.4522	0.0063	3.56	0.0020	0.4369	0.0046	226	0.52	488	836	1882
Environment and facilities (α = 0.723)	0.0128	0.3499	0.0354	4.64	0.0113	0.3347	0.0327	227	0.89	68	117	263
Discharge management (α = 0.788)	0.0034	0.7196	0.0047	4.91	0.0026	0.6848	0.0038	69	0.25	473	811	1825
Global quality rating	0.0510	2.6927	0.0186	6.91	0.0445	2.5095	0.0174	229	0.80	131	225	505

ICC, intraclass correlation coefficient: ICCs in bold are significant (P < 0.05); PCV, proportional change of variance.

^aAdjusted for age, gender and health status.

^bThe G‐coefficient indicates the reliability of the measurement at an A&E, given the actual sample size; G‐coefficients in bold are above the threshold of 0.7.

Table 5 shows numbers of low‐, average‐ and high‐performing A&Es and means of patients’ experience scores. Scores on the domains ‘information before treatment’ and ‘discharge management’ were the lowest experience scores. High scores were found for ‘attitude of health‐care professionals’, ‘professionalism of received care’ and ‘information provided during treatment’. A&E scores were plotted in caterpillar plots. Figure 1 shows patients’ experience scores on the timeliness domain, and other caterpillar plots are shown in an online only Appendix S1, Figures 1–7.

Table 5.

A&E quality performance and patients’ experience domain scores

Quality aspect	Quality of care (N)			Patients’ experiencesa
	Low	Average	High	Mean	Minimum	Maximum	Range
Information before treatment	0	21	0	2.08	2.00	2.17	0.17
Timeliness	3	15	3	3.39	3.21	3.56	0.35
Attitude of health‐care professionals	1	18	2	3.59	3.52	3.66	0.14
Professionalism of received care	2	17	2	3.48	3.41	3.60	0.19
Information during treatment	0	20	1	3.47	3.43	3.55	0.12
Environment and facilities	6	11	4	3.31	3.19	3.53	0.34
Discharge management	0	21	0	3.02	3.02	3.02	0.00
Global quality rating	2	15	4	7.65	7.38	8.07	0.69

Numbers of low, average and high performing A&Es.

^aPatients’ experiences domain scores adjusted for age, gender and health status.

Discussion

In this study, the construct validity of the CQI A&E was investigated. Furthermore, we studied the discriminative capacity of the CQI A&E. The questionnaire measured seven quality aspects of health‐care performance in the A&E, which were labelled: information before treatment, timeliness, attitude of health‐care professionals, professionalism of received care, information during treatment, environment and facilities, and discharge management.

The CQI A&E can be used for monitoring the quality of care in the A&E from the patients’ perspective. As mentioned in the introduction, a national report introduced three standards from the professional perspective to assess quality of care. Standards relate specifically to the quality management system, the availability and competence of medical staff, and the time within which any necessary airway management interventions are implemented.1 We propose adding the systematic measurement of patients’ experiences as a standard to monitor and improve health‐care performance among and within A&Es. We follow Cameron, Schull and Cooke, who mentioned patient‐centredness, accessed by measuring patients’ experiences, as a key element in a framework for quality measurement in the A&E.27

The questionnaire provides information for several stakeholders in emergency medicine. Individual questions provide tailor‐made information to pinpoint problems on a local level. Domain scores enhance clarity, comprehensibility and reliability of the data and are more informative for surveillance and benchmarks among A&Es. Thus, measuring patients’ experiences enhances transparency and enables benchmarks.

One large factor of the PCA was broken down into three separate domains. These three distinct domains seem easier to interpret and are, in our opinion, more informative and more specific than the ‘original’ domain with multiple quality aspects. This decision is beneficial for the face validity of the questionnaire. We accepted the domain ‘information before treatment’ although the Cronbach's alpha was below the threshold of 0.7.

To determine the discriminative capacity of the CQI A&E, several elements have to be discussed. Firstly, the PCV was calculated to clarify the effect of case‐mix adjustment. Proportional variances ranged from 3% for professionalism of received care to 7% for global quality rating. Influences of patients’ age, gender and health status were marginal. Therefore, we deemed extending models with variables such as educational level, country of birth and triage code, which were not significant predictors for all models, unnecessary. This is consistent with previous findings that factors contributing to the variability in patients’ experiences were patient's age and health status, and hospital factors were of less importance.16

Secondly, ICCs expressed differences among A&Es. The ICCs of five domains were capable to demonstrate differences in health‐care performance among A&Es as experienced by patients. These differences showed that there is room for improvement on these domains, and A&Es can learn from best practices. The domain with the highest discriminative capacity was ‘environment and facilities’. A maximum of 3.3% of the total variance was attributable to the difference among A&Es. However, the largest part of the variance among A&Es remained statistically unexplained. ICCs and PCVs are in line with comparable studies.14, 23, 28 Future study should elaborate on the relevance of the relatively small statistical differences for daily practice.

Thirdly, the reliability of the A&E‐level mean scores appeared to be sufficient for three domains with the given samples sizes. Low response rates for ‘information before treatment’ and ‘discharge management’ explain the poor reliability of both domains. To obtain a good reliability, sample sizes should be enhanced to 301 and 473 respondents, respectively; this would be difficult to accomplish for the discharge management domain due to a ‘skip to question’ link in the questionnaire, which precludes patients admitted to hospital wards (30%) completing several items. Another domain with limited reliability is the ‘information during treatment’ domain. Here, the problem is caused by the small difference in patients’ experience mean scores (range is 0.12). The value of increasing the sample size to 488 patients for the detection of such a small difference is questionable. Increasing the number of patients could benefit the discriminative capacity of the CQI A&E, and possibly reveal significant differences on more domains, but the consequence of higher costs to detect small differences should be considered.29

Two domains were unable to discriminate among A&Es. However, we argue that public reports should provide a complete overview of the quality of care. Therefore, we should reconsider the construct of the questionnaire to obtain discriminative domains. We estimate that increasing the number of respondents by adapting the skip questions to avoid elimination of patients to complete the items, which constituted the non‐significant domains, will be sufficient.

Publications on benchmarks and patient‐centred care in the A&E are limited. Chalder et al. compared patients’ satisfaction at walk‐in centres and A&Es. There was no evidence that walk‐in centres co‐located with A&Es had achieved the aim of increasing patients’ choice, preferences or satisfaction with received care.30 Recently, Raleigh et al. studied six domains of patients’ experiences across three service areas of trusts (outpatients, inpatients and A&Es) and reported three performance levels. 30 of 142 trusts performed better on all domains, and 6 of 142 performed worse on all domains. We found one of 21 A&Es that performed better on all five discriminative domains, and none that performed worse on all domains. The identification of a best practice A&E, which potentially would be a role model for other A&Es, might have a general positive effect on quality of care. However, sustained improvements tend to be achieved combined by, for instance, government targets, coupled with incentives and penalties.31, 32

Two main limitations of the study were response bias and selection bias. Respondents were somewhat older and more likely to be female. To control for response bias, case‐mix adjustment for age, gender and health status was applied on the domain scores. The discriminative capacity was not affected by this adjustment. Also, respondents were more often assigned to the orange and yellow triage categories compared with the non‐respondents, who were more often triaged in the green and blue (i.e. less urgent) categories. Not responding could be influenced by many factors related or unrelated to the quality of care, such as language differences or unconsciousness of patients (recall bias). The direction and magnitude of impact of such factors (and potentially others) is not known, and could potentially influence the survey results and therefore also their generalizability.

Secondly, we were able to include the required number of A&Es in our study. Participation of A&Es was voluntary and selection bias could have occurred. However, we think this influence is minimal, as A&Es varied in terms of patient volume, geographical area (urban or rural, and regions), trauma centre or non‐trauma centre, and teaching or non‐teaching status, reflecting the full variation present in Dutch A&Es.

Conclusions

The CQI A&E is a validated survey to measure health‐care performance in the A&E from patients’ perspective. Five domains regarding quality of care aspects and the ‘global quality rating’ had the capacity to discriminate among A&Es, and to identify best practices as experienced by patients. The global quality rating and four domains showed good reliability given actual sample sizes.

Funding Sources/Disclosures

This study was partly funded by AEGIS Healthcare Insurance. The views expressed are those of the authors and not of AEGIS Healthcare Insurance.

Conflicts of interest

None.

Supporting information

Appendix S1 Caterpillar plots of patients’ experience domain scores.