How to measure experiences of healthcare quality in Denmark among patients with heart disease? The development and psychometric evaluation of a patient-reported instrument

Abstract

Objective

Measuring the quality of care as experienced by patients is increasingly recognised as a way of improving healthcare services. However, disease-specific measures that take the patient journey into account are needed. This paper presents the development of such a measure for patients with heart disease and details the psychometric evaluation.

Design

The questionnaire was developed based on a literature review, qualitative interviews and a pilot-test. The psychometric evaluation of the measure was assessed using exploratory factor analysis (EFA) and confirmatory factor analysis (CFA), Cronbach’s alpha coefficient and differential item functioning analysis with data from a population-based survey.

Setting

Denmark in 2013–2014.

Study participants

Nineteen heart patients, four relatives and eight health professionals participated in qualitative interviews in the development phase, and 15 patients participated in the pilot-test. The questionnaire was subsequently sent to a random sample of 5000 heart patients who were diagnosed in 2013.

Results

The comprehensive development phase and pilot-testing contributed to high content validity of the questionnaire. Eligible questionnaire responses were received from 2496 patients. EFA indicated a nine-factor model: communication at the hospital, communication with the general practitioner, information on disease and treatment, information on psychosocial aspects, rehabilitation/support, organisation, medication, involvement of relatives and consideration of comorbidity. CFA confirmed the proposed factor structure (eg, goodness-of-fit index=0.88, adjusted goodness-of-fit index=0.86, root mean square error of approximation=0.05), and Cronbach’s alpha coefficient revealed good internal consistency of the factors (range: 0.69–0.93).

Conclusions

The results suggest that this disease-specific patient-reported experience measure is of good quality when measuring the quality of care among heart patients. The inclusion of patients in the development phase contributed to high content validity, and subsequent psychometric evaluation found high construct validity and internal consistency. This measure may be especially relevant when seeking information about which aspects of care require improvement and the impact on health outcomes.

Strengths and limitations of this study.

• A main strength of this study was the thorough psychometric evaluation of the patient-reported experience measure.

• The survey was nationwide and based on a population-based, random sample of 5000 heart patients, reducing the risk of selection bias.

• The inclusion of a patient journey provides an important basis to explore the relationship between patients’ experiences with the quality of healthcare and long-term health outcomes and to discover which aspects of care to address to improve patients’ experiences across sectors.

• Although this measure could be used in similar settings and patient groups, wider generalisation, across countries and patient groups, may be restricted.

• This study is limited by non-responders and item non-response.

Introduction

Due to the impact on daily life and the high mortality rates, heart diseases constitute a major public health concern in both Denmark and worldwide.^{1 2} Consideration of patients’ experiences are increasingly recognised as essential in achieving high quality healthcare provision. In early 2000, the WHO and the Institute of Medicine (IOM) stressed the importance of patients’ perspectives for assessing and improving healthcare.^{3 4} WHO introduced the concept of responsiveness and IOM identified the provision of patient-centred care as one of six objectives for improving healthcare. The care should be ‘respectful of and responsive to individual patient preferences, needs and values and ensuring that patient values guide all clinical decisions’.⁴ Accordingly, enhancing patients’ experience with the quality of healthcare, which includes aspects such as treating patients with dignity and respect and meeting the patients information and communication needs⁵ has become a legitimate independent goal for healthcare services. Moreover, increasing evidence suggests that good experiences are positively associated with clinical effectiveness, including adherence to treatment, lower length of stay and decreased mortality in various settings and patient groups.^6–9

As a result, surveys quantifying patients’ experiences with the quality of healthcare have become widespread and commonly include patient-reported experience measures (PREM).^10–12 However, several questions remain unanswered. There is no agreement about what constitutes quality of healthcare from the perspective of patients, resulting in several multidimensional theoretical frameworks and a range of PREM instruments.¹⁰ According to Beattie et al, the purpose and context of the survey is important when choosing the PREM, particularly in relation to content validity.¹⁰ Also, what is important to patients may differ between countries, patient groups and healthcare settings, evolving over time. Existing instruments have been specific to hospital inpatients or outpatients, primary care patients and often relate to a single contact. However, to truly represent the perspective of patients, it can be argued that we need to take their experiences with a full patient journey into account. This would provide important information on which aspects of the care patients feel require improvement and provide a unique basis to examine the relationship between patients’ experiences and health outcomes, such as readmissions and death, including the relationship with the single patient experience dimensions. The impact on health outcomes of patients’ experience with a patient journey across sectors is expected to be greater than their experiences with a single consultation or healthcare setting. Information about the impact of patients’ experience is scarce, particularly for the dimensions of emotional support, coordination of care, continuity and transition and involvement of relatives.^{9 13} Thus, there is a need to develop context-specific PREM measures which include a patient journey with the perspective of long-term follow-up. The aim of this paper was twofold: first, to present the development of a national questionnaire measuring patient-reported experience of healthcare quality following heart disease, covering aspects of the patient journey from first contact with the healthcare system to the inpatient and outpatient treatment and rehabilitation and next, to perform a psychometric evaluation of a measure covering these aspects.

Method

Study design and setting

The questionnaire development was based on a literature review, qualitative focus group interviews and individual interviews. It was pilot-tested using cognitive interviewing techniques. The psychometric evaluation was assessed using exploratory factor analysis (EFA) and confirmatory factor analysis (CFA), Cronbach’s alpha coefficient and differential item functioning (DIF) analysis with data from a population-based survey conducted in Denmark in 2014. The Danish healthcare system is publicly financed with the goal of free and equal access for all 5.6 million inhabitants. The Danish health system is divided into two sectors: primary healthcare that are provided by general practitioners (GPs) and the municipal health services, where responsibilities include disease prevention, health promotion and rehabilitation outside of the hospital and the secondary (hospital) care being responsible for more specialised medical treatment (inpatient and outpatient care).¹⁴

Questionnaire development

The questionnaire was developed between September 2013 and August 2014. An overview of the stages in the development of the questionnaire is available (see online supplementary figure 1). We carried out a literature review and a qualitative explorative investigation. The aim was to develop a context-specific measure with high content validity to examine experiences with the Danish healthcare system among patients with a heart disease and to include the most important measurements of their health status. The patient experience measure was intended to cover aspects of the patient journey from first contact with the healthcare system to the inpatient and outpatient treatment and rehabilitation. The literature search was conducted in the databases PubMed and PsycINFO and included reviews (<15 years) and quantitative and qualitative studies (<10 years). The review was predefined to include 50–100 articles and the following criteria were used to select them: They should cover the four heart diseases of interest (ischaemic heart disease, atrial fibrillation, heart failure and heart valve disease), different topics and different study types. Based on titles, 300 studies were selected by one researcher (LZ), which was reduced by half by another researcher (TH) after reading the abstracts. Through a consensus meeting, the two researchers agreed on appraisal of 86 studies: 23 reviews, 29 quantitative studies and 34 qualitative studies. We systematically extracted information of each eligible paper independently using two data charting forms developed by LZ and TH, one reporting on the material and methods of each study and one reporting the results of the studies. The literature review was used as a supplement to the qualitative investigation and as a basis to develop the interview guide. The qualitative investigation consisted of three focus group interviews with 19 heart patients in total, one focus group interview with four relatives of heart patients and eight individual interviews with health professionals working with heart patients. Patients and relatives were recruited through the Danish Heart Foundations’ membership newsletter to which 34 patients and 8 relatives responded. They were selected based on their gender and their address to enable focus groups of at least four persons within a reasonable driving distance. The Danish Heart Foundations’ counselling centre recruited another four patients and two relatives to ensure the groups were large enough. The interviews were conducted in three different parts of Denmark. The patients included 9 men and 10 women with different educational background and different heart diseases. The relatives included two wives, one husband and a daughter of heart patients (further characteristics of the participants are available in the online supplementary table 1). The health professionals to be interviewed were selected to achieve a group representing different health professions (eg, physician, nurse, psychologist), health sectors (eg, hospital, general practice) and stages of care (eg, acute medical treatment, rehabilitation). The interviews were conducted at professionals’ work places or as telephone interviews. All interviews were conducted by LZ and TH. The semistructured interview protocols (see online supplementary table 2) were developed through discussions in the project group. The group included professionals from public health (TH, LZ), anthropology (MH), sociology (PJ) and cardiology (AZ). It focused on what was perceived by the patients to be important for the quality of healthcare, including areas they found problematic or insufficient and on the challenges they experienced living with heart disease. The interview protocols were modified for patients, relatives and experts, and the interviewers were open and flexible so as to include any issues the interviewees brought up themselves. The focus group interviews lasted approximately 2 hours and the individual interviews lasted between 1–1½ hours. All interviews were audiorecorded, and by listening to these recordings LZ and TH condensed the material, meaning that the experiences and opinions expressed by the interviewees were provided in a shorter form. The analysis conducted by TH and LZ was data-driven and guided by conventional thematic analysis strategies allowing us to specify major themes and subthemes in the material.¹⁵ They both read all summaries to get an overall impression of the material and identified themes and subthemes in each interview (half each). After this, they met to discuss, refine and reduce the themes into major themes with subthemes. The results of the exploratory preinvestigation have been published in Danish separately  elsewhere.¹⁶

A reference group comprising eight health professionals participated together in a consensus workshop to ensure the questionnaire covered the results from the preinvestigation. This process resulted in seven themes concerning patient experiences with the quality of care: information, communication, organisation, psychosocial aspects, rehabilitation/support, medication and involvement of relatives. The group was also consulted to develop specific questionnaire items within their area of expertise and review the first version of the questionnaire. The reference group comprised the same health professionals who took part in the individual interviews, besides three replacements who were researchers within the field. We used existing Danish patient experience surveys, as inspiration and as a first basis for creating the item wordings of PREM.¹⁷ The questionnaire was pilot-tested through five individual interviews and two focus groups with 10 heart patients, based on cognitive interviewing techniques.¹⁸ The questionnaire was further tested by two researchers with expertise in constructing surveys. Minor adjustments to the questionnaire were made.

The questionnaire

The final questionnaire consisted of 48 PREM items covering the seven themes of patient experience with the healthcare system identified in the preinvestigation: information, communication, organisation, psychosocial aspects, rehabilitation/support, medication and involvement of relatives. They covered aspects of the patient journey, from first contact with the healthcare system to the treatment and rehabilitation in the hospital, in the municipality and by the GP. The questions were measured on a four-point Likert-type scale from ‘yes, to a great degree’ to ‘no, not at all’, though a few questions were binary (yes/no). Further reply options were ‘do not know’ and ‘not relevant’ (primarily managed by filter questions ‘if you answered…, go to question.’). Besides this, the questionnaire also included measures of health status, including physical status, emotional status, anxiety and depression, sexual health, quality of life, overall health and adherence to medication. This information uncovers patient’s needs and provides information on whether healthcare goals have been achieved as most healthcare goals relate to improvement in these areas, besides basic survival.¹⁹ Patient characteristic items including measures of patient activation/engagement, decision-making preferences, comorbidity, education and employment status were also included. This information was primarily measured by internationally validated questionnaire scales. An overview of the full content of the questionnaire which took approximately 30–45 min to answer is available (see online supplementary table 3) and the original version is available online.²⁰

Sample

A random population-based sample of 5000 patients diagnosed with incident ischaemic heart disease, atrial fibrillation, heart failure or heart valve surgery in 2013 was selected from the Danish National Patient Register (NPR). NPR includes information on all inpatient and outpatient contacts in all public and private Danish hospitals.²¹ Disproportionate stratified random sampling was used to reach at least 400 respondents for each of the four diagnostic groups (taking previous response percentages among heart patients into account) in order to ensure that results could be presented for each group within reasonable resources. The sampling was based on the incidences of disease in Denmark in 2012. Patients were included if they had none of the specified heart diseases in the previous 5 years (2008–2012), were ≥35 years, Danish residents and alive. They were selected based on their diagnosis at discharge according to ICD-10 codes, except for heart valve patients, who were identified using surgical codes according to the Danish version of the Nordic Classification of Surgical Procedures (NCSP-D).²² If they had more than one heart disease in 2013, they were selected based on their first contact. The sample consisted of 2116 patients with ischaemic heart disease (ICD-10: I20, I21, I23, I24, I25), 1695 patients with atrial fibrillation (ICD-10: I48), 642 patients with heart failure (ICD-10: I11.0, I13.0, I13.2, I42.0, I150) and 547 patients who had undergone heart valve surgery (NCSP-D: KFG, KFK, KFM, KFJE, KFJF) (these four diagnoses are referred to as patient groups). The selection criteria were established in consultation with two cardiologists (AZ and the research director at the Danish Heart Foundation) and based on former register-based studies within cardiovascular diseases.^{23 24} Further information is available in (see online supplementary table 4).

Data collection

The survey was conducted from October to December 2014. A paper questionnaire, a covering letter and a prepaid return envelope were posted to the patient sample group. The questionnaire could be completed on paper or online by using a unique password. If no response was received after 2 weeks, another questionnaire was sent and after 4 weeks they were contacted by telephone. Addresses were obtained from the Danish Civil Registration System (CRS) with information about all persons with a residence in Denmark, including name, date of birth, place of birth, place of residence and so on,²⁵ and the company NN Markedsdata linked publicly available phones numbers to addresses.

Statistical analysis

Sociodemographic characteristics of respondents and non-respondents were compared according to gender, age (at diagnosis), diagnosis, education, income, region, civil status, ethnic background and time from diagnosis to the distribution of questionnaires to examine whether the non-respondents deviated from the respondents. School and business education was combined into a single measure, according to the International Standard Classification of Education.²⁶ Average disposable income (after tax and interests) was calculated for the period 2008–2012 (100 DKK=13.44 EUR). Ethnic background was based on Statistics Denmark classification: persons of Danish origin (people born in Denmark with at least one Danish-born parent with Danish citizenship), immigrants (people born abroad with no Danish-born parents with Danish citizenship) and descendants (people born in Denmark with no Danish-born parents and without Danish citizenship). The information was obtained from CRS, NPR, the Population’s Education Register and the Income Statistics Register.²⁷

We conducted factor analysis to explore if our data measured the dimensions (factors) identified in the preinvestigation (construct validity). EFA was used as preliminary analysis to explore the structure of relationships between the items using both varimax rotation (orthogonal) and promax rotation (oblique). A priori, seven factors were identified. We expected their respective items to have relatively high loadings on their specific factor and low loadings on other factors. A few items could theoretically enter more than one of the predefined factors and EFA provided insight into which factor was the best fit. To assess whether data were appropriate for factor analysis, the Kaiser-Meier-Olkin measure of sampling adequacy was used and a value above 0.6 is considered tolerable.²⁸ To extract factors, we used the eigenvalue-greater-than-one rule, the scree plot and the proportion of variance accounted for. Further, the items that load on a factor should share a conceptual meaning and the items that load on different factors should appear to measure different constructs.²⁸ Items were excluded when EFA showed low factor loadings (<0.4) and when Cronbach’s alpha increased markedly without the item. Cronbach’s alpha measures internal consistency (reliability) with a coefficient ≥0.7 considered to be good.²⁹ Each dimension was also analysed separately, pairwise and so on, and ‘do not know’ responses were recoded to ‘no, not at all’ in a sensitivity analysis of the full model to allow most valid data to be used. The primary investigator (LZ) and a statistician (AKE) together compared the results from all sensitivity analyses with the main analyses and evaluated whether they pointed in the same direction.

To confirm the proposed factor structure by the EFA, CFA was performed. The overall adequacy of fit for the model was evaluated using a χ² test describing the similarity of the observed and expected matrices and by a number of goodness-of-fit indices: the goodness-of-fit index (GFI) with values>0.90 indicating a good model fit, the adjusted (for df) goodness-of fit index (AGFI) with values>0.80 indicating a good model fit, the standardised root mean square residual (SRMR) with a cut-off value close to 0.08, the root mean square error of approximation (RMSEA) with a small value indicating better model fit, typically 0.06 or less and the comparative fit index (CFI) with a larger CFI value indicating better model fit, typically 0.95 or greater.^{28 30} Nevertheless, the cut-off values are arbitrary but the measurements together provide a solid basis to evaluate the model fit. To include respondents with missing values, we used the full information maximum likelihood (FIML) method, which is suggested to be the most appropriate way to handle missing data in CFA.³¹ As a sensitivity analysis, we performed complete case analyses using the method ML and robust ML, the latter being robust to non-normality. Furthermore, ‘do not know’ responses were recoded to missing in an additional FIML analysis (otherwise ‘do not know’ responses were excluded). Finally, we excluded the largest patient group (ischaemic heart disease) from the main analysis.

Differential item functioning (DIF) (both uniform and non-uniform) was examined to evaluate whether men versus women and patients with ischaemic heart disease versus the three other patient groups respond differently to the measurement items using ordinal logistic regression,³² for example, if men and women responded differently to the item ‘Do you feel informed about how the disease may affect your future?’ after accounting for their overall level of information (uniform DIF) and if the direction of the difference changed as their overall level of information changed (non-uniform DIF). The likelihood ratio test was used to compare the goodness-of-fit of the models (with and without DIF). A significance level of 0.01 was used because the large number of significance tests increases the risk of type 1 error and the large sample size increases the risk of detecting unimportant, but statistically significant DIF.³² When DIF was statistical significant, the measure of effect size R² was used to evaluate the clinical or practical significance using an R² difference cut-off level of 0.02.³²

Data were analysed using SAS V.9.3 (SAS Institute, Cary, North Carolina, USA).

Ethics

The survey with individual-linked information to registries is approved by the University of Southern Denmark (2015-57-0008, no. 17/329 and 17/8592) according to the Permission of the Danish Data Protection Agency. All participants were informed about the aim of the study and were assured that participation was voluntary and results would be anonymised. Verbal consent was obtained from individuals participating in the interviews, and consent was assumed for survey participants when they returned the questionnaire. This type of study does not require further formal ethical approval, according to Danish law.^{27 33} The method of surveying a sample identified in NPR entails ethical considerations. Some of the selected heart patients may not recognise their own condition, especially patients with asymptomatic atrial fibrillation, due to the character of the disease. Furthermore, register information is not always erased, even if subsequent examinations show that the patient does not have the disease. This can be due to administrative errors. We, therefore, informed all patients that they had been selected to receive a questionnaire because, according to register information, they had been examined/treated for a heart disease in 2013. Each patient was also asked whether they had been diagnosed with a heart disease. If not, they were asked to return the questionnaire without answering further questions.

Results

Study population

The questionnaire was returned by 3029 patients (61% of the invited population). Of these, 533 stated they did not have a heart disease in 2013, and they were excluded from the sample (adjusted sample 4467). A total of 2496 patients were eligible for inclusion (56%) (see figure 1). A total of 43% of the respondents had ischaemic heart disease, 11% heart failure, 30% atrial fibrillation and 16% had undergone heart valve surgery and 35% of the patients were women. Sociodemographic characteristics of the respondents are shown in table 1.

Figure 1.

Flow diagram of the process from the sample to the final study population.

Table 1.

Sociodemographic characteristics of the respondents and non-respondents

	Respondents	Non-respondents	Source of information
	(n=2496)	(n=1971)
Gender, women, %	35.3	42.8	CRS
Age (at diagnosis), mean years (25th/75th percentile)	68.7 (62.1/76.4)	69.3 (59.0/80.6)	CRS
Diagnosis, %			NPR
Ischaemic heart disease	43.4	41.3
Atrial fibrillation	30.3	36.1
Heart failure	10.6	15.2
Heart valve disease	15.7	7.5
Civil status, %			CRS
Married	64.2	48.1
Widow/widower	16.3	23.5
Divorced	12.0	16.7
Unmarried	7.5	11.7
Educational level, %			PER
Basic school	36.2	47.1
Upper secondary or vocational school	41.9	37.1
Higher education	21.9	15.9
Income, EUR, mean (25th/75th percentile)	27,596 (17,597/32,493)	24,744 (16.389/27,763)	ISR
Ethnic background, %			CRS
Danish origin	94.7	91.8
Immigrant	5.1	7.9
Descendant	0.2	0.3
Region, %			CRS
Capital Region of Denmark	25.2	31.4
Central Denmark Region	22.8	20.4
North Denmark Region	10.5	10.0
Region of Southern Denmark	24.0	23.5
Region Zealand	17.5	14.8
Time from diagnosis to distribution of questionnaires, Months*, mean (25th/75th percentile)	16.3 (13.1/19.6)	16.5 (13.3/19.8)	NPR

*Standard month=30 days.

CRS, The Danish Civil Registration System; ISR, The Income Statistics Register; NPR, The Danish National Patient Register; PER, The Population’s Education Register.²⁷

Non-respondents analyses

The respondents differed from the non-respondents according to sociodemographic characteristics (table 1). Patients who were either male, married, had a heart valve disease or a higher education seemed to be over-represented in the response group. The difference in percentage points was below 10, with more than half below five (civil status ‘married’ did however have a 16.1 points difference and educational level ‘basic school’ a 10.9 points difference).

Validation of the PREM measurement

The predefined PREM dimensions from the preinvestigation, their items, responses and proportion of missing values are presented in table 2. The mean item-non-response was 12.1%.

Table 2.

Predefined dimensions from the preinvestigation, their items, responses (valid responses, don’t know, not relevant, missing) and proportion of missing values

Dimensions and their items		Valid responses	Do not know	Not relevant	Missing	Proportion of missing values
		n	n	n	n	%
Communication
A1	Was your condition taken seriously the first time you or anyone else contacted a doctor or the emergency dispatch centre in connection with your heart disease?	2169		149	178	7.6
A2	Do you think that the hospital staff involved you in decisions regarding your treatment?	2201	68	0	227	9.1
A3	Do you think that the hospital staff took you seriously?	2213	46	0	237	9.5
A4	Do you think that the hospital staff did what they could to answer your questions?	2189	59	0	248	9.9
A5*	Do you think that the hospital staff treated you as a human being and was not only interested in your illness and medication?	2237	41	0	218	8.7
A6	Do you think that the hospital staff listened to you?	2221	48	0	227	9.1
A7	Do you think that your GP involved you in decisions regarding your treatment?	1763	76	468	189	9.3
A8	Do you think that your GP took you seriously?	1821	30	468	177	8.7
A9	Do you think that your GP did what he/she could to answer your questions?	1786	40	468	202	10.0
A10*	Do you think that your GP treated you as a human being and was not only interested in your illness and medication?	1810	39	468	179	8.8
A11	Do you think that your GP listened to you?	1838	20	468	170	8.4
Information
B1†	Do you feel informed about which test/examinations you had to undergo and about the course of treatment?	2101	97	0	298	11.9
B2	Do you feel informed about how the disease may affect your future?	1954	220	0	322	12.9
B3	Do you feel informed about what you can do to get better for instance with physical activity or diet?	2061	117	0	318	12.7
B4†	Do you feel informed about who to contact in case your disease is worsening?	2035	131	0	330	13.2
B5	Do you feel informed about what your body can physically endure?	1990	182	0	324	13.0
B6	Do you feel informed about possible causes of your disease?	1801	356	0	339	13.6
B7†	Have you received conflicting information from different staff about your disease?	2002	225	0	269	10.8
Rehabilitation/support
C1	Have you been offered smoking cessation guidance in connection with your disease?	611	28	1670	187	22.6
C2	Have you been offered physical training in connection with your disease?	2220	36	0	240	9.6
C3	Have you been offered dietary guidance in connection with your disease?	2116	78	0	302	12.1
C4	Have you been offered education about your illness and its treatment in connection with your disease?	2056	125	0	315	12.6
C5*	Have you been offered guidance on sex life and relationships in connection with your disease?	2078	98	0	320	12.8
C6*	Have you been offered psychological/emotional support in connection with your disease?	2095	96	0	305	12.2
Psychosocial aspect
D1‡	Do you feel informed about the emotional reactions you may experience as a result of your disease?	1938	234	0	324	13.0
D2‡	Do you feel informed about the emotional reactions your relatives may experience as a result of your disease?	1896	270	0	330	13.2
D3‡	Do you feel informed about how the disease may affect your relationship with family, friends and others?	1877	289	0	330	13.2
D4‡	Do you feel informed about what the disease can mean for your sex life and relationships?	1821	295	0	380	15.2
D5§	Did healthcare staff at any time ask if you have experienced emotional problems in connection with your disease?	2117	205	0	174	7.0
D6§	Did you get an assessment of whether or not you had developed anxiety and/or depression by means of a questionnaire in connection with your disease?	2079	211	0	206	8.3
Organisation
E1‡	Was contact information provided to you at the hospital wards you have been in contact with?	2249		0	247	9.9
E2¶	Did you experience that the hospital staff was informed about the treatment of your disease?	2154	172	0	170	6.8
E3	Do you believe that a doctor or a nurse at the hospital have had an overall view of the course of treatment of your disease?	2195	135	0	166	6.7
E4	Do you think that you and/or your relatives have been partly responsible for the course of treatment which you believe the hospital should have been?	2067	203	0	226	9.1
E5¶	Did you experience that your GP was informed about the course of treatment of your disease?	1770	104	468	154	7.6
E6	Have you had to press for any guidance or support in connection with your disease?	1891	337	0	268	10.7
E7‡	Did you know what was going to happen when you were discharged from the hospital?	1295	62	955	184	11.9
E8‡	Were you told who to contact if you had questions about your illness or your course of treatment when you were discharged from the hospital?	1264	93	955	184	11.9
E9	Did the hospital staff make sure that you had an appointment for a consultation with your GP when you were discharged from the hospital?	1230	137	955	174	11.3
E10	Did the hospital staff take your other diseases (comorbidities) into consideration in the course of treatment of your heart disease?	996	262	512	726	36.6
E11	Did your GP take your other diseases (comorbidities) into consideration in the course of treatment of your heart disease?	849	193	879	575	35.6
Medication
F1§	Have you been offered an individual consultation about medication in connection with your disease?	2101	123	0	272	10.9
F2‡	Do you feel informed about why you get medication for your disease?	2108	47	78	263	10.9
F3‡	Do you feel informed about possible side effects of medications for your disease?	1972	125	78	321	13.3
F4‡	Do you feel informed about how long you should take medication for your disease?	1733	399	78	286	11.8
F5¶	Were you taken seriously when you contacted a doctor due to side effects?	815	14	1424	243	22.7
Involvement of relatives
G1	Has the staff made you aware of the possibility of having your relative(s) with you to consultations?	2216	101	0	179	7.2
G2	Has the staff involved your relative(s), the way you would like?	1886	182	238	190	8.4

The questionnaire was developed in Danish, and the translation to English has not been linguistically validated. Unless specified in the questionnaire items, the questions concern all the sectors, the patient has been in contact with (the hospital, GP and the municipality).

*Could theoretically also enter the dimension psychosocial aspects.

†Could theoretically also enter the dimension organisation.

‡Could theoretically also enter the dimension information.

§Could theoretically also enter the dimension rehabilitation/support.

¶Could theoretically also enter the dimension communication.

GP, general practitioner.

Based on an overall picture of several analyses all pointing in the same direction (including subanalyses and sensitivity analyses) the EFA provided support for the predefined dimensions. The exceptions to this were the dimension ‘communication’ that was split in two and the dimension ‘consideration of comorbidity’ was found to be an independent dimension separated from ‘organisation’. Thus, it resulted in a nine-factor model: (1) communication at the hospital, (2) communication with the GP, (3) information on disease and treatment, (4) information on psychosocial aspects, (5) rehabilitation/support, (6) organisation, (7) medication, (8) involvement of relatives and (9) consideration of comorbidity, with a total of 41 items. Four items (E2, E5, B7, D5) were found to fit another dimension, rather than their predefined dimension, and seven items (A1, B6, D6, E1, E8, E9, F1) were excluded because they did not load substantially on any factors. Item E3 ‘Do you think a doctor or a nurse at the hospital have had an overview of the course of treatment of your heart disease at all times?’ was the only item with cross-loading. Based on theoretical considerations, it was placed within the dimension ‘organisation’ although it also loaded onto the dimension ‘communication at the hospital’. When the final nine dimensions were analysed separately, to allow most valid data to be used, EFA suggested one factor for each of them. The factor loadings were between 0.49 and 0.95, and the dimensions revealed good internal consistency with Cronbach’s alpha coefficients ranging from 0.69 to 0.93 (table 3). Results with all dimensions analysed simultaneously using promax rotation are presented in the online supplementary table 5. The results were similar regardless of rotation type (promax/varimax).

Table 3.

Results from the final exploratory factor analysis exploring each dimension separately

Dimension	Items	N	KMO	Eigenvalue*	Factor loadings†	Variance explained‡	Cronbach’s alpha§
Communication at the hospital	A2, A3, A4, A5, A6, E2	1915	0.89	3.95	0.57–0.89	3.60	0.89
Communication with the GP	A7, A8, A9, A10, A11, E5	1648	0.91	4.46	0.58–0.92	4.21	0.93
Information on disease and treatment	B1, B2, B3, B4, B5	1727	0.83	3.14	0.65–0.79	2.68	0.85
Rehabilitation/support¶	C1, C2, C3, C4, C5, C6, D5	451	0.87	3.67	0.52–0.77	3.08	0.83
Information on psychosocial aspects	D1, D2, D3, D4	1654	0.84	3.34	0.78–0.95	3.14	0.93
Organisation**	E3, E4, E6, E7, B7	937	0.76	2.24	0.52–0.61	1.56	0.69
Medication††	F2, F3, F4, F5	719	0.74	2.32	0.49–0.73	1.80	0.76
Involvement of relatives	G1, G2	1830	0.50	1.70	0.84	1.41	0.83
Consideration of comorbidity	E10, E11	749	0.50	1.63	0.79	1.26	0.77

The EFA suggested one factor for each of the dimensions.

*Preliminary eigenvalue.

†Factor loadings (correlations) on the factor (no rotation since the factors are analysed separately).

‡Variance explained by the factor (no rotation since the factors are analysed separately).

§ The overall standardised Cronbach’s coefficient alpha.

¶The results were similar when excluding C1 increasing the number of observations to 1612.

**The results were similar when excluding E7 increasing the number of observations to 1467.

††The results were similar when excluding F5 increasing the number of observations to 1659.

GP, general practitioner; N, number of observations; KMO, overall Kaiser-Meier-Olkin measure of sampling adequacy.

The CFA supported the nine-factor model because it provided an acceptable overall fit: GFI=0.88, AGFI=0.86 SRMR=0.06, RMSEA=0.05 and CFI=0.93. The χ² test did though indicate a poor fit (p<0.001), but this test is limited by its sensitivity to sample size and model complexity. Thus, the model was expected to be rejected because of our large sample size and complex model, and therefore more emphasis was put on the other fit indices. The correlation between the factors (range=0.02–0.80, 54% below 0.50), the factor loadings for each item (range=0.43–0.95, 75% above 0.6) and the proportion of the variance for each item that could be explained by the factor (range=0.19–0.90, 58% above 0.50) are detailed in figure 2. All the sensitivity analyses confirmed the results. Statistical significant uniform DIF was detected in two items for gender (A2 and B1), for patient groups in four items (B3, B4, C3 and D5) and non-uniform DIF in one item (F3). Importantly, none of the effect sizes were clinical significant. The full text of the final nine-factor model with 41 items constituting the patient-reported instrument that measures experiences of healthcare quality in Denmark are available in the online supplementary table 6.

Figure 2.

Results from the confirmatory factor analysis, using the full information maximum likelihood method. N=814; χ²=15674, p=<0.0001; GFI=0.88; AGFI=0.86; CFI=0.93; RMSEA=0.05; SRMR=0.06. *N=507; χ²=12531, p=<0.001; GFI=0.87; AGFI=0.85; CFI=0.93; RMSEA=0.05; SRMR=0.06. *Complete case analysis, using the robust maximum likelihood method. Abbreviations: AGFI, the adjusted goodness-of fit index; CFI, comparative fit index; GFI, goodness-of-fit index; GP, general practitioner; RMSEA, root mean square error of approximation; SRMR, standardised root mean square residual. Note: The coefficients associated to the left are the correlations between the latent factors. The coefficients associated with arrows leading from the latent factors to the item products show the factor loadings for each item product. The coefficients to the right of the item products are the proportion of the variance for the item product that could be explained by the latent factor. n/a: Items excluded from the analysis, because they were irrelevant for a large proportion of respondents, reducing the data substantial: C1, E7, F5 and the factor consideration to comorbidity (E10 and E11). Subanalyses and sensitivity analyses supported the inclusion of these items.

Discussion

This study conducted a psychometric evaluation of a national questionnaire measuring patient-reported experience with the quality of patient care following a heart disease, covering aspects of the patient journey from first contact with the healthcare system to the inpatient and outpatient treatment and rehabilitation and presented the development of the questionnaire. The results suggest that the instrument is of good quality. The inclusion of heart patients in the comprehensive development phase and pilot testing of the questionnaire contributed to high content validity, and the subsequent psychometric evaluation found high construct validity and internal consistency. EFA indicated a nine-factor model: communication at the hospital, communication with the GP, information on disease and treatment, information on psychosocial aspects, rehabilitation/support, organisation, medication, involvement of relatives and consideration of comorbidity with a total of 41 items, and CFA confirmed this structure provided a good fit to the data suggesting high construct validity. Furthermore, Cronbach’s alpha coefficient revealed good internal consistency of the dimensions, and no clinical significant DIF was detected.

Some challenges regarding the measure were observed. CFA suggested that the factors organisation and communication at the hospital were highly overlapping as the correlation between the factors was 0.80. A factor correlation that exceeds 0.80 or 0.85 is often used as the criterion to define poor discriminant validity.³¹ We kept the factors separated, because they are conceptually different, another issue important to consider before collapsing factors.³¹ In future, it may be valuable to refine or expand the questions intended to capture the dimension organisation. The mean item-non-response was 12.1%, indicating a reasonable acceptance of the questionnaire. Even so, the question on smoking cessation guidance, if side-effects were taken seriously and the two questions regarding consideration of comorbidity had a proportion of missing values above 20%. These questions were irrelevant for a large proportion of the respondents and they might have skipped them without noticing the ‘not relevant’ response option. Simple filter questions may work better. The high proportion of ‘do not apply to me responses’ in certain items indicates a challenge with developing questionnaires to a wide group of heart patients. We chose to maintain such items to ensure that all aspects of importance to heart patients are represented in the questionnaire.

Implications

We believe this is the first PREM instrument among heart patients, covering aspects of the patient journey from first contact with the healthcare system to the inpatient and outpatient treatment and rehabilitation. This makes it particularly relevant when seeking information on aspects of care requiring improvement and when exploring long-term health outcomes. This new measure and the possibility of linking individual level patient data from Danish registries would allow future follow-up studies to examine the relationship between patients’ experiences with the quality of care and health outcomes such as readmissions and mortality. The measure would also be a suitable national surveillance tool to track performance of the healthcare system over time, as seen from the perspective of patients with heart disease. However, in acknowledging that what is important to patients may shift over time,¹⁰ we suggest refining the measurement on a regular basis. The measure is not appropriate for benchmarking hospital performance because several questions span across different sectors. Other measures are available for this purpose.¹⁰

Strengths and limitations

A main strength of this study was the thorough psychometric evaluation of the measure, which was based on a large population-based random sample of patients with heart disease limiting introduction of selection bias. However, a number of limitations should be considered. Missing-item non-response, ‘do not know’ and ‘do not apply to me’ responses reduced the number of valid responses and the data for the factor analysis substantially, even though the numerous subanalysis and sensitivity analysis all confirmed the results. The results of this study may not be generalisable to other patient groups; including patients with heart diseases other than those included in this study, patients below 35 years of age and patients only treated at their GP, since these patient groups are not represented in the survey. Nevertheless, it included the four most common heart diseases and the age groups where heart disease is most frequent.

Only half of the invited patients completed the survey, which is in accordance with other patient experience surveys,^{12 34} and the small differences between non-respondents and respondents indicate only minor implications. Also, no noteworthy differences were found according to age, educational level and region when comparing the respondents to the total population of heart patients (new cases in 2013),³⁵ although men were over-represented (data not shown). The respondents were diagnosed between 10 and 22 months prior to the data collection and the delay might make it difficult for patients to remember their first encounters with the healthcare system, but it might also help to distinguish between minor irritations and more serious failings.⁵ Importantly, it allows for the inclusion of a patient journey from first contact with the healthcare system to the inpatient and outpatient treatment and rehabilitation. Also, the variation in time since diagnosis may have influenced the patients’ answers. To our knowledge, no studies have considered the possible effects of time on PREM. The instrument is developed among relatively newly diagnosed patients, and even though the instrument is developed to cover aspects of the patient journey, it does only give an indication of their experience until that point in time. Nevertheless, as long as it is possible to identify and specify which course of disease is under consideration, we believe the questionnaire can be used by anybody with a heart disease. However, this assumption has not been tested. This measure is recommended for similar settings and patient groups; however, wider generalisation across countries may be restricted due to differences in the organisation of healthcare systems and so on. Moreover, generalisation to other patient groups must be done with caution, because what is important to diverse patient groups may differ.¹⁰ Also, we acknowledge the importance of conducting qualitative studies to complement patient experience surveys and vice versa.

Conclusion

The results suggest that the developed PREM is of good quality for measuring the quality of care among patients with heart disease covering aspects of the patient journey from first contact with the healthcare system to the inpatient and outpatient treatment and rehabilitation. The involvement of heart patients in the development phase contributed to high content validity, and subsequent psychometric evaluation found high construct validity and internal consistency. The inclusion of a patient journey provides an important basis on which to explore the relationship between patients’ experiences with the quality of healthcare and long-term health outcomes and which aspects of care to address to improve patients’ experiences.