Quality of integrated chronic care measured by patient survey: identification, selection and application of most appropriate instruments

Abstract

Objective  To identify the most appropriate generic instrument to measure experience and/or satisfaction of people receiving integrated chronic care.

Background  Health care is becoming more user‐centred and, as a result, the experience of users of care and evaluation of their experience and/or satisfaction is taken more seriously. It is unclear to what extent existing instruments are appropriate in measuring the experience and/or satisfaction of people using integrated chronic care.

Methods  Instruments were identified by means of a systematic literature review. Appropriateness of instruments was analysed on seven criteria. The two most promising instruments were translated into Dutch, if necessary, and administered to a convenience sample of 109 people with a chronic illness. Data derived from respondents were analysed statistically. Focus‐group interviews were conducted to assess the semantic and technical equivalence as well as opinions of people about the applicability and relevance of the translated instruments.

Results  From 37 instruments identified, the Patients’ Assessment of Care for chronIc Conditions (PACIC) and the short form of the Patient Satisfaction Questionnaire III (PSQ‐18) were selected as most promising instruments. Both instruments produced similar median scores across people with different chronic conditions. The overall PACIC and its subscales and the overall PSQ‐18 were highly internally consistent, but not the PSQ‐18 subscales. Overall, the PACIC demonstrated better psychometric characteristics. PACIC and PSQ‐18 scores were found to be moderately correlated. Whereas more respondents preferred the PSQ‐18, focus‐group participants regarded the PACIC to be more applicable and relevant. The technical and semantic equivalence of both instruments were sufficient.

Conclusions  Because of its psychometric characteristics, perceived applicability and relevance, the PACIC is the most appropriate instrument to measure the experience of people receiving integrated chronic care.

Introduction

Health care is becoming more user‐centred and, as a result, the experience of users of care and the evaluation of their experience and/or satisfaction are taken more seriously, more often measured systematically and used to evaluate the delivered care. ¹ , ²

Notwithstanding the lack of clarity concerning the meaning of patient or user satisfaction, ³ , ⁴ users’ experience and satisfaction are intertwined. The research on satisfaction with health care has been primarily empirical and little attention has been paid to the conceptualization of patient or user satisfaction. ³ , ⁵ One attempt to conceptualize patient satisfaction comes from Linder–Pelz and says that it is the ‘the individuals’ positive evaluation of distinct dimensions of health care’. ⁶ In Pascoe’s conceptualization of patient satisfaction, the users’ reaction is a comparison of the experience with a subjective standard. ⁷ If two individuals differ in their satisfaction with health care, it may be because of differences in their perception of experiences with health care, in their expectations for health care, or both.

Despite the differences in conceptualization, both users experience and satisfaction can, if appropriately measured, indicate the quality of care and act as important information to improve the quality of care. ³ Moreover, it could also be used to evaluate care innovations for chronically ill people. As people with a chronic illness consume a large amount of health‐care services for a relatively long time, measuring experience and/or satisfaction among them is of extra importance. ²

There is a considerable amount of literature about patient experience and/or satisfaction, but it is not clear how appropriate the instruments are to measure user experience and/or satisfaction with care for chronically ill people. This question becomes even more important when, as for example in the Dutch health care, satisfaction instruments co‐validated by health plans are being introduced as a marketing tool. ⁸ This is especially the case in current strategies towards the integration of chronic care.

Integration of care is defined by the WHO as ‘bringing together inputs, delivery, management and organization of services related to diagnosis, treatment, care, rehabilitation and health promotion wherein integration is regarded as a means to improve the services in relation to access, quality, user satisfaction and efficiency’. ⁹ In Maastricht for example, this definition guides programmatic approaches towards reorganizing chronic care. Main features of this approach are: central coordination, protocolized assignment of people with a chronic illness to general practitioner (GP), nurse specialist or medical specialist, central data collection with yearly feedback, and regular training and education of the caregivers.

When taking users’ perspectives seriously in this integrated care approach, a measurement instrument is needed to appropriately assess the experience and/or satisfaction of people with a chronic illness. Integrated care is, in contrast with disease management initiatives, not aimed at a single disease and involves the collaboration of multiple disciplines and services. Therefore, at least these two characteristics of integrated care need to be reflected by the measurement instrument.

This study therefore reports on the identification and application of appropriate instruments to measure patient experience and/or satisfaction with integrated chronic care. In particular, we sought instruments that would reliably and validly assess whether care met the needs of people with a chronic illness.

Methods

To answer the research question, we: (i) identified and selected instruments that measure experience and/or satisfaction of people with chronic care and (ii) administered two selected instruments to chronically ill subjects to assess their feasibility, reliability and validity for measuring satisfaction with integrated chronic care in the region of Maastricht, The Netherlands. As two of the authors, with extensive competencies in the field of assessing quality of integrated chronic care in the Netherlands (LMGS and HJMV), expected to identify only non‐Dutch instruments, analytical procedures for translation were on forehand adopted in the research methods.

Identification and selection of instruments

The Patient‐Reported Outcome and Quality of Life Instruments Database (PROQOLID) and MEDLINE were searched to identify studies that evaluated instrument(s) for measuring patient satisfaction with chronic care. Databases were searched for English and Dutch‐language articles published between January 1990 and May 2007. The following combinations of keywords were used: ‘patient satisfaction instruments chronic’, ‘integrated chronic care AND patient satisfaction’, ‘shared chronic care AND patient satisfaction’, ‘managed chronic care AND patient satisfaction’, ‘chronic disease management AND patient satisfaction’ and ‘transmural care AND patient satisfaction’. Transmural care can be regarded as the Dutch equivalent for shared care. Titles of articles and abstracts were assessed for appropriateness by two authors (RB and LMGS) and, if found to be so, the full‐text article was retrieved. Reference lists of the included articles were also reviewed and provided additional relevant citations. To be included in this review, studies had to contain an instrument capable of assessing interventions that bring together health‐care services for chronically ill people with the aim to reach a higher level of system quality.

For the selection of instruments, the following seven criteria were used:

• 1 The instrument should be standardized, i.e. all respondents should be asked identical questions, presented in the same order and with the same response formats. ¹⁰
• 2 The instrument should be multidimensional, i.e. it should consist of multiple items probing experience and/or satisfaction with different aspects. ¹¹
• 3 The instrument should be generic rather than disease specific.
• 4 The instrument should measure directly, i.e. it should focus on the users’ personal experiences with care, rather than on the users’ attitudes towards care and the health‐care system in general. ¹¹
• 5 The instrument should measure satisfaction with a team consisting of both generalists and specialists or with a collaboration between intramural and extramural care.
• 6 The instrument should be valid, i.e. the instrument should measure what was intended to measure. ¹²
• 7 The instrument should be reliable, i.e. the instrument should reflect true differences between individuals when measuring variability. ¹²

For every criterion met (answer: yes), one point was awarded. In case no clear answer could be given, a question mark was registered and no point was awarded if a criterion was not met. According to this procedure, a maximum of seven points could be awarded. The grading of instruments was performed independently by two authors (RB and LMGS) with the final decision left to the first author (HJMV) in case no agreement was reached. For final selection, instruments had to reach at least six points. In the event that more than two instruments would score at least six points, it was decided to base the final selection on the psychometric characteristics of the instruments.

Application of selected instruments

We chose to administer the two instruments most consistent with the above criteria to a convenience sample of people with a chronic illness known to receiving integrated care for self‐administration. This was performed to find out which questionnaire is preferred by users of integrated care and to investigate the psychometric characteristics of the selected instruments. To reduce respondent burden, we decided to include only two questionnaires in the final selection. Notwithstanding the public availability of both instruments, permission was obtained from the developers of both instruments for use in this study.

The convenience sample of 109 people with a chronic illness was derived from the region of Maastricht, the Netherlands, and consisted of 30 persons with chronic obstructive pulmonary disease (COPD), 30 persons with heart failure, 30 persons with arthritis and 19 persons with geriatric disorders. All these people were receiving transmural care, i.e. care provided by a team of a nurse specialist, a GP and a medical specialist in the office of the GP, with the nurse being the first point of contact for people with a chronic illness and serving as a liaison between the GP and the medical specialist in the hospital. All people with a chronic illness were asked for informed consent. They were systematically selected from address files of 13 general practices (in case of COPD) or from consult registrations of five nurse specialists (in case of arthritis and heart failure). The selection was made by one researcher (RB) who was not familiar with any of the persons with a chronic illness, practices or nurse and who randomly selected the names of people from alphabetically ordered lists for COPD, heart failure or arthritis. Because of a limited number of people with geriatric disorders, the fact these people often have limited cognitive function and receive care for only a short period of time, it was, however, not possible to systematically select them. The specialized geriatric nurse therefore recommended 27 people with geriatric disorders for participation (of whom 18 gave informed consent) and was provided with another 12 questionnaires to hand out in person.

Each questionnaire package included an introductory letter, the instruments, a questionnaire that asked for demographic information and the preference for either of the two instruments (‘which questionnaire did you prefer?’), and a return envelope. The demographic characteristics asked for were the person’s gender, age, education level and mother language. Furthermore, respondents were asked to explain which instrument they preferred, to write down any missed aspects or additional comments, and the amount of time spent (in minutes) to fill‐out each of the instruments.

Analyses

If a selected instrument was not formulated in Dutch, we translated it into Dutch with the use of the so‐called forward–backward procedure. ¹³ Translation into Dutch was performed independently by two native Dutch speakers. To arrive at one version, both translations were compared and discussed by both translators, two authors (RB and LMGS) and two people with a chronic illness. This forward‐translated version was then translated back into English by a native American‐English speaker and compared with the original version.

Respondent characteristics and time taken to fill out the instruments were described by percentages. To measure the internal consistency of selected instruments, we computed Cronbach’s alphas for the overall scales and each subscale. The internal consistency reliability was considered sufficient when Cronbach’s alpha values were ≥0.70. ¹⁴

To test the normal distribution of (subscales of) instruments, Shapiro–Wilk tests were conducted. Potential differences in satisfaction among respondents with different types of illness were evaluated using the nonparametric Kruskall–Wallis test, and Tukey’s multiple comparison test was used after significant difference between medians were detected. Furthermore, Pearson moment correlation coefficients were conducted to assess the extent to which the scales of the finally selected instruments were related (convergent validity).

To analyse (reasons for) questionnaire preferences, percentages were calculated. Potential differences in preference were tested by the chi‐squared test and differences in preference among the different chronic illnesses by the Kruskall–Wallis test.

Furthermore, three focus groups were conducted to assess opinions about the face validity of the subjects from questionnaires. ¹⁵ In total, six people with COPD, six people with heart failure and six people with rheumatic disorders agreed to participate. Each focus group included participants with different chronic illnesses, and in each focus group, the same three subjects were discussed for each questionnaire: the technical qualifications, the semantic qualifications, the relevance and applicability of the items. To discuss the technical qualifications of instruments, participants were asked for their opinion about the readability and comprehensiveness of the instruments. With regard to the semantic qualifications, the clearness of items was discussed. Finally, people were asked to what extent they found items of both instruments relevant and/or applicable when evaluating their experience with integrated care.

Each focus‐group interview was audiotaped. Directly after the focus‐group discussions, the two moderators (RB and LMGS) listened to the tape recordings and took notes on their immediate impressions. Transcripts of the tapes were made; for each session, two different authors (RB and LMGS) worked on the transcript analysis to ensure that logical conclusions were drawn from the data. As key issues were identified, a grid was developed to show which issues emerged in each session. When completed, the grid showed clearly which concerns were shared by each of the focus groups.

For all statistical analyses, significance was taken at the 5% level and spss 15.0 (SPSS Inc., Chicago, IL, USA) was used. The study was approved by the local ethics committee.

Results

Identification and selection of instruments

The search identified 813 studies of which we accepted 103 for further screening. After reading titles and abstracts, 72 papers were excluded for not reporting on instruments to measure patient experience and/or satisfaction with integrated chronic care. As a result, 31 different instruments were found in the literature (Table 1). For the selected instruments, the references cited provide information about validity and reliability. Other papers involving studies wherein instruments are applied can be provided upon request.

Table 1.

 List of identified instruments and feasibility scores

No.	Instrument	First author (ref.)	Criteria							Total
			A	B	C	D	E	F	G
1	PACIC	Glasgow 16	1	1	1	1	1	1	1	7
2	DTSQ	Bradley 27	1	1	0	1	0	1	1	5
3	VSSS	Ruggeri 17	1	1	0	1	1	1	1	6
4	SUQ	Osborn 28	1	1	0	1	1	1	?	5
5	No name	Poole 29	1	1	0	1	0	?	?	3
6	No name	Cherkin 30	1	1	0	1	0	1	1	5
7	MISS‐21	Maekin 18	1	1	1	1	0	1	1	6
8	PSI	Corrigan 19	1	1	1	1	0	1	1	6
9	ERS	Pascoe 31	1	1	1	1	0	?	?	4
10	PSQ‐18	Rand 20	1	1	1	1	0	1	1	6
11	PACE SQ	Atherley 32	1	1	0	1	?	1	1	5
12	PRP	Montori 33	1	1	0	1	0	1	1	5
13	PSHCPS	Marsh 34	1	1	1	1	0	?	1	5
14	SHC	Hall 35	1	1	0	1	?	?	1	4
15	PSH	Lubeck 21	1	1	1	1	0	1	1	6
16	SAT‐P	Majani 22	1	1	1	1	0	1	1	6
17	SAT‐16	Franchignoni 36	1	1	0	1	0	1	1	5
18	SSS‐30	Attkinson 37	1	1	1	1	0	?	1	5
19	CASC	Brédart 38	1	1	0	1	0	1	1	5
20	CSQ	Attkisson 23	1	1	1	1	0	1	1	6
21	EDITS	Althof 39	1	1	0	1	0	1	1	5
22	GSQ	Huxley 40	1	1	0	1	?	?	?	3
23	ITSQ	Anderson 41	1	1	0	1	0	1	1	5
24	TSQM	Atkinson 24	1	1	1	1	0	1	1	6
25	CAPHS	Hargraves 42	1	1	0	1	0	1	1	5
26	QUOTE	Sixma 43	1	1	1	0	0	1	1	5
27	PPE‐15	Jenkinson 44	1	1	0	1	0	1	?	4
28	RHSQ	WHO 45	1	1	1	0	0	1	?	4
29	GPSQ	Saum 46	1	1	0	1	0	?	?	3
30	SCQ	Koch 47	1	1	0	1	0	1	1	5
31	EUROPEP	Wensing 48	1	1	1	1	0	0	1	5

1 = present; 0 = absent; ? = unknown; total = sum of all scores; A = standardized; B = multidimensional; C = generic; D = directly; E = team/collaboration; F = valid; G = reliable. ¹¹ , ¹²

One instrument ¹⁶ had the maximum score and eight instruments ¹⁷ , ¹⁸ , ¹⁹ , ²⁰ , ²¹ , ²² , ²³ , ²⁴ scored six points. Of these, one instrument is not generic ¹⁷ and the others do not measure experiences and/or satisfaction with the health‐care team or with the collaboration between intramural and extramural care. One instrument measures patient satisfaction with individual doctor–patient consultations, ¹⁸ two instruments are concerned with intramural care ¹⁹ , ²² and one instrument with medication. ²⁴ Another instrument focuses on health‐care service in general. ²³ After comparing the two remaining instruments, ²⁰ , ²¹ it was decided to select the short form of the Patient Satisfaction Questionnaire III (PSQ‐18) ²⁰ having the best psychometric characteristics of the two. The final selection thus consisted of two instruments: the Patients’ Assessment of Care for chronIc Conditions (PACIC) ¹⁶ and the PSQ‐18. ²⁰

The PACIC is an instrument assessing patient’s receipt of clinical services and actions consistent with the chronic care model (CCM). ¹⁶ It includes 20 items aggregated into five subscales that emphasize patient–health‐care team interactions and, in particular, aspects of self‐management support: ‘Patient Activation’, ‘Delivery System Design/Decision Support’, ‘Goal Setting/Tailoring’, ‘Problem‐Solving/Contextual Counselling’ and ‘Follow‐up/Coordination’. Each PACIC score can range from 1 to 5, with higher scores indicating a higher extent to which patients received specific forms of care that are congruent with various aspects of the CCM. Each scale is scored by simple averaging of items completed within that scale, and an overall PACIC is scored by averaging scores across all 20 items. ¹⁶

The PSQ‐18 is a short‐form version of the 50‐item Patient Satisfaction Questionnaire III, including 18 items constructed as statements of opinion that are aggregated into the following seven subscales: ‘General Satisfaction’, ‘Technical Quality’, ‘Interpersonal Manner’, ‘Communication’, ‘Financial Aspects’, ‘Time Spent with Doctor’ and ‘Accessibility and Convenience’. ²⁰ Each PSQ‐18 item is scored on a five‐point scale ranging from 1 to 5, with higher scores indicating greater satisfaction. Items within the same subscale are averaged to create the seven subscale scores, and by averaging all scores the overall score is created. ²⁰ For each chronic condition included in the research, a slightly adjusted version was developed and, in contrast to the original version of the PSQ‐18, the patient was asked to evaluate the health‐care provider (s)he has most contact with in the Dutch versions of the PSQ‐18.

Testing of translated instruments

In total, 108 questionnaire packages were sent by mail and one package was personally distributed. Fifty‐eight participants returned the questionnaires at first request and another 31 after a telephone reminder. The total response rate therefore was 82% and differed by illness: 80% for COPD, 87% for heart failure, 87% for rheumatic disorders and 68% for geriatric disorders.

In Table 2, basic characteristics of respondents are presented. Half of respondents were male and more than half were older than 65 years. About 60% of respondents had lower education and all speak Dutch. Due to too many missing values, seven people were excluded from the analyses of the PACIC and seven people from the analyses of the PSQ‐18.

Table 2.

 Basic characteristics of respondents (n = 89)

Characteristic	Value	n (%)
Illness	COPD	24 (27)
	Heart failure	26 (29)
	Rheumatic disorder	26 (29)
	Geriatric disorder	13 (15)
Sex	Male	44 (49)
Age (years)	<30	4 (5)
	30–45	7 (8)
	46–65	28 (32)
	>65	50 (56)
Educational level	Low	52 (59)
	Mediate	16 (18)
	High	18 (20)
	Missing	3 (3)
Mother language	Dutch	89 (100)

On average respondents reported taking 9.5 (±1.7) minutes to fill out each of the instruments.

Validation of the Dutch version of the PACIC

For the total scale, a Cronbach’s alpha of 0.91 was found, which indicates good reliability. All items had a strong correlation to the total score and an Alpha if Item Deleted value of either 0.90 or 0.91, meaning that the reliability of the PACIC would not increase after elimination of any item.

Four PACIC subscales also had sufficient reliability. The second subscale (‘Delivery System Design/Decision Support’) was the only subscale with insufficient reliability (α = 0.64). All questions belonging to this subscale had a strong correlation to the total score and an Alpha if Item Deleted smaller than 0.64, meaning that the reliability would only decrease more if one of the items were discarded.

Validation of the Dutch version of PSQ‐18

For the total scale, a Cronbach’s alpha of 0.88 was found, which indicates good reliability. Most items had a moderate to strong correlation to the total score and an Alpha if Item Deleted of either 0.87 or 0.88. This means that the reliability of the PSQ‐18 would not increase if one of its items were eliminated. Only one question (‘I feel confident that I can get the medical care I need without being set back financially’) had an Alpha if Item Deleted greater than the Cronbach’s alpha of the total scale. When this question would be eliminated, the reliability of the PSQ‐18 would slightly increase.

With the exception of two subscales, ‘Communication’ and ‘Time Spent with Doctor’, no PSQ‐18 subscale had sufficient reliability. The Cronbach’s alpha of the subscale ‘Technical Quality’ would slightly increase when one question (‘Sometimes doctors make me wonder if their diagnosis is correct’) would be eliminated, but the reliability of the total scale would not change. The Cronbach’s alpha of the subscale ‘Accessibility and Convenience’ also increases when one of its questions would be eliminated (‘I have easy access to the medical specialists I need’). The subscale would then, however, still have an insufficient reliability and the reliability of the total scale would not increase. For subscales ‘General Satisfaction’, ‘Interpersonal Manner’ and ‘Financial Aspects’, the Alpha if Item Deleted could not be provided, since these subscales include only two questions each.

Application of instruments

Outcomes of the Dutch version of the PACIC

According to the Shapiro–Wilk test, data from three of five subscales were not normally distributed and therefore medians were used to measure central tendency (Table 3). For the overall PACIC, a median score of 2.60 was found and the median scores on the subscales ranged from 2.00 for the ‘Follow‐up Coordination’ scale to 3.33 on the ‘Delivery System Design/Decision Support’ scale.

Table 3.

 Outcomes of the Dutch version of the PACIC instrument (median), n = 82

Scale (1–5)	Disease				Total
	COPD	Heart failure	Rheumatic disorder	Geriatric disorder
Patient activation	2.33	2.67	3.33	3.00	3.00
Delivery system design/decision support	3.33	3.67	3.33	2.67	3.33
Goal setting/tailoring	2.20	2.20	2.40	2.20	2.20
Problem solving/contextual counselling	2.50	2.75	3.00	2.75	2.88
Follow‐up/coordination	1.60	1.80	2.20	3.00	2.00
Overall score	2.25	2.60	2.75	2.75	2.60

According to the Kruskall–Wallis test, there was only a statistically significant difference between the different chronic conditions on the ‘Follow‐up/Coordination’ scale. The Tukey test indicated a significant difference between respondents with geriatric disorders (3.00) and respondents suffering from COPD (1.60) and between respondents with geriatric disorders and respondents with heart failure (1.80).

Outcomes of the Dutch version of the PSQ‐18

According to the Shapiro–Wilk test for normality, data from all scales, except the ‘Accessibility and Convenience’ scale, were not normally distributed. Table 4 reports the median scores on all PSQ‐18 scales by the type of illness. For the overall PSQ‐18, a median score of 3.94 was found and the median scores on the subscales ranged from 3.75 on the ‘General Satisfaction’, ‘Technical Quality’ and ‘Accessibility and Convenience’ scale to 4.50 on the ‘Interpersonal Manner’ scale. According to the Kruskall–Wallis test of variance, there were no statistically significant differences among the respondents with different chronic conditions.

Table 4.

 Outcomes of the Dutch version of the PSQ‐18 instrument (median), n = 82

Scale (1–5)	Disease				Total
	COPD	Heart disorder	Rheumatic failure	Geriatric disorder
General satisfaction	4.00	4.00	3.50	3.50	3.75
Technical quality	3.75	4.00	4.00	4.00	3.75
Interpersonal manner	4.00	4.00	4.50	4.50	4.50
Communication	4.00	4.00	4.00	4.50	4.00
Financial aspects	4.00	4.00	3.50	4.00	4.00
Time spent with doctor	4.00	4.00	4.50	4.00	4.00
Accessibility and convenience	4.00	3.75	3.75	4.00	3.75
Overall score	3.89	3.94	4.06	4.22	3.94

Correlations between Dutch version PACIC and PSQ‐18 scales

To examine the correlation between the scales of the two instruments, three hypotheses were computed. We hypothesized that there would be a moderate correlation between the overall PACIC and the overall PSQ‐18 scores, and between all PACIC scales and the PSQ‐18 ‘General Satisfaction’ scale. The rationale for these hypotheses was that user directedness, user activation and self‐management are expected to stimulate user satisfaction. The other hypothesis was that the PACIC ‘Patient Activation’ scale would correlate moderately with the PSQ‐18 ‘Communication’, ‘Interpersonal Manner’ and ‘Time Spent with Doctor’ scales. The rationale for this hypothesis was that the American PACIC ‘Patient Activation’ scale correlated moderately with the Safran ‘Communication and Interpersonal Care’ scale, ¹⁶ which are similar respectively to the PSQ‐18 ‘Communication’ and the PSQ‐18 ‘Interpersonal Manner’ and ‘Time Spent with Doctor’ scales.

As shown in Table 5, a significant Pearson correlation was found between the overall PACIC and the overall PSQ‐18 (r = 0.39), meaning that the first hypothesis was confirmed. The third hypothesis was also confirmed: the PACIC ‘Patient Activation’ scale correlated moderately with the PSQ‐18 ‘Communication’ (r = 0.35), the ‘Interpersonal Manner’ (r = 0.40) and the ‘Time Spent with Doctor’ scales (r = 0.25). The second hypothesis, i.e. that all PACIC scales should correlate moderately with the PSQ‐18 ‘General Satisfaction’ scale, in contrast, was not confirmed. However, many other significant correlations were found.

Table 5.

 Correlations between Dutch version PACIC and PSQ‐18 scales (Pearson moment correlation coefficients)

PSQ‐18 scales	PACIC scales					Overall score
	Patient activation	Delivery system design/decision support	Goal setting/ tailoring	Problem solving/ contextual	Follow‐up/ coordination
General satisfaction	0.22*	0.29	0.07*	0.30	0.10*	0.23
Technical quality	0.39	0.37	0.25	0.42	0.22*	0.40
Interpersonal manner	0.40	0.37	0.26	0.40	0.22	0.40
Communication	0.35	0.24	0.23	0.44	0.21*	0.36
Financial aspects	−0.09*	−0.03*	−0.13*	−0.01*	−0.02*	−0.07*
Time spent with doctor	0.25	0.17*	0.13*	0.31	0.21*	0.27
Accessibility and convenience	0.29	0.42	0.29	0.41	0.24	0.40
Overall score	0.35	0.37	0.23	0.45	0.24	0.39

*Correlation is not significant at the 0.05 level.

Instrument preferences

The PSQ‐18 was preferred above the PACIC by more than half of the questionnaire respondents (58.4%). Almost one‐third (31.6%) preferred the PACIC, and 10% did not prefer one of the questionnaires above the other. According to the chi‐squared test, these differences in preference were significant (P = 0.00). According to the Kruskall–Wallis test, there were no significant differences in questionnaire preference between the respondents with different chronic conditions (P = 0.406).

Focus‐group results

Among the 15 people who participated in the focus group, five had COPD, four had heart failure and six had a rheumatic disorder. Only three (20%) were male. The average age of participants was 55 years (range 26–77).

Technical equivalence

In general, focus‐group participants regarded the translated version of the PACIC as being readable and comprehensible. The first and fourth questions were, however, considered to be problematic. For the first item (‘asked for my ideas when we made a treatment plan’), it was suggested to replace the word ‘treatment plan’ by ‘stepwise approach’, and for the fourth item (‘given a written list of things I should do to improve my health’) the word ‘list of things I should do’ by ‘information folder’. Other participants had trouble with the word ‘organized’ (item 5: ‘satisfied that my care was well organized’). However, no substitute was suggested for the latter. In addition, some participants felt that there is overlap between a few questions and that questions are not applicable for patients who have suffered from their illness a long time, but others agreed that a time period >6 months should be evaluated.

The translated version of the PSQ‐18 was considered to be clear and comprehensible in all three focus groups. However, some participants had the opinion that the questionnaire should not be filled out for only one health‐care provider, given the team consists of more than one provider. Some thought that a few questions overlap each other and that the questions are not applicable for patients who suffer from their illness a long time. Moreover, the words ‘perfect’ (item 3: ‘the medical care I have been receiving is just about perfect), ‘office’ (item 2: ‘I think my doctor’s office has everything needed to provide complete care’) and ‘emergency treatment’ (item 9: ‘where I get medical care, people have to wait too long for emergency treatment’) were considered unclear.

Semantic equivalence

To improve the semantic equivalence between the translated and original version of the PACIC, three questions (item 5: ‘satisfied that my care was well organized’; item 7: ‘asked to talk about my goals in caring for my illness’; and item 14: ‘helped to plan ahead so I could take care of my illness even in hard times’) were discussed in more detail as these initiated discussion during the translation. According to participants, the word ‘organized’ is too broad and can be interpreted in different ways. Participants interpreted the seventh question wrongly. It was not clear what was meant by ‘my goals’. The fourteenth question was well‐understood.

For the PSQ‐18, two questions (9 ‘where I get medical care people have to wait long for emergency treatment’ and 16 ‘I find it hard to get an appointment for medical care right away’) were discussed in more detail as both raised questions during translation.

Participants were asked what they think about the decision to let patients evaluate the health‐care provider they have most contact with. Participants generally disagreed with the decision to let users only evaluate the health‐care provider they have most contact with.

Applicability and relevance of major questionnaire topic areas

In contrast to the PSQ‐18, all major topic areas covered by the PACIC were considered to be important by participants. While most participants found the PSQ‐18 to be important, some regarded the subject of general satisfaction as being a bit vague and did not agree with the importance of the subject of financial aspects. Moreover, although technical quality was considered to be an important subject, they thought it is almost impossible to evaluate this objectively.

Discussion

Evidence to date supports efforts to make care more user‐centred, but we still have much to learn about what aspects of care impact outcomes and/or are valued by people with a chronic illness. ¹ Like in other service industries, sustained profitability in health‐care stems, from meaningful customer focus, collaboratively designed services, and positive interpersonal exchanges. ²⁵ Notwithstanding health‐care organizations being keen to take users perspectives seriously, this does not seem to be as simple.

In this study, 31 different patient‐satisfaction instruments were identified in the literature. Using seven criteria to assess the applicability to measure experiences and/or satisfaction of people with a chronic illness receiving integrated care, we selected two instruments and administered them to a convenience sample of 109 people with COPD, heart failure, rheumatic or geriatric diseases in the region of Maastricht. The PACIC fulfils all seven criteria, while the PSQ‐18 does not measure satisfaction with the health‐care team nor with the collaboration between intramural and extramural care. Although the PACIC was intended to assess the receipt of user‐centred care, it is regarded that both user satisfaction and user experience are connected to each other and that patient directedness, patient activation and self‐management stimulate user satisfaction. ¹

Both the PACIC and PSQ‐18 showed good reliability. The internal consistency of subscales was sufficient for four of five subscales of the PACIC and for only two of seven PSQ‐18 subscales.

Based on the PACIC, it was found that questionnaire respondents in Maastricht are more satisfied with ‘patient activation’ and ‘delivery system design/decision support’ than with ‘goal setting/tailoring’, ‘problem‐solving/contextual counselling’ and ‘follow‐up/coordination’, with the two first‐mentioned subscales being in the room for improvement. From the scores on the PSQ‐18, rather high satisfaction scores were found on all subscales, with no clear room for improvement on one of them. Certain subscales of the PACIC and the PSQ‐18 and their overall scores correlate moderately. Despite the slightly better quantitative characteristics of the PACIC, it was found that more questionnaire respondents prefer the PSQ‐18. The focus‐group interviews did not provide reasons for this difference in terms of technical qualifications, semantic qualifications, relevance and applicability of the items. Moreover, focus‐group participants seemed to agree more on the importance of subjects from the PACIC than from the PSQ‐18, with suggestions for improvement being given for both instruments. Considering that the Netherlands offers universal coverage to all its citizens with additional protection for people with chronic conditions, it is not surprisingly that participants found the item on financial aspects less relevant.

In a review of the role of assessing treatment satisfaction, Weaver et al. ³ selected 19 articles from more than 1400 abstracts dealing with satisfaction measures. They concluded that the quality of measurement is relatively poor and recommended that researchers and decision makers devote more attention to qualitative research with patients, and to studying the attributes of the measures, and the covariates. ³ Notwithstanding differences between treatment satisfaction and satisfaction of users with chronic care, this study did try to make use of these recommendations. Moreover, this study shows the important contribution of people with a chronic illness in evaluating instruments to measure their experience and satisfaction with chronic care.

Our study has its strengths and limitations. Strengths include: the use of both questionnaire application and the use of focus groups, a relatively high response rate on the questionnaires and inclusion of people with different chronic conditions. Limitations include the relatively modest scope of the literature review, the narrow assessment of the reliability and the validity of instruments and the relatively small sample size, especially people with geriatric disorders. Although the last did not seem to have influenced the results, another administration method could be considered for including more people with geriatric disorders. Another limitation is the fact that not all questionnaires were sent by regular mail. However, since only one questionnaire was distributed in person, the results are unlikely to have been biased by the different methods of administration. Finally, two other limitations of the study include the facts that only three focus‐group participants were male and that two focus groups did not have the planned six participants. It does not appear that these have weakened the results.

The objective of the research was to identify an appropriate generic instrument to measure patient or user satisfaction with integrated chronic care that could be used by the Maastricht University Hospital on a regular basis. Although the PSQ‐18 was preferred by more respondents than the PACIC, the latter was found to have better psychometric characteristics. For us, the inclusion of items measuring satisfaction with the cooperation between health‐care providers in the PACIC and its good psychometric properties made it the preferable instrument. Therefore, it is concluded that the PACIC is currently the most appropriate instrument to measure the satisfaction of people with a chronic illness receiving integrated care in Dutch‐speaking populations. Recently, similar findings were reported from a German study. ²⁶

In general, this study offers useful insights to those who want to select patient experience or satisfaction instruments for efforts to monitor and improve the quality of health care in similar or other settings, i.e. when dealing with different health conditions or located in different health‐care systems.

It needs to be assessed to which extent the PACIC is suitable for the evaluation and/or comparison (inter)nationally. Further research is also recommended to explore the psychometric characteristics of the Dutch version of the PACIC when applied on more as well as people with a chronic condition other than COPD, heart failure, rheumatic or geriatric disorders. Before doing this, attention should be paid to the suggestions of people with a chronic illness for recommendations as found in this study.