Patient satisfaction questionnaires for primary care out-of-hours services: a systematic review

Abstract

Background

Patient satisfaction questionnaires are increasingly used for assessing quality of care.

Aim

To review the evidence for the reliability and validity of patient satisfaction questionnaires for out-of-hours care.

Design

Systematic review.

Setting

Primary care out-of-hours services.

Method

Searches of CINAHL, EMBASE, MEDLINE® and PsycINFO using terms relevant to the measurement of patient satisfaction and out-of-hours services. Abstracts were reviewed and information relating to questionnaire content, data quality, reliability, and validity were extracted from articles by two independent researchers.

Results

Four questionnaires were found, two from the UK — the Patient Satisfaction with Out-of-Hours Care (PSOC) and Short Questionnaire for Out-of-Hours Care (SQOC) — and two from the Netherlands — the van Uden and Moll van Charante questionnaires. Questionnaire content was based on literature reviews and expert opinion; the PSOC and Moll van Charante questionnaires were also developed following interviews or focus groups with patients. Cronbach's α values were below 0.7 for some scales within the PSOC and van Uden questionnaires. Test–retest reliability was reported for the PSOC and Moll van Charante questionnaires. Tests of validity were few and did not give explicit consideration to the size of expected associations.

Conclusion

Potential users wishing to assess patient satisfaction should carefully consider the content of the questionnaires and its relevance to the application and patient group. The four questionnaires have limitations relating to their development and evaluation. The PSOC and van Uden questionnaires have low levels of reliability for some scales, which should be used with caution in future surveys.

INTRODUCTION

There has been considerable change in the provision of out-of-hours primary care in the UK and Europe in the last 20 years.^1–5 The growth of GP cooperatives and deputising services has resulted in a change in the care setting, away from the home to the primary care centre and telephone consultations.⁵

It has been argued that attempts to evaluate the quality of out-of-hours care have been concerned with the process of care,⁶ and that there is a lack of appropriate outcome measures that incorporate the views of patients, including their satisfaction with services.⁷ Studies that have attempted to measure the outcomes or the quality of care of out-of-hours care from a patient perspective have included comparisons of different providers, including observational studies and randomised controlled trials.^8–11 Other studies have sought to compare different modes of care from the same provider, including consultations at primary care centres, home visits, and telephone advice.¹⁰ There has been a national survey of patient views of out-of-hours care within the Netherlands,⁵ and it has been recommended that providers monitor patient satisfaction in the UK.^12,13

These studies included surveys of patient satisfaction in the form of interview- or self-administered questionnaires based on summated rating scales. Several such questionnaires are now available for assessing patient satisfaction with out-of-hours care, more recent examples being based on more rigorous approaches to development and testing.^1,5,7 Following recommendations that providers should regularly monitor patient satisfaction,^12,13 it is important that the selection of questionnaires designed to measure such concepts takes account of the needs of users while taking into consideration evidence relating to the data quality, reliability, and validity.

How this fits in

The evaluation of out-of-hours care requires appropriate measures of patient satisfaction. Systematic searches produced four questionnaires which differed in content and evidence to support their application. Two questionnaires had scales with unsatisfactory reliability, and all had limited evidence for validity. Questionnaire content and evidence for data quality, reliability, and validity should be considered before application.

The work that follows is based on a systematic review of questionnaires designed to measure patient satisfaction with out-of-hours care. Following structured searches of the literature, information relating to the development, content, and evidence for data quality, reliability, and validity were extracted. The review will inform potential users, including service providers, of the availability and appropriateness of questionnaires. The identification of limitations of existing questionnaires will inform future developmental and evaluative work.

METHOD

Identification of studies

The search strategy was designed to retrieve articles relating to the development and evaluation — including data quality, reliability, and validity — of questionnaires designed to assess patient experiences or satisfaction with GP out-of-hours services. The following databases were searched: CINAHL (1982–2006), EMBASE (1980–2006), MEDLINE® (1966–2006) and PsycINFO (1967–2006). The searches included text words and medical subject headings that followed previous reviews within the field of patient satisfaction,^14,15 with terms relevant to out-of-hours care, truncating where necessary: ‘patient experiences or patient satisfaction’ and ‘emergency or acute or after hours or out of hours or night care or out-of-hours care or after-hours primary care services’ and ‘questionnaire or survey or instrument or tool or measure’.

Questionnaires were reviewed that had evidence of testing for both reliability and validity in non-English-speaking populations, and that were published in an English language journal. English abstracts relating to non-English language journals were also considered for inclusion. The citation lists of relevant articles were examined for further evaluative work and other questionnaires. The authors of identified questionnaires were emailed with requests for information relating to any further development work or publications, scoring instructions, and a copy of the questionnaire.

Following close collaboration to ensure a high level of agreement, two researchers independently assessed abstracts for inclusion and extracted relevant information from articles meeting the inclusion criteria.

Data extraction

The approach to data extraction was consistent with previous reviews within the fields of patient-reported outcomes,^16,17 patient satisfaction,¹⁴ and recommendations for questionnaire development and evaluation.¹⁸ Where available, the data extracted related to the questionnaire content, development, data quality, reliability, and validity. Data quality relates to levels of missing data and floor and ceiling effects at the item and scale level. The results of the two forms of reliability testing including item–total correlation, Cronbach's α, and test–retest correlation coefficients, were extracted. Evidence from the results of factor analysis or principal component analysis that supports the questionnaire scales or internal construct validity was extracted. Finally, comparisons with variables relevant to construct validity, together with any hypothesised associations, are described. Supplementary Table 1 summarises the quality of reporting within the review.

RESULTS

Identification of studies

Supplementary Figure 1 summarises the decision process relating to the identification of studies. The search strategy gave 2508 references, of which 18 included some measurement related to patient satisfaction with out-of-hours care. Four of these articles described the development and evaluation of four questionnaires. All gave consideration to reliability and validity and hence were reviewed. Of the remainder, one related to the measurement of preferences for different models of out-of-hours care, and 13 related to the application of questionnaires, for example in the assessment of the service quality. Nine of these studies used one of the four questionnaires described below. The remainder did not refer to a named or published questionnaire, and searches using the names of the authors did not produce any articles that were potentially relevant. The content of the four questionnaires is summarised in Table 1. Table 2 shows the stages in the evaluation of the questionnaires that included patients.

Table 1.

Descriptive statistics and reliability for the four questionnaires.

Questionnaire/scale	Number of items	Missing item range, %	Missing data scale, %	Mean (SD)	Item–total correlation range	Cronbach's α	Test–retest correlation
PSOC11			3.5 (2.9–4.3)a
Communication and management	7			65.78 (21.81)b,c		0.88	0.86
Doctor's attitude	5			72.19 (23.92)		0.87	0.82
Continuity of care	4			58.31 (19.92)		0.69	0.72
Delay until visit	3			46.54 (23.65)		0.65	0.81
Access to out-of-hours care	3			69.71 (20.01)		0.61	0.76
Initial contact person	2			69.37 (21.57)		0.72	0.62
Telephone advice	4			63.04 (23.17)		0.79	–
Overall satisfaction	4			66.12 (23.14)		0.77	0.82
Van Uden1
Telephone advice				66.2 (1.30)d
Accessibility by phone	3		0.55	76.5 (18.9)		0.72
Doctor's assistant's attitude	5		0.82	72.8 (22.1)		0.91
Questions asked by doctor's assistant	2		1.37	58.6 (25.4)		0.64
Advice given by doctor's assistant	5		4.10	53.7 (27.3)		0.93
Urgency of complaint	2		0.82	69.1 (24.5)		0.86
Overall satisfaction	5		1.37	64.2 (26.1)		0.93
Consultation at the GP cooperative				75.1 (1.31)
Accessibility by phone	3		1.79	79.3 (17.6)		0.73
Doctor's assistant's attitude	5		1.53	79.8 (16.3)		0.88
Questions asked by doctor's assistant	2		2.04	63.5 (23.0)		0.65
Urgency of complaint	2		2.04	72.0 (21.5)		0.79
Waiting time at the cooperative	2		1.28	61.5 (25.8)		0.62
Waiting room	2		2.81	65.6 (20.3)		0.60
Distance to the GP cooperative	2		1.02	66.7 (21.2)		0.75
Treatment by the GP	6		3.83	81.0 (18.9)		0.93
Overall satisfaction	4		0.00	73.7 (19.8)		0.88
Home visit				72.5 (1.37)
Accessibility by phone	2		2.74	80.9 (18.4)		0.86
Doctor's assistant's attitude	5		2.24	80.6 (18.6)		0.90
Questions asked by doctor's assistant	2		4.73	59.2 (26.6)		0.73
Urgency of complaint	2		4.73	86.7 (16.0)		0.78
Treatment by the GP	6		5.47	84.4 (19.7)		0.96
Waiting time until GP arrives	1		8.21	60.0 (30.7)		—
Overall satisfaction	4		2.99	74.6 (22.4)		0.92
SQOC7	7		5.26d	75.93 (23.57)e,f		0.94e
Moll van Charante5,g
Telephone advice						0.81	0.79
Telephone nurse	11	4.8–16.5			0.77–0.91	0.98	0.85
Organisation	3	6.2–43.2			0.53–0.59	0.74	0.92
Centre consultation						0.76	0.94
Telephone nurse	7	9.0–21.8			0.84–0.92	0.97	0.91
Doctor	12	3.3–13.1			0.81–0.94	0.98	0.93
Organisation	10	5.2–34.2			0.49–0.69	0.88	0.89
Home visit						0.85	0.89
Telephone nurse	7	12.0–26.1			0.86–0.93	0.98	0.91
Doctor	12	6.7–23.3			0.79–0.93	0.98	0.95
Organisation	4	10.2–45.1			0.62–0.74	0.86	0.90

^aMedian % (interquartile range %) for all of the scales.

^bPSOC scale scores are calculated by summing items and expressing the total as a percentage of the maximum possible score for the scale where 100 is the highest level of satisfaction. Responders must complete half or more of the items within a scale to produce a score.

^cNumbers are given as two decimal places, but some results were only reported as one decimal place.

^dVan Uden Scale scores are calculated by summing item responses and are scaled from 0–100 where 100 is the highest level of satisfaction.

^eSQOC results are for the final version that uses smiley face and a Likert scale.

^fSQOC items sum to produce a single score on a 0–100 scale where 100 is the highest level of satisfaction. Scale scores are not computed if more than one item is missing.

^gMoll van Charante Items sum to give a score from 1–10 where 10 is the highest level of satisfaction. Responders must complete two-thirds or more items to produce a score. PSOC = Patient Satisfaction with Out-of-Hours Care. SQOC = Short Questionnaire for Out-of-Hours Care.

Table 2.

Patient^a groups contributing to the development of the four questionnaires.

Questionnaire (country)	Development stage	Setting	Gross n	Reminders	Net, n (response rate %)
Patient Satisfaction with Out-of-Hours Care (UK)11	Two focus groups	Patients and carers recruited from general practice registers and community groups			11
	Semi-structured interviews	Patients recently requesting out-of-hours care from two large city practices or deputising service			28
	Preliminary interviews	Patients recently requesting out-of-hours care from two large city practices or deputising service			41
	Postal pilots	Within 72 hours of patients or carers requesting out-of-hours care at Leicester (n = 6) and Manchester (n = 1) practices			378b
	Main survey interviews	Within 24–129 hours of patients requesting out-of-hours care at Leicester (n = 3) and Manchester (n = 11) practices	1466		1402 (95.6)
	Test–retest	Return of a second questionnaire the same day as the main survey	200		112 (56.0)
Van Uden questionnaire (the Netherlands)19	Main postal survey	Patients and carers contacting GP cooperatives across regions in the South of the Netherlands were mailed questionnaire within 3 weeks	2733	1 reminder at 3–4 weeks	1160 (42.4)
SQOC (UK)7	Pilot studies	Three sites in Scotland
	Main postal surveyc	Patients transferred to a single GP cooperative during a 6-week period mailed a questionnaire within 7 days	748	1 reminder	342 (45.7)
Moll Van Charante (the Netherlands)5	Panel review of items	Patients from a regional patient federation			6
	Postal pilot	Consecutive patients or carers within 48 hours of request for care	696	No	285 (41.0)
	Review of items	Patients recently contacting a cooperative (n = 13)
	Postal pilot			No	87 (48.3)
	Main postal survey	26 GP cooperatives covering 25% of the Dutch population sent questionnaires to 200 consecutive patients within 48 hours of contact	14 400	1 reminder after 10 days	7520 (52.2)
	Test–retest	Responders from one cooperative from the main survey mailed at 1 week	338		155 (45.9)

^aData relating to responder characteristics were both inconsistently and seldom reported across studies and hence are not reported here.

^bResponse rates were over 50% in most practices.¹⁰

^cThree different questionnaires were used; one with the SQOC only, one with a longer questionnaire and one with both. The table includes the results for the SQOC only questionnaire. SQOC = Short Questionnaire for Out-of-Hours Care.

The Patient Satisfaction with Out-of-Hours Care (PSOC)

The PSOC was developed to assess patient satisfaction with domiciliary out-of-hours care suitable for large-scale service evaluation within the UK.¹¹ It can be interview- or self-administered. The 32 items have a five-point Likert scale. Item development was based on a review of the patient satisfaction literature including existing questionnaires, patient focus groups, and semi-structured interviews. Following two postal pilots (Table 2), items with poor data quality were removed. The questionnaire was evaluated concurrently within a comparative trial of out-of-hours care provided by deputising services and practice doctors at 14 practices, and included 1466 patient participants; 1402 (95.6%) patients completed the questionnaire by interview administration.

Van Uden questionnaire

This questionnaire was developed to assess patient satisfaction with current out-of-hours care organised in GP cooperatives, and to assess factors associated with this satisfaction in the Netherlands.¹⁹ There are three versions of the questionnaire for each type of contact: GP consultation (28 items), home visit (24 items) and telephone advice (22 items). Items have a five-point Likert scale.

The questionnaire was developed following a review of existing questionnaires, interviews with GPs and healthcare managers, consultation with patient organisations, and a review of discussions of out-of-hours care in newspapers. The resulting items were sent to patient organisations, health insurance funds, and five GP cooperative organisations for review. The questionnaire was then modified and assessed for clarity by five people with experience of out-of-hours primary care. Questionnaires were mailed to 2733 carers and patients within 3 weeks after contacting the GP cooperative, with reminders after 3–4 weeks; 1160 (42.4%) responded (Table 2).

Short Questionnaire for Out-of-Hours Care (SQOC)

The SQOC was designed to be a short measure of patient satisfaction for different providers of out-of-hours care.⁷ The questionnaire is based on the longer PSOC described above and comprises seven items representing the PSOC scales (Table 1). Pilot studies of the draft questionnaire were carried out in three sites in Scotland (Table 2). The main survey was conducted in a general practice cooperative with 77 GPs providing out-of-hours care for 139 000 patients. Questionnaires were mailed to 1906 consecutive patients randomised to different versions of the questionnaire with alternate scaling formats. Reminders were sent after 14 days. Response rates ranged from 39.7 to 45.7%, and a five-point scale (very satisfied, satisfied, neutral, dissatisfied, and very dissatisfied) with smiley faces gave the best results (Table 1).

Moll van Charante questionnaire

The questionnaire was designed for patients contacting their out-of-hours GP cooperative as part of a national survey in the Netherlands.⁵ There are three versions for patients receiving a centre consultation (29 items), home visit (23 items), and telephone advice (14 items). Items have a 10-point scale with a not-applicable option. Item development was based on a literature review and interviews with patients and health professionals. Eight GPs and four telephone nurses reviewed the items, which were then independently assessed by three experts in the field of questionnaire development. Six patients from a regional patient organisation formed a panel that further reviewed the questionnaire. This was followed by two postal pilots and further interviews with 13 patients, which informed the removal or rephrasing of items (Table 2). All GP cooperatives in the Netherlands were invited to participate and 26 took part, covering around a quarter of the total population. Cooperatives sent postal questionnaires to 14 400 patients within 48 hours of contact, and 7520 (52.2%) were returned.

Questionnaire content

There is some overlap in the content of the three longer questionnaires but they differ in the number of items relating to specific aspects of satisfaction and the scales that they form (Table 1). These three assess aspects of patient satisfaction relating to care received from the doctor in person or through telephone advice. The former is measured with one scale within the Dutch questionnaires and three scales within the PSOC — communication and management, doctor's attitude, and continuity of care. All three include items relating to telephone access as represented by scales within the PSOC and van Uden questionnaire, and one item within the organisation scale of the Moll van Charante questionnaire. The two Dutch questionnaires also include items relating to organisation or waiting times at the GP cooperative in the form of two scales for the first, and one scale relating to organisation for the latter. The PSOC includes a scale relating to the delay until the home visit. The Dutch questionnaires address this aspect of satisfaction with single items, the van Uden questionnaire with a single-item scale, and Moll van Charante within the organisation scale. The PSOC and van Uden questionnaires include scales assessing overall satisfaction. The van Uden questionnaire includes two scales not covered by the other questionnaires relating to the urgency of the patient's complaint and the distance to the cooperative. The Moll van Charante questionnaire includes two items not covered by the others within the organisation scale, relating to accessibility of the pharmacy and the time between the initial contact and being seen at the GP cooperative. The content of the questionnaires is summarised in further detail in Supplementary Table 2.

Data quality, reliability, and validity

There was some variation in the reporting of data quality and the extent to which this informed development for the different questionnaires. Missing data at the item level were only reported for the Moll van Charante questionnaire, but items with high levels of missing data were removed from the PSOC during development. The former had considerably high levels of missing data, which was largely due to the inclusion of patients responding ‘not applicable’. The levels of missing data at the scale level were either reported or discernible for all but the Moll van Charante questionnaire. This is an important omission given the high levels of missing data at the item level. The mean scale scores were not reported for this questionnaire but were for the others. Given the mean item scores, this questionnaire will, like the others, have scale scores skewed towards higher levels of satisfaction.

Evidence supporting the internal structure or scales of the questionnaires following principal component analysis was reported for all but the SQOC. The four questionnaires have been evaluated for internal consistency. However, item–total correlation was only reported for the Moll van Charante questionnaire. The SQOC and Moll van Charante questionnaires both had acceptable levels of Cronbach's α. The PSOC scales of access to out-of-hours care, continuity of care, and delay until visit did not meet the criterion of 0.7 for α. The van Uden scale of questions asked by doctor's assistant had levels of α below this criterion for the telephone-advice and GP-consultation versions of the questionnaire. The scales of waiting time at the cooperative and waiting room also failed to meet the 0.7 level for the GP-consultation version of the van Uden questionnaire. Test–retest was assessed for the PSOC and Moll van Charante questionnaires, and with the exception of the scale of initial contact person for the former, all estimates were above 0.7.

All four questionnaires have limited evidence for construct validity. There were no a priori hypotheses, few tests were reported, and of those that were, few were explicit tests of validity. The PSOC, van Uden and SQOC questionnaire scores were correlated with overall satisfaction. The SQOC items had only low to moderate levels of correlation with the scores for the parent questionnaire, the PSOC. However the SQOC is based on single items that sum to give a unidimensional scale and hence it differs from the structure of the PSOC. There was a significant association between SQOC scores and whether the patient was happy with their care setting. Following previous findings,^8–10 compared to patients receiving care in other settings, those receiving telephone advice had lower levels of satisfaction as assessed by the two Dutch questionnaires and SQOC. Finally, older patients reported higher levels of satisfaction as assessed by the van Uden questionnaire, which follows findings from the general patient satisfaction literature.¹⁵

DISCUSSION

Summary of main findings

Four published questionnaires were found following systematic searches of several electronic databases. All four were developed using a combination of literature reviews and expert opinion. The PSOC and Moll van Charante questionnaires also involved patient interviews or focus groups, which, insofar as they include aspects of out-of-hours services that are relevant to patients, lends them content validity. The Moll van Charante and van Uden questionnaires have different versions or questionnaire modules that reflect the type of out-of-hours contact. Following further development, there are now versions of the PSOC for the three care settings (R McKinley, personal communication, 2006).

The review identified serious shortcomings relating to published evidence for data quality and measurement properties that are cited as being important for questionnaires designed to measure patient satisfaction.¹⁴ Much of the work involved several large samples of patients, which afforded the opportunity for more extensive item and scale testing than reported. Content validity was explicitly considered for just two questionnaires, the PSOC and Moll van Charante. Principal component analysis informed scale construction for the Moll van Charante, PSOC and van Uden questionnaires, however, its role was not clear for the latter. The use of item–total correlation may have informed item removal and hence improved the reliability of the PSOC and the van Uden questionnaires, some of the scales of which have low levels of Cronbach's α. Some of the scales within the Moll van Charante questionnaire have more items than necessary.⁵ Item–total correlation may have informed the removal of items while maintaining necessary levels of α and improving the acceptability to patients of this relatively lengthy questionnaire. The scales within the PSOC and van Uden questionnaires, which have low reliability estimates from either internal consistency or test–retest methods, should be considered for refinement and used with caution in future surveys of patient satisfaction.

Tests of construct validity were the weakest component of questionnaire evaluation. Given the lack of explicit tests and the complete lack of a priori hypotheses, a pragmatic approach was taken to assessing the evidence involving consideration of the wider literature.^8–10,15 The PSOC, van Uden and SQOC were associated with overall satisfaction. Telephone advice produced relatively lower satisfaction levels for the Moll van Charante, van Uden and SQOC questionnaires.

Strengths and limitations of the study

According to the authors of these questionnaires there are a number of unpublished questionnaires that are being used for assessing patient satisfaction with out-of-hours services which do not have evidence for reliability and validity.^5,7,19 This further highlights the importance of this review of questionnaires that have published evidence for their data quality, reliability, and validity. The identification of existing questionnaires will allow potential users wishing to evaluate out-of-hours care to select the questionnaire that is most appropriate in terms of content and measurement properties for assessing the satisfaction of their patients. The use of existing questionnaires that have evidence supporting their application will improve the quality of surveys of patient satisfaction, and the use of standardised questionnaires will improve generalisability.

The existence of unpublished questionnaires may be evidence for publication bias but, as has been commented,^5,7,19 such questionnaires are not evaluated against the criteria necessary for a measure of patient satisfaction. There are a number of such questionnaires in use in Norway that have not been evaluated for data quality, reliability, or validity. However, such questionnaires may help in the identification of aspects of care that are relevant to patient satisfaction at the local level. In selecting instruments for application it is important that the content of such questionnaires is considered in relation to that of the four questionnaires reviewed here.

Comparison with existing literature

Searches of the literature did not reveal any other published reviews of patient satisfaction questionnaires for out-of-hours care. There are a number of questionnaires available for assessing patient satisfaction with out-of-hours care. Hence, this review of published questionnaires is timely and has as its focus evidence to support questionnaire application, including reliability and validity.

Implications for future research or clinical practice

The measurement of patient experiences and satisfaction is likely to continue to be an important component of primary care out-of-hours service evaluation. There are many of questionnaires available for assessing satisfaction,^4,6,18 but just four have published evidence documenting their development and evaluation against necessary criteria including reliability and validity. Those wishing to evaluate services from a patient perspective should use these questionnaires in preference to questionnaires that do not have evidence relating to reliability and validity. The four questionnaires that were reviewed differ in their content, including the items and scales that they use to assess satisfaction. This is an important consideration in choosing a questionnaire for a proposed application. The SQOC is much shorter than the other questionnaires, having just seven items that sum to form a single score. This makes it more acceptable to patients and it can be easily accommodated alongside other patient-reported outcome measures within a longer questionnaire. Users wishing to understand how the delivery of care relates to specific aspects of patient satisfaction are advised to consider one of the other three questionnaires that comprise a number of scales relating to different aspects of satisfaction.

In conclusion, the four questionnaires have limitations in terms of their development and evaluation. The development of the van Uden and SQOC questionnaires was not based on the views of patients, which has implications for content validity. Scales within the PSOC and van Uden questionnaires had poor reliability estimates and there was evidence for item redundancy within the Moll van Charante questionnaire. Given the small number of tests and lack of hypotheses, evidence for the validity of the four questionnaires was limited. The PSOC has undergone further development work which has yet to be published (C Salisbury, personal communication, 2006). The van Uden and Moll van Charante questionnaires require refinement. Together with questionnaire content, these limitations should be carefully considered when considering the application of the four questionnaires in future patient surveys.

Supplementary information

Additional information accompanies this article at http://www.rcgp.org.uk/bjgp-suppinfo

Funding body

This study was funded by the Norwegian Social and Health Directorate (Project Number 328)

Competing interests

The authors have stated that there are none