Abstract
Background
Simulated patient, or so-called ‘mystery-shopper’, studies are a controversial, but potentially useful, approach to take when conducting health services research.
Aim
To investigate the construct validity of survey questions relating to access to primary care included in the English GP Patient Survey.
Design and setting
Observational study in 41 general practices in rural, urban, and inner-city settings in the UK.
Method
Between May 2010 and March 2011, researchers telephoned practices at monthly intervals, simulating patients requesting routine, but prompt, appointments. Seven measures of access and appointment availability, measured from the mystery-shopper contacts, were related to seven measures of practice performance from the GP Patient Survey.
Results
Practices with lower access scores in the GP Patient Survey had poorer access and appointment availability for five out of seven items measured directly, when compared with practices that had higher scores. Scores on items from the national survey that related to appointment availability were significantly associated with direct measures of appointment availability. Patient-satisfaction levels and the likelihood that patients would recommend their practice were related to the availability of appointments. Patients’ reports of ease of telephone access in the national survey were unrelated to three out of four measures of practice call handling, but were related to the time taken to resolve an appointment request, suggesting responders’ possible confusion in answering this question.
Conclusion
Items relating to the accessibility of care in a the English GP patient survey have construct validity. Patients’ satisfaction with their practice is not related to practice call handling, but is related to appointment availability.
Keywords: access, family practice, patient simulation, survey, validity of results
INTRODUCTION
Although uncommon in health research, the use of simulated patient (‘mystery-shopper’) methods is a potentially powerful means of obtaining information regarding the availability of health care. Although these techniques have been widely used in commercial and marketing environments over several decades, questions have been asked about the ethics of this, inherently deceptive, approach.1 Within health and social research, concerns regarding physician privacy and informed consent have also been raised as objections.2,3 In contrast, perceived advantages include reduced user recall bias, reduced reliance on surveys, and reduced social desirability bias in reporting access arrangements.
Between 2008 and 2011, around 5.5 million adult patients were surveyed annually as part of the English national GP Patient Survey, with survey results reported for all English practices. Items addressed key aspects of UK primary care, including four questions on the accessibility of care, as well as questions on overall satisfaction with care and the willingness of patients to recommend their practice.
In spite of earlier research (C Paddison et al, unpublished data, 2013),4 an assessment of the validity of the survey remains to be undertaken. To provide evidence regarding the validity of access-related items, a mystery-shopper survey of appointment and doctor availability was conducted;5 the findings were related to the results obtained in the national survey of patients’ experience of care.
METHOD
All 249 practices in Devon and Cornwall were ranked according to their scores on a question addressing the patient’s ability to obtain a GP appointment ‘fairly quickly’ in the 2009–2010 GP Patient Survey. Practice list sizes and settings were noted. The study aimed to identify (with the assistance of the local Primary Care Research Network) and recruit 40 practices,6 with an approximately even split between those with a low (lowest third of national ranked scores), medium, and high (highest third) score. Practices agreed to participate after being given an explanation of the study and its simulated patient methodology.
Between May 2010 and March 2011, a researcher simulated a patient and telephoned practices to request the first available routine appointment for a non-urgent problem; they asked to be seen as soon as possible. If probed, the researcher described the problem as non life-threatening, but declined to give further details; if requested to do so, they disclosed their identity as a researcher. Requests were presented each month during randomly selected weekdays and time slots (08:30–09:00, 09:01–10:00, 10:01–12:00, 12:01–17:00) with up to six dialling attempts made if necessary. The researcher requested the first available appointment, either with any doctor or, on 50% of occasions, with a specified doctor randomly selected from those working more than five sessions per week.
How this fits in
Mystery shopper’ studies have a potentially useful place in health research. This approach was used to assess the validity of questions in the English GP Patient Survey. The results support the validity of questions relating to access to care and appointment availability.
It has been suggested that the availability of the third appointment is a preferred measure of practice availability.7–9 The researcher, therefore, declined the first two available appointments as unsuitable and, after obtaining details of the third, stated that they would contact the practice again to finally book. For each request, the following data were documented:
the type of call system (automated/manual);
the number of diallings necessary to get through to the practice;
the time from the end of dialling to the point at which the call was answered (manually or automatically);
the time from the end of dialling to complete the successful call, representing the time taken to resolve the appointment requests; and
the number of working days until the first and third available appointments.
Based on an intraclass correlation coefficient of 0.4, provided by Salisbury et al in an earlier study (C Salisbury, personal communication, 2009),6 it was calculated that this study would achieve reliability exceeding 0.85 in estimating practice level mean times to the first available appointment. Salisbury et al’s study was concerned with a similar area of investigation (appointment availability, continuity of care, and workload in general practices), and had also adopted a mystery-shopper approach to the enquiry.
Analysis
List sizes for participating and non-participating practices were compared. Seven summary outcome measures of access were calculated; these aggregated call level data at both practice level and at the level of the GP Patient Survey access score groups (high-, medium-, or low-score survey groups). Four of these measures related to practice call-handling performance:
percentage of calls answered by an automated system;
percentage of calls not answered on the first dialling;
median times (seconds) to answer the call; and
median times (seconds) to complete the call.
The remaining three were measures of appointment availability:
median waiting times (days) to the first appointment;
median waiting times (days) to the third appointment; and
percentage of first appointments not available within 2 working days.
Differences in these summary measures between calls made to the low-, medium-, or high-score survey groups were ascertained with χ2 and Kruskal-Wallis tests. Linear mixed-effect models were used to assess the differential effect that requesting a specific doctor in practices in the three survey groups had on the call completion time and on the first and third appointment waiting times.
Seven practice level scores were extracted from the GP Patient Survey 2010–2011 dataset (approximately midpoint in the study, survey available at www.gp-patient.co.uk/questionnaires, accessed 5 January 2013) concerning patients’ experiences of obtaining appointments and their overall satisfaction with care at the practice. Those seven scores reflected the percentage of responders, who reported:
finding it ‘very’ or ‘fairly’ easy getting through to the practice on the phone;
finding the receptionists ‘very’ or ‘fairly’ helpful;
being able to see a doctor within 2 working days;
being able to get an appointment more than 2 working days in advance;
being able to see their preferred GP;
being ‘very’ or ‘fairly’ satisfied with the care received at the practice; and
they ‘would definitely’ or ‘might’ recommend the practice to someone new to the local area.
These GP Patient Survey scores between practices in the low-, medium-, or high-score survey groups were tested for differences using Kruskal-Wallis tests. The seven practice-level, mystery-shopper measures were correlated with the seven GP Patient Survey measures.
RESULTS
Of 49 practices approached, 41 (84%) agreed to participate (11 low-, 11 medium-, and 19 high-scoring survey practices). Participating practices were similar in respect of the number of registered patients when compared with all other local practices (data not shown), were based in rural (n = 7), urban (n= 33) or inner-city (n = 1) settings, and 22 of 41 (54%) operated automated telephone-answering systems. Study practices had between 117 and 467 GP Patient Survey questionnaires returned, sufficient for high practice-level reliability (>0.9) on all seven measures of interest.4 Of 410 appointment booking attempts, the researcher accessed the practice at the first dialling on 314 (77%) occasions and successfully obtained potential appointments on 399 (97%) occasions (Table 1), of which only 17 (4.3%) involved revealing their identity.
Table 1.
Number of dialling attempts before contact made
| Dialling attempts | Calls, n(%) |
|---|---|
| 1 | 314 (77) |
| 2 | 64 (16) |
| 3 | 20 (5) |
| 4 | 6 (1) |
| 5 | 2 (0) |
| 6 | 4 (1) |
| Total | 410a (100) |
On 11 of these occasions, no successful contact was made, either because the caller could not get through (telephone persistently engaged across six attempts (n = 1), unhelpful automated answering system (n = 3), or because no bookings were being taken at the time of the call (n = 7).
Practices in the low-, medium-, and high-score survey groups were similar in their use of automated answering systems and the proportion of calls answered first time, but varied in respect of the other five measures; this variation was largely consistent with their survey-score groupings (Table 2).
Table 2.
‘Mystery-shopper’ call handling and appointment availability measures, and national GP Patient Survey scores for high-, medium-, and low-scoring practices
| Practice access classificationa | P-value for test of group differences | |||
|---|---|---|---|---|
| High | Medium | Low | ||
| Mystery-shopper measures of accessb | ||||
| Call handling | ||||
| Calls answered by an automated system, %c | 57 | 45 | 50 | 0.090 |
| Calls not answered first time, %c | 23 | 21 | 27 | 0.506 |
| Time to answer the phone in seconds, median (IQR)d | 4 (7) | 7.5 (17) | 8 (11) | 0.001 |
| Time to complete the call in seconds, median (IQR)d | 70 (33) | 84.5 (55) | 81.5 (60) | 0.002 |
| Appointment availability | ||||
| Waiting time for first appointment in days, median (IQR)d | 1 (3) | 2 (6) | 3.5 (6) | < 0.001 |
| Waiting time for third appointment in days, median (IQR)d | 2 (4) | 3 (7) | 5 (7) | <0.001 |
| First appointment not available within 2 days, %c | 26 | 44 | 60 | <0.001 |
| National GP Patient Survey scores, median % (IQR) | ||||
| Find it easy to get through by phoned | 92 (7) | 70 (29) | 83 (9) | 0.001 |
| Find the receptionists helpfuld | 98 (2) | 94 (4) | 92 (4) | 0.001 |
| Can get an appointment fairly quicklyd | 93 (8) | 84 (3) | 72 (15) | <0.001 |
| Can get an appointment aheadd | 81 (17) | 69 (13) | 76 (14) | 0.187 |
| Can see their preferred doctord | 82 (22) | 70 (22) | 60 (15) | 0.002 |
| Are satisfied with the practiced | 96 (5%) | 93 (5) | 89 (7) | <0.001 |
| Would recommend the practice to othersd | 93 (6) | 88 (9) | 84 (9) | 0.001 |
Classification based on national scores of timeliness of appointment availability when a prompt appointment was requested.
Measures across high-, medium-, and low-scoring practices based on 190, 110, and 110 research calls (respectively) made to practices.
Differences between practices in the high-, medium-, and low-scoring groups tested by χ2 tests.
Differences tested by Kruskal-Wallis tests.
Requesting a specific doctor had little effect on the time taken to conclude the telephone request process, but significantly increased mean waiting times for the first and third appointments. These increased waiting times were greater in low-scoring practices than in high-scoring practices. On average, the additional wait for the first available appointment in low-scoring practices was 3.4 days compared with 1.2 days in high-scoring practices. The corresponding additional waits for the third available appointment were 3.8 and 1.7 days respectively (data not shown).
Practices’ GP Patient Survey scores for ease of telephone access were not significantly correlated with three of the four call-handling measures (P > 0.05) but were significantly associated with the time taken to complete the call and, somewhat surprisingly, with two of the three measures of appointment availability (Table 3).
Table 3.
Spearman correlation of seven measures of access with seven GP Patient Survey derived markers of patient experience of care for 41 practices at the practice levela
| GP Patient Survey scores, ρ (P-value) | ||||||||
|---|---|---|---|---|---|---|---|---|
| Mystery-shopper’ outcomes | Patients who find it easy to get through by telephone | Patients who find the receptionists helpful | Patients who can get an appointment fairly quickly | Patients who can get an appointment ahead | Patients who can see their preferred doctorb | Patients who are satisfied with the practice | Patients who would recommend the practice to others | |
| Type | Measure | |||||||
| Call handling | Calls answered by an automated system, % Calls not answered first time, % Time to answer the phone, seconds Time to complete call, seconds | –0.10 | –0.07 | 0.01 | –0.22 | 0.02 | 0.13 | –0.12 |
| (0.553) | (0.676) | (0.948) | (0.174) | (0.922) | (0.425) | (0.454) | ||
| –0.16 | –0.16 | –0.17 | 0.02 | –0.17 | –0.17 | –0.12 | ||
| (0.326) | (0.312) | (0.297) | (0.913) | (0.284) | (0.298) | (0.458) | ||
| –0.07 | 0.01 | –0.11 | 0.10 | –0.07 | –0.17 | 0.03 | ||
| (0.654) | (0.936) | (0.513) | (0.549) | (0.680) | (0.289) | (0.828) | ||
| –0.51 | –0.34 | –0.40 | –0.46 | –0.16 | –0.24 | –0.33 | ||
| (0.001)a | (0.029)a | (0.010)a | (0.003)a | (0.308) | (0.124) | (0.033)a | ||
| Appointment availability | Wait for first appointment, days | –0.26 | –0.40 | –0.66 | –0.07 | –0.49 | –0.42 | –0.40 |
| (0.098) | (0.009)a | (< 0.001)a | (0.680) | (0.001)a | (0.006)a | (0.010)a | ||
| Wait for third appointment, days | –0.40 | –0.44 | –0.70 | –0.25 | –0.56 | –0.46 | –0.43 | |
| (0.010)a | (0.004)a | (<0.001)a | (0.118) | (< 0.001)a | (0.002)a | (0.005)a | ||
| First appointments not within 2 days, % | –0.34 | –0.47 | –0.66 | –0.24 | –0.49 | –0.38 | –0.31 | |
| (0.029)a | (0.002)a | (< 0.001)a | (0.131) | (0.001)a | (0.014)a | (0.046)a | ||
Correlations where P<0.05.
Results in this column are based on only those appointments where a specific doctor was requested.
Strong associations10 were evident between the proportion of patients reporting that they could obtain an appointment ‘fairly quickly’ and all three experimental measures of appointment availability. The reported ability of patients to book an appointment in advance was uncorrelated with six of the seven experimental measures, but was moderately correlated with call completion times.
Practice scores for the ability of patients to see their preferred GP, and their overall satisfaction with, and willingness to recommend, their practice were associated with shorter waiting times for appointments. The correlations of GP Patient Survey measures with third available appointment times were only very slightly stronger than those with first appointment times. This may be ascribed to the strong correlation between the availability of the first and third appointments, when measured either at individual call level (ρ = 0.846, P < 0.001) or at practice level (ρ = 0.890, P < 0.001).
DISCUSSION
Summary
The results presented here support the validity of questions relating to access to care and appointment availability in the GP Patient Survey. Practices with higher scores relating to patients being able to get appointments ‘fairly quickly’ were observed to have more favourable performance across five of the seven measures of access in this study. The strongest association was evident in respect of GP Patient Survey scores for prompt access and this study’s measures of appointment availability following a request for an appointment ‘fairly quickly’.
The modest association of GP Patient Survey access scores with appointment availability and the time taken by reception staff to complete the call may be indicative of a less-coherent organisation among lower-scoring practices. Responders appeared to correctly distinguish between the GP Patient Survey questions addressing timeliness of access and the ability to book in advance.
The ability of patients to see a doctor with whom they are familiar when they wish to do so has been reported as a priority by patients. It has also been identified as ‘relational’; one of three domains of continuity, the others being ‘management’ and ‘informational’.11,12 In this study, the GP Patient Survey measure of relational continuity of care was neither related to the three measures of practice telephone-system efficiency, nor to a proxy measure of receptionist efficiency: namely, the time taken to successfully resolve the appointment request. However, relational continuity was related to appointment availability; practices characterised by the GP Patient Survey as providing less continuity of care experienced increased delays when an appointment with a specific doctor was requested.
Although proposed as having merits,5,8,9,13,14 the current study did not, in general, provide support for the use of the third available appointment as a preferred measure of access,7 when compared with first-appointment availability. The authors would counsel against the need for this more complicated and potentially less-reliable6 measure of practice access.
Strengths and limitations
The simulated patient methodology allowed for the recording of experiences of accessing primary care, while avoiding the problems of using less-direct methods to gather these data.3 This study was implemented carefully and with the informed consent of participating general practices. It also addressed some of the negative evaluations of other mystery-shopper studies.3,15–17 The researchers were able to collect practice data in a natural and unaffected way; such an approach may offer underutilised potential when assessing the accessibility and efficiency of healthcare provision.
There are some limitations to the observational design of this study,18including the potential for limited generalisability of the findings to areas outside of the south-west of England. Particular doctors were specified randomly when presenting simulated appointment requests; in reality, it is likely that appointment requests are concentrated on more popular doctors. In these circumstances, this study’s estimates of the delays incurred when requesting a specific doctor are likely to be underestimates of the true figure.
Comparison with existing literature
Mystery-shopper approaches have been used in the US to monitor and support healthcare provision.19 However, the study is aware of no other studies that have adopted a simulated patient approach to investigate the validity of a major national survey.
Patient satisfaction, and the likelihood of patients recommending their practice, were both related to the availability of appointments but not to the practice’s call-handling arrangements. Although a significant practice-level association between appointment availability and overall satisfaction was found, it has previously been shown that communication between doctor and patients is a stronger driver of overall satisfaction, than appointment availability at the patient level (C Paddison et al, unpublished data, 2013).
Implications for practice
These results suggest that the national GP Patient Survey, which now forms an important source of evidence in the UK NHS outcomes framework,20 is a valid means of collecting data on patients’ experiences of accessing primary care. Although intuitively valid, a question relating to the ease of telephone access lacked discriminatory potential. There also appears to be potential for confusion for responders in conflating ‘getting through’ on the telephone with the whole process of making an appointment. The study would, therefore, suggest a need for further cognitive and psychometric testing on the question of ease of telephone access.
This study suggests that prompt access to appointments is a factor influencing patients’ reported experience of primary care and their overall satisfaction with service provision. Previous research has indicated that an emphasis on improving access times in general practice may jeopardise the patient’s choice of primary care doctor and continuity of care.21–23 The development of appointment systems that facilitate achieving a balance between prompt access to healthcare and maintaining an effective relationship between the patient and the doctor should continue to be a key concern for policy makers.
The mystery-shopper approach proved to be useful with regard to assessing the validity of the access questions within the GP Patient Survey, particularly those that focus on appointment availability. Future research must continue to identify and model the drivers of patients’ reported primary care experiences, recognising that access arrangements are just one ‘colour’ in the complex patchwork of those experiences.
Acknowledgments
The authors gratefully acknowledge the support of participating practices in facilitating this research, and Professor Chris Salisbury, University of Bristol in supporting the early development of the project.
Funding
The study was funded by a grant from the UK Department of Health. The views expressed are those of the authors and not necessarily those of the Department of Health. All authors had full access to all of the data in the study and take responsibility for the integrity of the data and accuracy of the data analysis.
Ethical approval
This study was approved by South West Research Ethics Committee 2 [ref 09/H0202/65].
Provenance
Freely submitted; externally peer reviewed.
Competing interests
Martin Roland and John Campbell act as academic advisors to Ipsos MORI for the General Practice Patient Survey. All other authors have declared no competing interests.
Discuss this article
Contribute and read comments about this article on the Discussion Forum: http://www.rcgp.org.uk/bjgp-discuss
REFERENCES
- 1.Ng Kwet Shing. Spence LM. Investigating the limits of competitive intelligence gathering: is mystery shopping ethical? Business Ethics a European Review. 2002;11(4):343–353. [Google Scholar]
- 2.Department of Health. Research governance framework for health and social care. London: DoH; 2005. [Google Scholar]
- 3.Rhodes K. Taking the mystery out of ‘mystery shopper’ studies. N Engl J Med. 2011;365(6):484–486. doi: 10.1056/NEJMp1107779. [DOI] [PubMed] [Google Scholar]
- 4.Lyratzopoulos G, Elliott MN, Barbiere JM, et al. How can health care organizations be reliably compared?: Lessons from a national survey of patient experience. Med Care. 2011;49(8):724–733. doi: 10.1097/MLR.0b013e31821b3482. [DOI] [PubMed] [Google Scholar]
- 5.Salisbury C, Goodall S, Montgomery AA, et al. Does Advanced Access improve access to primary health care? Questionnaire survey of patients. Br J Gen Pract. 2007;57(541):615–621. [PMC free article] [PubMed] [Google Scholar]
- 6.Salisbury C, Montgomery AA, Simons L, et al. Impact of Advanced Access on access, workload, and continuity: controlled before-and-after and simulated-patient study. Br J Gen Pract. 2007;57(541):608–614. [PMC free article] [PubMed] [Google Scholar]
- 7.Murray M, Berwick DM. Advanced access: reducing waiting and delays in primary care. JAMA. 2003;289(8):1035–1040. doi: 10.1001/jama.289.8.1035. [DOI] [PubMed] [Google Scholar]
- 8.Institute for Healthcare Improvement. Primary care access measures at Institute for Healthcare Improvement. Cambridge, MA: IHI; 2012. [Google Scholar]
- 9.Dixon S, Sampson FC, O’Cathain A, Pickin M. Advanced access: more than just GP waiting times? Fam Pract. 2006;23(2):233–239. doi: 10.1093/fampra/cmi104. [DOI] [PubMed] [Google Scholar]
- 10.Cohen J. A power primer. Psychol Bull. 1992;112(1):155. doi: 10.1037//0033-2909.112.1.155. [DOI] [PubMed] [Google Scholar]
- 11.Haggerty JL, Reid RJ, Freeman GK, et al. Continuity of care: a multidisciplinary review. BMJ. 2003;327(7425):1219–1221. doi: 10.1136/bmj.327.7425.1219. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Cheraghi-Sohi S, Hole AR, Mead N, et al. What patients want from primary care consultations: a discrete choice experiment to identify patients’ priorities. Ann Fam Med. 2008;6(2):107–115. doi: 10.1370/afm.816. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.NHS Practice Management Network. Improving access, responding to patients; a ‘how-to’ guide for GP Practices. London: DoH; 2009. [Google Scholar]
- 14.Oldham J. Advanced access in primary care. Manchester: National Primary Care Development Team; 2001. [Google Scholar]
- 15.Asplin BR, Rhodes KV, Levy H, et al. Insurance status and access to urgent ambulatory care follow-up appointments. JAMA. 2005;294(10):1248–1254. doi: 10.1001/jama.294.10.1248. [DOI] [PubMed] [Google Scholar]
- 16.Bisgaier J, Rhodes KV. Auditing access to specialty care for children with public insurance. N Engl J Med. 2011;364(24):2324–2333. doi: 10.1056/NEJMsa1013285. [DOI] [PubMed] [Google Scholar]
- 17.The Medicaid Access Study Group. Access of Medicaid recipients to outpatient care. N Engl J Med. 1994;330(20):1426–1430. doi: 10.1056/NEJM199405193302007. [DOI] [PubMed] [Google Scholar]
- 18.Altman DG. Practical statistics for medical research. London: Chapman & Hall; 1991. [Google Scholar]
- 19.Centers for Medicare & Medicaid. Contract year 2011. Annual Election Period (AEP) Baltimore, MD: CMS; 2011. http://www.cms.gov/Medicare/Health-Plans/ManagedCareMarketing/Downloads/2011_AEPReport091311.pdf (accessed 14 Jan 2013) [Google Scholar]
- 20.Department of Health. The NHS Outcomes Framework 2012/13. London: DoH; 2012. [Google Scholar]
- 21.Gerard K, Salisbury C, Street D, et al. Is fast access to general practice all that should matter? A discrete choice experiment of patients’ preferences. J Health Serv Res Policy. 2008;13(Suppl 2):3–10. doi: 10.1258/jhsrp.2007.007087. [DOI] [PubMed] [Google Scholar]
- 22.Robertson R, Dixon A, Le GJ. Patient choice in general practice: the implications of patient satisfaction surveys. Health Serv Res Policy. 2008;13(2):67–72. doi: 10.1258/jhsrp.2007.007055. [DOI] [PubMed] [Google Scholar]
- 23.Rubin G, Bate A, George A, et al. Preferences for access to the GP: a discrete choice experiment. Br J Gen Pract. 2006;56(531):743–748. [PMC free article] [PubMed] [Google Scholar]