Look who’s taking notes in your clinic: mystery shoppers as evaluators in sexual health services

Abstract

Objectives  To test the feasibility of professional patients as a tool for sexual health service evaluation. Professional patients are paid to use services specifically for audit or evaluation purposes without disclosing their identity as evaluators.

Methods  Professional patients visited five large sexual health departments used by 3000 clients per week in two inner London Boroughs with very high rates of sexual ill health. They recorded their experience on a structured evaluation form. Semi‐structured telephone interviews were completed with seven service providers to document their experience of the programme.

Results  Recruitment and training for professional patients is described. Forty professional patients made 105 visits during two rounds of visits 9 months apart. After 47% (round 1) and 62% (round 2) of visits, the professional patients felt that they would recommend the service to a friend. The professional patients provided detailed and specific feedback on all aspects of service provision. This information was highly valued by service providers who reported few objections from staff to the visits. A small number of examples of very poor care were documented.

Conclusions  Professional patients are a useful tool for sexual health service evaluation. They provide high quality feedback because they are both ‘experts’ on sexual health service provision and users of sexual health services. This method of evaluation raises ethical issues about the acceptability of deception as part of the evaluation process, the right of staff to anonymity and to refuse to be visited. Professional patient programmes provide an opportunity for regular cycles of user feedback to monitor quality improvement.

Introduction

We are working to improve sexual health services in an inner city, deprived area with high rates of sexual ill health. (The Lambeth and Southwark Sexual Health Modernization Programme is funded by the Guys and St Thomas’ Charity to redesign sexual health services starting from the perspective of a service user.) Improving client experience is thought to increase use of sexual health services ¹ and therefore access to effective contraception and management of sexually transmitted infections (STIs). ²

We required a user involvement system to provide robust, rapid and detailed feedback of user experience across a whole system of sexual health service provision as a basis for service improvement. However, user involvement in sexual health service provision poses specific challenges:

• • Users visit services intermittently. They are less likely to build a relationship with a service than those with chronic illnesses and therefore less likely to participate in service development.
• • Users describe sexual health service use as stigmatizing ³ , ⁴ and concerns about confidentiality limit uptake of user involvement opportunities.
• • Users may feel disempowered in a service whose use they find stigmatizing and are consequently reluctant to evaluate it.

Professional patient programmes pay user/evaluators to visit services and report their experience. We chose this method because it collects quantitative and qualitative data on actual experience of service provision and potentially provides consistent, regular feedback to monitor service improvement.

Professional patients

Professional patient evaluation methods were developed in the private sector but have been used to evaluate health services for several decades. An early example is the study of psychiatric hospitals ⁵ where researchers reported hallucinations to gain admission and then behaved ‘normally’ while recording hospital life.

As service evaluators, professional patients play the role of clients and report on their experience. The visits are ‘standardized’ with similar scenarios presented to different service providers and the results compared. The health professionals being assessed may or may not know the true identity of the professional patients. The latter approach is more likely to document health professional behaviour without observation bias. ⁶

Much of the research reporting the use of simulated patients as an evaluation tool seeks to assess clinician performance against standards agreed by doctors ⁷ , ⁸ or access to specific services. ⁹ , ¹⁰ The work described here is unusual in that it used simulated patients to assess user satisfaction. The elements of service provision assessed are those already identified as important by local service users in focus groups. ¹¹ Focusing on these elements, the professional patients evaluate services in response to their personal expectations and standards rather than those set by professionals.

Setting

This study was conducted in two departments of genito‐urinary medicine, two sexual and reproductive health services and one specialist young people’s centre. These services, in two inner London Boroughs with very high rates of sexual ill health, an ethnically diverse population and high rates of socio‐economic deprivation are used by more than 3000 clients per week. Provider representatives from all services supported and participated in the project.

Design

Recruitment

Forty professional patients were recruited through an informal process ¹² from the following sources:

• • a group of gynaecology training associates (gynaecology training associates are laywomen trained to teach pelvic examination while themselves acting as the patient);
• • a support group for gay, lesbian and bisexual young people;
• • a sexual health peer educators project for young people;
• • a local database of sexual health service users interested in contributing to service development.

Invitation letters were sent to all those on the database and the invitation was given verbally by those leading the groups listed above. The recruitment process targeted population groups known to be at high risk of sexual ill health. Sixteen of the professional patients are male and 24 female with an average age of 26.5 years (range 16–50+). Their ethnicity is given in Table 1.

Table 1.

 Ethnicity of professional patients

Ethnicity	Frequency	Ethnicity	Frequency
White British	9	Black African	2
Black British	12	Mixed ethnicity	3
Black other	1	Not stated	7

Training

The professional patients attended a training session covering health and safety, the evaluation form, support for difficult visits or positive results and the three test scenarios, (a request for either emergency contraception, a screen for STI or advice on contraception). They received a comprehensive information pack detailing their obligations as professional patients and the obligations of their employers.

Professional patient visits

The professional patients may present the scenarios as they choose, but the presentation must be straightforward to evaluate basic service provision rather than the management of complex situations. For example, professional patients are asked not to simulate an excessively demanding or difficult client. They record their experience on a structured evaluation form and are paid a fee on receipt of the fully completed form. Professional patients are expected to make a reasonable attempt to access the service, which may require more than one visit and several telephone calls. Where the scenario requires it they may undergo investigations, for example, testing for STIs.

The data collection tool

The evaluation form was developed from focus groups with 93 service users ¹⁰ to identify a list of issues important to them. These are consistent with the published literature on this subject and include:

• • confidentiality and staff attitudes;
• • the quality of waiting room environments;
• • the length of waiting time;
• • the quantity and quality of information provided.

The evaluation form includes closed questions to generate quantitative data (yes/no answers or Likert scales) and open questions to generate qualitative responses (free text).

Interviews with service managers/senior clinicians

Structured telephone interviews were conducted with senior service managers and clinicians by a researcher who was independent of the professional patient programme (KCB). Respondents were selected because of their seniority involvement in service development and improvement. In all cases, the person interviewed was the most senior manager or clinician with responsibility for service improvement in each setting. Notes were taken during and immediately after each interview on a structured form. Respondents were invited to participate at the start of the interview and gave verbal consent.

Analysis

All of the comments for a particular service were analysed together. The quantitative data were presented as simple proportions and the qualitative data were analysed thematically. The comments on each section of the clinic journey were collated and read and re‐read to identify themes. The data were then coded according to these themes. Data from each service were fed back to a senior manager and a senior clinician in that service.

Themes from the telephone interviews with service providers were identified and the interviews coded according to these themes. Where this additional information contradicted the themes, the interpretation was reviewed until re‐reading of the data generated no further modifications.

Results

Two rounds of professional patient visits were completed 9 months apart.

Round 1

Forty‐three visits were made to 13 sexual health clinics (nine by men and 34 by women). Individual clinics received between one and five visits. The time spent in the clinic ranged from 5 to 395 min (mean = 94 min).

Fewer than half of the professional patients were satisfied with the quality of the services provided and after only 47% visits would have recommended the service to a friend. Some problems recur across most of the services. Almost half had difficulty in finding the clinic, a similar proportion had a poor perception of the reception and waiting facilities. Reception was described as unfriendly and lacking confidentiality. The waiting facilities were overcrowded and uncomfortable with insufficient chairs, a tense atmosphere and a lack of refreshments and entertainment. There are a small number of reports documenting a very poor standard of care, including one where access to a service had been inappropriately denied and one where tests had been taken without consent.

Round 2

In the second round, 62 visits (27 by men and 34 by women) were made to 18 sexual health clinics. Individual clinics received between one and five visits. The time spent in the clinics ranged from 3 to 225 min (mean = 67.6 min). The reports focussed on similar issues to those from round 1 and 34/55 (62%) would recommend the service to a friend. Again there were a small number of reports of a very poor standard of care including a clinical examination done in a very insensitive manner.

The response of service providers to the professional patient feedback

Seven service managers from all of the services involved completed telephone interviews including five clinicians. Two of those interviewed are the lead clinician for their clinic and two are the manager responsible for that service. All agreed that high quality data on user experience is an essential basis for service improvement.

It’s hard to make improvements if you don’t really know what the service is like from the outside (Clinician manager)

The experience had felt ‘daunting’ for one and ‘positive’ or ‘fine’ for the others. Five of those interviewed had been involved in the initial discussions on the value of such a programme and had therefore had an opportunity to comment on it during its development. The respondent who felt that it was daunting reported that her staff were initially concerned that it would mean extra work and criticism of their practice; however, after two or three rounds of the professional patient programme, they did not feel that these concerns were justified. Two respondents felt that their staff had recognized shoppers and in one case the clinician had found a professional patient’s conduct inappropriate. She was particularly concerned about the impact of this additional work in a busy clinic.

The mystery shopper made the consultation into a very prolonged process when actually it was a very busy clinic which felt a little inappropriate (Clinician manager)

None of the participants reported staff who were unwilling to be visited, there were no problems obtaining consent and although some staff were apprehensive at first they have become familiar with the idea and value the feedback.

Managers valued the feedback as more specific and detailed than patient satisfaction questionnaires. Those who normally got their information on user satisfaction from complaints monitoring valued the positive in addition to the negative feedback. Some felt that it reinforced what they already knew and others found it ‘eye opening’, ‘think we are providing a wonderful service and turns out there are things we need to brush up on’.

None of the managers had ethical concerns about the project. One manager commented on the repetitive nature of the feedback, interpreting this as a failure to move beyond reporting the same aspects of the experience rather than a failure of services to improve these aspects. One noted that the professional patient feedback was unusual, as users rarely feel sufficiently empowered to comment on the clinical consultation.

Examples of professional patient data

Table 2 is an example of the type of data that is fed back to service managers as a basis for service improvement.

Table 2.

 How polite was the receptionist? (all services)

	Round 1 (%)	Examples of comments (focus on negative comments that indicate need for improvement)	Round 2 (%)	Examples of comments (focus on negative comments that indicate need for improvement)	Action taken
Very rude	2 (4.7)	She humiliated me completely in front of her colleague This was an uncomfortable and embarrassing experience The receptionists gave me the full impression that they had better things to do with their time rather than speak to me	3 (4.8)	The receptionist in the sexual clinic was too abrupt. She was firm. I think she was a bit too harsh They were stern and sounded more like stressed out business men	Receptionist training in ‘customer care’ offered to all services
Rude	5 (11.6)		7 (11.3)
Polite	16 (37.2)		38 (61.3)
Very polite	15 (34.9)		11 (17.7)
No response	5		3
Total	43		62

Adverse incidents

Out of 92 visits, three caused the professional patients significant distress. Where this was reported, the options for addressing the problem were limited by the terms of the professional patient programme. This specified that data on the behaviour of individual would not be fed back in such a way as to identify them. In these situations, feedback to individuals could happen only if the professional patient agreed to make a formal complaint.

One professional patient was recognized by a receptionist from a visit to another clinic and was publicly questioned about this. The receptionist did not realize that the user was a professional patient and this member of staff’s behaviour was inappropriate. It is legitimate for service users to seek a second opinion at a different service if they choose to and their decision should never be questioned without an opportunity to discuss the reasons for their visit.

This was very difficult and awkward for me as the receptionist recognised me from an earlier visit this week to xxx and she gave me hell for going to another clinic and said I could only go to one!!

In another incident, a professional was recognized by a clinician who asked that the consultation be terminated because, although she had consented to be visited, she felt unable to continue the consultation once the true identity of the professional patient was known to her. Telephone support is now available for all professional patients.

Professional patients – patients or professionals?

Some professional patients clearly made an effort to see services from the perspective of those using them for the first time:

How did I feel? I felt fine because I was just pretending to have an STI!! But if I had been a proper patient, I would have felt quite annoyed and frustrated, having tried and failed to get info by phone, and then being turned away.

The receptionist was extremely unpleasant and this stage of the visit made me feel like if I had been unwell I would rather go home and die than risk such treatment again.

Whereas others record their own feelings during the visit:

A sheet of paper was thrust at me by a not very helpful receptionist … I felt very pressurized and nervous.

Practical issues in relation to the delivery of a professional patient evaluation programme

Professional patients need regular contact to maintain their interest/engagement in the programme. Continual recruitment and overallocation of visits are necessary to counter drop out of shoppers and allocated visits not being completed or reported. Establishing a payment system that is quick, efficient, transparent and auditable is essential and challenges current National Health Service (NHS) policies and standard operating procedures. Feedback to providers should be linked to the development of action plans to improve user experience.

Discussion

Patient and public involvement is a core policy theme within the NHS ¹³ and a key driver for service development. Our study shows that professional patient evaluation programmes are feasible in sexual health services and acceptable to service providers.

Although user involvement is becoming an essential part of service improvement, it often consists of one‐off activities and not the ‘user involvement cycle’ embedded within service development that has been recommended. ¹⁴ The two rounds of visits reported here are part of an on‐going programme of professional patient evaluation of a whole system of sexual health service provision. The data form the baseline evaluation of client satisfaction prior to the implementation of a service improvement programme.

Strengths and weaknesses of professional patient programmes

The advantages of professional patients as evaluators is that they provide detailed and specific feedback on pre‐specified elements of service provision. They keep contemporaneous records and are less anxious than ‘normal’ service users. They are not unwell and do not require the services they are provided with. They therefore have more opportunity for observation. Multiple visits lead to unique knowledge of the different clinics and encourage comparisons within and between services.

A possible disadvantage is that the more experienced the professional patients become the less like normal users they are. Their experience could theoretically translate into a heightened awareness of one or more aspects of the quality of service provision or a familiarity with sexual health service use that would make them less sensitive to the concerns of inexperienced service users such as confidentiality in reception. In both cases, their experience would differ from that of non‐expert users. This study shows that the shoppers do make specific efforts to put themselves in the shoes of inexperienced service users, but we have no information on the effectiveness of these efforts.

This problem could be minimized if the professional patients were asked to evaluate standardized aspects of service delivery, for example, adherence to a specific clinical protocol. In this study, however, the aim was to assess more generic standards of care such as ‘were the receptionists polite?’. The data therefore reflect the evaluator’s personal perceptions of appropriate receptionist behaviour and it is likely that there is some variation in experience. By working with a large pool of professional patients, we aim to reflect a diversity of expectations. The results of the evaluation can only specify that one or more professional patients found a particular practice unacceptable. They are therefore a starting point for further investigation.

Professional patient programmes require the recruitment, training and support and payment of the patient evaluators. This raises questions about their value in relation to other methods of documenting users’ experience of health services. User satisfaction surveys are known to produce high levels of reported satisfaction even where there is clear evidence of dissatisfaction from qualitative interviews with the same users. ¹⁵ , ¹⁶ , ¹⁷ Users are reluctant to criticize health professionals on whom they depend for care and this situation is exacerbated when they experience their condition as stigmatizing. Qualitative interviews provide a safer environment for users to comment on health services but are feasible for small numbers of service users only. Further work is required to identify how the data generated compare with more traditional methods like qualitative interviews and to document the relative costs of professional patient‐programmes in comparison with traditional methods.

Ethical issues

The deception that is an inevitable part of professional patient study raises questions about the ethics of evaluating the performance of health‐care workers without their knowledge. ⁹ Health service providers have a responsibility to ensure high standards of care but ‘ethical problems (arise) when health‐care workers become unwitting subjects swept up in studies that they may know nothing about and may run counter to their interests’. ⁹

Potential ethical objections to this methodology include:

• • the opportunity costs for the clinical service – health professionals spend time seeing professional patients when they could be providing care to genuine patients;
• • possible negative effect of the health‐care experienced for professional patients who consented to this care for the purposes of the research only;
• • breakdown of trust between health professionals and those conducting the research as a result of the deception that is a part of professional patient evaluations.

In this study, the potential ethical objections were minimized using the following strategies:

• • All participating health professionals gave consent to be visited, unannounced, by a professional patient during an agreed time period. A small number chose not to participate and their decision was respected.
• • The anonymity of individual health professionals was maintained by the study and general feedback was provided to service managers rather than information on the performance of specific staff members. This policy caused problems when professional patients reported practice of an unacceptably low standard. In this situation, unless the shopper was prepared to make a formal complaint, details of the incident could not be passed to managers.
• • The study was discussed by the local ethics committee, providing an opportunity for discussion of the ethical issues it raises with an independent group and ethical approval was obtained.
• • The length of the consultation by the professional patient and therefore the disruption to the service was kept to a minimum by briefing the professional patients to present a simple problem in a straightforward manner.

Conclusion

Professional patient evaluation programmes are feasible in sexual health services and acceptable to service providers. We were able to recruit a pool of service users who were prepared to visit and evaluate local sexual health services. Detailed and specific feedback on pre‐specified elements of service quality was obtained that was free from observation bias. However, as professional patients become more experienced, their expectations may diverge from those of typical service users.

This approach to data collection raises important ethical issues about the acceptability of deception in health service evaluation and the opportunity costs of this approach. In this programme, ethical concerns were minimized by the involvement of service representatives in programme development and discussion with the local ethics committee.

The professional patient programme generated specific practical difficulties for services including systems for paying users and programmes to maintain professional patient engagement in this work. Further work is required to compare the data generated by professional patient programmes with those generated by alternative qualitative data collection techniques. Such research would identify those circumstances when a professional patient programme is most useful.

Competing interests

None.

Contribution of authors

Paula Baraitser initiated the mystery shopper programme, contributed to data analysis and wrote the paper.

Vikki Pearce, Jo Holmes, Richard Cooper and Lovelle Smith developed and managed the mystery shopper programme.

Nathalie Walsh contributed to data analysis.

Kirsty Collander Brown interviewed the service providers.

Petra Boynton commented on the manuscript.